Low-illumination image enhancement method based on Retinex model

Abstract

The invention belongs to the field of computer vision, and particularly relates to a low-illumination image enhancement method based on a Retinex model. The method comprises the steps that a low-illumination image data set is acquired, and illumination priori is obtained through calculation according to low-illumination images; inputting the illumination priori and low-illumination image into a pre-lighting module for processing to obtain a pre-lighting image and an illumination feature image; inputting the pre-lighted image and the illumination feature image into a refined reconstruction module for processing to obtain an enhanced image; calculating the total loss of the model and adjusting model parameters according to the total loss of the model to obtain a trained low-illumination image enhancement model; performing low-illumination image enhancement by using the trained model; according to the method, more feature information can be extracted while less calculation amount is consumed, and the accuracy of the enhanced image is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and particularly relates to a low-light image enhancement method based on the Retinex model. Background Art

[0002] Low-Light Image Enhancement (LLIE) is a classic low-level task in the field of computer vision and plays an important role in many high-level computer vision tasks, such as object recognition, face recognition, object tracking, text recognition, and medical image processing, showing great potential in future applications. Low-light image enhancement is widely used in fields such as visual surveillance, autonomous driving, and computational photography, and has significant practical value.

[0003] With the rapid development of communication technology and information technology, people's requirements for the quality of pictures are getting higher and higher, and these directly obtained low-light images are not sufficient to meet people's needs. For example, insufficient contrast in the image will reduce the observer's ability to understand and analyze the image. For low-light videos lacking information such as contrast, many image enhancement methods have been proposed by researchers. The traditional ones are manually designed mathematical methods, such as histogram equalization in contrast enhancement, and the mapping curve from low-light images to enhanced-light images. In dark images with uneven illumination, low contrast will cause the details of both dark and bright areas to disappear. In this case, the contrast enhancement method may lead to over-enhancement of the bright area.

[0004] In recent years, with the development of deep learning technology, deep learning-based methods have become the mainstream of image enhancement methods. Compared with traditional methods, deep learning-based solutions have better accuracy, robustness, and speed, and thus have attracted more and more attention in recent years. Deep learning is based on the learnable parameters in the model, and the model is established by optimizing the parameters to reduce the loss by inputting the original data set. The established model can be used to enhance low-light pictures that have not been learned.

[0005] The Retinex theory states that an image can be decomposed into a reflection component and an illumination component. By constructing an objective function with a priori regularization terms, the illumination component is estimated. Finally, the enhanced illumination component is fused with the reflection component to obtain the enhanced image. In recent years, researchers have made many remarkable progress in this field by combining the Retinex theory and deep learning, including networks with Transformer as the backbone. However, when dealing with high-resolution images, the computational complexity of the self-attention matrix increases rapidly with the improvement of spatial resolution, which directly makes it difficult to apply Transformer in each layer of the U-shaped network commonly used in low-light image enhancement tasks. In addition, the downsampling process of the U-shaped network is prone to losing key information such as boundaries, scales, and textures, and this information loss will directly affect the image enhancement effect and weaken the performance of downstream tasks such as target recognition in dark scenes. On the other hand, the upsampling module used to restore high-resolution features in the traditional U-shaped network lacks sufficient flexibility. Therefore, it is necessary to research and develop an image enhancement algorithm for low-light environments that can extract more feature information while consuming less computational power to improve the image quality and usability in applications. Summary of the Invention

[0006] Aiming at the deficiencies of the existing technology, the present invention proposes a low-light image enhancement method based on the Retinex model. The method includes: obtaining a low-illumination image to be enhanced and inputting it into a trained low-light image enhancement model to obtain an enhanced image;

[0007] The training process of the low-light image enhancement model includes:

[0008] S1: Obtain a low-light image dataset and calculate the illumination prior according to the low-light image;

[0009] S2: Input the illumination prior and the low-light image into a pre-lighting module for processing to obtain a pre-lighting map and an illumination feature map;

[0010] S3: Input the pre-lighting map and the illumination feature map into a refined reconstruction module for processing to obtain an enhanced image;

[0011] S4: Calculate the total loss of the model and adjust the model parameters according to the total loss of the model to obtain a trained low-light image enhancement model.

[0012] Preferably, the data process of the pre-lighting module includes: the illumination prior and the low-light image are sequentially processed by a first convolutional layer, a depthwise separable convolutional layer, and a second convolutional layer to obtain an illumination mapping map and an illumination feature map; the illumination mapping map is multiplied element-wise with the low-light image to obtain a pre-lighting map.

[0013] Preferably, the fine-grained reconstruction module is a U-shaped encoding-decoding structure, which connects the encoder and decoder features through skip connections, including convolutional layers, CCAM modules, HaarDown modules, and DyUp modules.

[0014] Furthermore, the data processing process of the CCAM module includes: the input features of the CCAM module are sequentially processed through the LN layer and the IG-MDSA layer to obtain the first intermediate feature; the first intermediate feature is added element-wise to the input features to obtain the second intermediate feature; the second intermediate feature is sequentially processed through the LN layer and the FFN layer to obtain the third intermediate feature; the second intermediate feature is added element-wise to the third intermediate feature to obtain the output feature of the CCAM module.

[0015] Furthermore, the data processing process of the IG-MDSA layer includes:

[0016] The first input feature and the second input feature are evenly divided in the channel dimension, and the number of divisions is equal to the number of self-attention heads; where the second input feature is the illumination feature map;

[0017] In each attention head, self-attention calculation is performed on the first input feature and the second input feature of different divided parts to obtain the attention feature; the attention feature is processed through a fully connected layer to obtain the output of the attention head;

[0018] The outputs of multiple attention heads are fused to obtain the output of the IG-MDSA layer.

[0019] Furthermore, the formula for obtaining the attention value is expressed as:

[0020]

[0021] where, Attention(Q i ,K i ,V i ,Y i ) represents the attention feature, Y i represents the second input feature under the i-th attention head, Q i represents the query calculated from the first input feature under the i-th attention head, K i represents the key calculated from the first input feature under the i-th attention head, V i represents the value calculated from the first input feature of the i-th attention head, α i represents the learnable parameter of the i-th attention head, X i represents the first input feature of the i-th attention head; and represent three different 1×1 convolutional weight matrices; and represent three different 3×3 depthwise separable convolution weight matrices.

[0022] Furthermore, the HaarDown module consists of a Haar wavelet encoding module and a feature learning module; among them, the feature learning module includes a standard convolutional layer, a batch normalization layer, and a ReLU activation layer.

[0023] Preferably, the total loss of the model is the weighted sum of the spatial loss and the Fourier loss.

[0024] The beneficial effects of the present invention are as follows: The present invention uses Haar wavelet downsampling to fully fuse high-frequency and low-frequency features, effectively alleviates the loss of high-frequency or low-frequency information that may be caused by pooling operations, and maximally retains image details; the dynamic upsampling technology refines features by learning the generation of the sampling set, not only significantly improves the performance of upsampling, but also effectively avoids the artifact problems that may be brought about by methods such as transposed convolution upsampling; compared with the prior art, the present invention is outstanding in computational efficiency, especially good at processing high-resolution images, can fully fuse high-frequency and low-frequency features, and improves the accuracy of enhanced images. Description of the Drawings

[0025] Figure 1 is a schematic structural diagram of the low-light image enhancement model in the present invention;

[0026] Figure 2 is a schematic structural diagram of the CCAM module in the present invention;

[0027] Figure 3 is a schematic structural diagram of the HaarDown module in the present invention;

[0028] Figure 4 is a schematic structural diagram of the DyUp module in the present invention. Detailed Embodiments

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] The present invention proposes a low-light image enhancement method based on the Retinex model, and the method includes:

[0031] Construct a single-stage low-light enhancement network HaarReformer based on the Retinex model and the attention mechanism, that is, a low-light image enhancement model, as Figure 1As shown, it includes a pre - lighting module and a refined reconstruction module. Obtain the low - illumination image to be enhanced, input it into the trained low - light image enhancement model, and obtain the enhanced image.

[0032] The training process of the low - light image enhancement model includes:

[0033] S1: Obtain a low - light image dataset and calculate the illumination prior according to the low - light image.

[0034] Collect a low - light image dataset, including low - light images and their corresponding normal - light images, which can be sourced from public datasets and processed such as transformation of public - dataset images, as well as images obtained by electronic device sensors. At the same time, calculate the illumination prior, which can be obtained by calculating the average value of the low - light image at the channel level.

[0035] S2: Input the illumination prior and the low - light image into the pre - lighting module for processing to obtain a pre - lit map and an illumination feature map.

[0036] High - dynamic - range imaging technology (HDR) reveals that images with different exposures are complementary in terms of detail retention and local contrast enhancement, and effective fusion can enhance the enhancement effect of the image. The pre - lighting module explores the interaction between regions with different lighting conditions and generates a pre - lit image I p and illumination features where the illumination features are guided by the illumination prior.

[0037] The data process of the pre - lighting module includes: The illumination prior and the low - light image are successively processed by a first convolutional layer (1×1), a depth - separable convolutional layer (5×5), and a second convolutional layer (1×1) to obtain an illumination map and an illumination feature map; the illumination map and the low - light image are multiplied element - by - element to obtain a pre - lit map.

[0038] S3: Input the pre - lit map and the illumination feature map into the refined reconstruction module for processing to obtain the enhanced image.

[0039] The pre - lighting stage may introduce various degradations such as noise and color distortion, and regions with sufficient light can provide semantic context information for regions with insufficient light. The refined reconstruction module is designed to adjust the pre - lit image with degradations and output a clean enhanced image.

[0040] The refined reconstruction module designed in the present invention includes a convolutional layer, a Haar wavelet down - sampling module (HaarDown), a dynamic up - sampling module (DyUp), and a cross - channel context aggregation module (CCAM). Specifically:

[0041] To finely adjust the pre - lit image I p, the refined reconstruction module adopts a three-scale symmetric U-shaped encoding-decoding structure. In each layer of the encoder-decoder, multiple cross-channel context aggregation modules (CCAMs) are integrated to aggregate the context information of the latent features from the encoder and the decoder.

[0042] The encoder converts the pre-lit image into deep features gradually reducing the spatial size and expanding the channel capacity to achieve effective feature extraction. The decoder takes the low-resolution latent features as input and gradually restores the high-resolution feature representation To enhance the effect of feature extraction and reduce the information loss caused by traditional upsampling and downsampling, the HaarDown module and the DyUp module are respectively adopted. At the same time, the encoder features are connected to the decoder features through skip connections to help retain the structural and texture details in the restored image.

[0043] CCAM module:

[0044] In the transformer-based CCAM, the main computational overhead comes from the multi-head self-attention, and the performance gain of the network is significantly positively correlated with the window size. For traditional global self-attention, the time and memory complexity of the key-query dot product interaction grow quadratically with the spatial resolution of the input, that is, for an image with a spatial scale of W×H pixels, its complexity is O(H 2 W 2 ). It is not feasible to directly apply global multi-head self-attention to the U-shaped CNN containing high-resolution images. To alleviate this problem, the present invention proposes illumination-guided multi-head self-attention (IG-MDSA), which has a linear complexity in the spatial scale of the image. The key part is to apply self-attention to the channels rather than the spatial dimensions, that is, to generate an implicit encoded global context attention map by calculating the cross-covariance between channels.

[0045] As the basic unit of the refined reconstruction module, the structure of the cross-channel context aggregation module (CCAM) is as Figure 2 shown. The data processing process of the CCAM module includes: the input features of the CCAM module are sequentially processed by the LN layer (layer normalization) and the IG-MDSA layer to obtain the first intermediate feature; the first intermediate feature is added element-wise to the input feature to obtain the second intermediate feature; the second intermediate feature is sequentially processed by the LN layer and the FFN layer to obtain the third intermediate feature; the second intermediate feature is added element-wise to the third intermediate feature to obtain the output feature of the CCAM module.

[0046] The data processing process of the IG-MDSA layer includes:

[0047] The first input feature and the second input feature are evenly divided in the channel dimension, and the number of divisions is equal to the number of self-attention heads; where the second input feature is the illumination feature map. Specifically: the input feature is first evenly divided into several parts in the channel dimension and expressed as:

[0048]

[0049] where represents the feature evenly divided into equal parts by channel, C represents the channel dimension, and k represents the number of divisions.

[0050] To ensure matching the shape of the input feature X, the reshaped Y of the illumination feature map F lu is divided into equal parts by channel so as to fuse the illumination feature map with the input feature. This process can be expressed by the following formula:

[0051] Y = [Y1, Y2, …, Y k

[0052] In each attention head, self-attention calculation is performed on the first input feature and the second input feature of different divided parts to obtain the attention feature; specifically:

[0053] Convolution operations are applied to aggregate pixel-level cross-channel context information, and depthwise separable convolutions are used to encode channel-level spatial features, thereby generating projections of Query (Q), Key (K), and Value (V) with rich local context information. The formula is as follows:

[0054]

[0055] where X i represents the first input feature of the i-th attention head, W p (·) represents a 1×1 convolution operation, and W d (·) represents a 3×3 depthwise separable convolution. The network structure in the present invention uses an unbiased convolutional layer. The projections of Key and Value are reshaped and interact through dot product to generate a transposed attention map A of size , rather than the original large-scale spatial resolution attention map, thereby improving the computational efficiency and model performance.

[0056] Well-exposed regions can generate rich semantic context representations, which in turn promote image enhancement in dark regions. The illumination feature map F lu is input into IG-MDSA to guide self-attention. The self-attention of each head can be expressed as:

[0057] ​

[0058] Among them, Attention(Q i , K i , V i , Y i ) represents the attention feature, and Y i represents the second input feature under the i-th attention head, Q i represents the query calculated from the first input feature under the i-th attention head, K i represents the key calculated from the first input feature under the i-th attention head, V i represents the value calculated from the first input feature of the i-th attention head, and α i represents the learnable parameter of the i-th attention head.

[0059] The attention features of multiple attention heads are processed by a fully connected layer and added to the positional encoding to generate the output feature, which is the output of the IG-MDSA layer.

[0060] The computational complexity O(G-MSA) of global multi-head self-attention grows quadratically with the expansion of the input space size, while the computational complexity of the module of the present invention has a linear relationship with the space size, so that IG-MDSA can be more flexibly embedded into the CCAM module of the U-shaped network.

[0061] HaarDown module:

[0062] The present invention uses the HaarDown module to achieve lossless transformation. The HaarDown module consists of a Haar wavelet encoding module and a feature learning module, as Figure 3 shown. The wavelet encoding module can effectively reduce the spatial size of the feature while fully retaining all information.

[0063] The Haar wavelet transform is a widely recognized compact, binary, and orthogonal transform, which is widely used in fields such as image coding, edge extraction, and binary logic.

[0064] This means that a signal of a certain length can be divided into two parts, and the length of each part corresponds to the low-pass and high-pass decomposition filters respectively. Applying the Haar wavelet transform to a two-dimensional signal (such as a grayscale image) can obtain four components, and the spatial resolution of each component is half of the original signal.

[0065] Figure 3The Haar wavelet transform in it shows how to decompose an image with a resolution of H×W using the Haar wavelet transform. H0 and H1 represent the low-pass and high-pass decomposition filters respectively, and these filters are used to extract low-frequency and high-frequency information from the image. The symbol ↓2 represents downsampling of the approximation component and the detail component.

[0066] The Haar wavelet transform generates four components: an approximation (low-frequency) component (A), a horizontal detail component (H), a vertical detail component (V), and a diagonal detail component (D). The spatial resolution of each component is It is worth noting that the spatial resolution of each component is reduced to However, the number of channels of the feature map increases by a factor of four. This means that the Haar wavelet transform can successfully encode the information in part of the spatial dimension into the channel dimension without losing any information. Subsequent network layers can extract representative features with a lower spatial resolution from the transformed components.

[0067] The feature learning module includes a standard 1×1 convolutional layer, a batch normalization layer, and a ReLU activation layer for extracting discriminative features. This module has two purposes. One is to adjust the number of channels of the feature map to align with the subsequent layers. Assuming the size of the input feature map is H×W×C, the output size of the HaarDown module is H / 2×W / 2×C. The other is to filter redundant information as much as possible so that the subsequent layers can learn more representative features. The HaarDown module of the present invention is a plug-and-play module and can replace any downsampling method, such as max pooling or strided convolution.

[0068] DyUp module:

[0069] The DyUp module, as a lightweight and efficient upsampler, is also a key component of the refined reconstruction module. The upsampler generates the required set of point samples based on the information of the input feature to be sampled. Given a feature map χ of size C×H1×W1 and a set of samples of size 2×H2×W2 The feature map can be resampled based on the position information in the set of samples using the grid sampling function to obtain an upsampled feature map of size C×H2×W2. This process is as Figure 4 shown, and its mathematical expression is as follows:

[0070]

[0071] In the formula, grid_sample() is the point sampling function, such as bilinear interpolation, nearest neighbor upsampling, etc.

[0072] Given an upsampling scale factor (i.e., the upsampling multiple) and a feature map of size C×H×W DyUp generates offsets through a linear layer Its input and output channels are C and s respectively 2 The generated offset size is s 2 ×H×W. The offset is reshaped to a size of 2×sH×sW. By adding the offset to the original sampling grid the sample set is obtained Through grid sampling, an upsampled feature map of size C×sH×sW can be generated as shown in the above formula

[0073] The pre-lit map obtained in the pre-lit module is used as the input of the refinement reconstruction module. When the CCAM module in the refinement reconstruction module processes data, it combines the illumination feature map obtained in the pre-lit module. Finally, the enhanced image is obtained by adding the pre-lit map to the output obtained in the refinement reconstruction module

[0074] S4: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained low-light image enhancement model

[0075] The present invention constructs a simple and efficient loss function that combines information in the spatial domain and the frequency domain. The design of the spatial loss aims to ensure the stability of the gradient, which is crucial for preventing the gradient explosion problem and maintaining good convergence during training. In order to make the prediction result close to the true value while containing as few non-zero elements as possible, and the L1 loss tends to encourage sparse solutions. Therefore, the L1 loss is used as the spatial loss Its expression is as follows

[0076]

[0077] where, I (i) represents the i-th low-light image, I en(i) represents the i-th enhanced image, and n represents the number of low-light images

[0078] The Fourier loss is introduced to optimize the model in the frequency domain. By introducing spectral information, the Fourier loss provides richer features, enabling the model to more accurately evaluate the relative importance of each frequency component in the image and achieve faster convergence in the low-frequency region, thereby improving the training efficiency and overall performance. The Fourier loss function has comprehensive spectral features and can effectively distinguish high-frequency and low-frequency training samples. This enables the model to show higher accuracy when processing images of different frequencies. The Fourier loss can be expressed as the following formula

[0079]

[0080] where, denotes the Fourier transform of the i-th low-light image, denotes the Fourier transform of the i-th enhanced image.

[0081] The total loss function combines the spatial loss and the Fourier loss:

[0082]

[0083] where λ represents the weight used to balance the contributions of the two losses.

[0084] Adjust the model parameters according to the total loss of the model. When the loss function converges or reaches the maximum preset number of iterations, stop the training, save the model parameters, and obtain a trained low-light image enhancement model. Obtain the low-light image to be enhanced and input it into the trained low-light image enhancement model to obtain the enhanced image.

[0085] In summary, the present invention uses U-shaped Transformer as the backbone for high-resolution images, designs a linear self-attention structure for the LLIE task, which generates an implicitly encoded global context attention map for guidance by calculating the cross-covariance between channels. A downsampling module based on Haar wavelet transform is introduced, which has the advantage of multi-scale fast decomposition and can effectively retain high-frequency information during downsampling. At the feature upsampling stage, a super-lightweight and efficient dynamic upsampling module DyUp is introduced. DyUp redefines upsampling from the perspective of point sampling, significantly improves the upsampling performance, and effectively avoids artifacts that may occur in methods such as deconvolution upsampling. The present invention can extract more feature information while consuming less computational power, improving the accuracy of the enhanced image.

[0086] The above embodiments further illustrate in detail the purpose, technical solutions, and advantages of the present invention. It should be understood that the above embodiments are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made to the present invention within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A low-light image enhancement method based on the Retinex model, characterized in that Including: Obtain a low-light image to be enhanced, and input it into a trained low-light image enhancement model to obtain an enhanced image; The training process of the low-light image enhancement model includes: S1: Obtain a low-light image dataset, and calculate the illumination prior according to the low-light image; S2: Input the illumination prior and the low-light image into a pre-lighting module for processing to obtain a pre-lighting map and an illumination feature map; S3: Input the pre-lighting map and the illumination feature map into a refinement reconstruction module for processing to obtain an enhanced image; S4: Calculate the total model loss and adjust the model parameters according to the total model loss to obtain a trained low-light image enhancement model.

2. The low-light image enhancement method based on the Retinex model according to claim 1, wherein The data process of the pre-lighting module includes: The illumination prior and the low-light image are sequentially processed by a first convolutional layer, a depthwise separable convolutional layer, and a second convolutional layer to obtain an illumination mapping map and an illumination feature map; The illumination mapping map and the low-light image are multiplied element by element to obtain a pre-lighting map.

3. A low-light image enhancement method based on the Retinex model according to claim 1, characterized in that, The refinement reconstruction module is a U-shaped encoding-decoding structure, which connects the encoder and decoder features through skip connections, and includes a convolutional layer, a CCAM module, a HaarDown module, and a DyUp module.

4. A low-light image enhancement method based on the Retinex model according to claim 3, characterized in that The data processing process of the CCAM module includes: The input feature of the CCAM module is sequentially processed by an LN layer and an IG-MDSA layer to obtain a first intermediate feature; The first intermediate feature and the input feature are added element by element to obtain a second intermediate feature; The second intermediate feature is sequentially processed by an LN layer and an FFN layer to obtain a third intermediate feature; The second intermediate feature and the third intermediate feature are added element by element to obtain the output feature of the CCAM module.

5. A low-light image enhancement method based on the Retinex model according to claim 4, characterized in that The data processing process of the IG-MDSA layer includes: The first input feature and the second input feature are evenly divided in the channel dimension, and the number of divisions is equal to the number of self-attention heads; where the second input feature is the illumination feature map; In each attention head, perform self-attention calculation on different divided parts of the first input feature and the second input feature to obtain an attention feature; The attention feature is processed by a fully connected layer to obtain the output of the attention head; Fuse the outputs of multiple attention heads to obtain the output of the IG-MDSA layer.

6. A low-light image enhancement method based on the Retinex model according to claim 5, characterized in that, The formula for obtaining the attention value is expressed as: Among them, Attention(Q i , K i , V i , Y i ) represents the attention feature, and Y i represents the second input feature under the i-th attention head, Q i represents the query calculated from the first input feature under the i-th attention head, K i represents the key calculated from the first input feature under the i-th attention head, V i represents the value calculated from the first input feature of the i-th attention head, α i represents the learnable parameter of the i-th attention head, X i represents the first input feature of the i-th attention head; and represent three different 1×1 convolution weight matrices; and represent three different 3×3 depthwise separable convolution weight matrices.

7. A low-light image enhancement method based on the Retinex model according to claim 3, characterized in that, The HaarDown module is composed of a Haar wavelet encoding module and a feature learning module; Where The feature learning module includes a standard convolutional layer, a batch normalization layer, and a ReLU activation layer.

8. A low-light image enhancement method based on the Retinex model according to claim 1, characterized in that, The total model loss is the weighted sum of the spatial loss and the Fourier loss.

Images