A Low-Light Image Enhancement Method Based on Feature Fusion and Attention Embedding

Abstract

A low-light image enhancement method based on feature fusion and attention embedding, belonging to the field of computer image technology, is proposed. Using the Transformer Block as the basic unit, it comprises three modules: a feature extraction module, a U-Former module, and a detail restoration module. First, the feature extraction module utilizes three color spaces with different color gamuts to extract shallow features, thereby preserving rich color and detail information. Furthermore, a channel attention mechanism is introduced into the U-Former structure to compensate for the insufficient spatial dimension information interaction, prevent the loss of channel dimension information, and suppress noise amplification. Finally, to address the problem of insufficient model expressive power caused by the use of only the L1 loss function in most existing methods, four loss functions are selected and trained on the LOL dataset. Experimental results demonstrate that this method achieves state-of-the-art performance on various benchmark datasets.

Description

[0001] This invention belongs to the field of computer image technology, and specifically relates to a low-light image enhancement method based on feature fusion and attention embedding, which is a low-light image enhancement method based on color space fusion and channel attention embedding. Background Technology

[0002] Smartphones and cameras are widely used as primary sources of information, leading to an increasing number of digital images being captured in daily life and work. However, the quality of these images can be significantly affected by adverse environmental conditions, such as low light, noise, fog, or motion blur. Images taken in low light conditions suffer from low brightness, poor contrast, color distortion, and loss of local details, making it difficult to extract valuable information directly from them. Furthermore, these images and videos severely degrade the performance of advanced computer vision tasks, such as image recognition. [1] Target detection [2] Semantic segmentation [3] Therefore, low-light image enhancement has significant theoretical and practical implications, and an increasing number of researchers are actively participating in this field.

[0003] Low-light image enhancement techniques can be broadly categorized into traditional methods and deep learning methods. Traditional methods optimize models by leveraging existing prior theories and manually setting parameters, while deep learning methods adaptively learn the required parameters using the powerful expressive capabilities of deep neural networks. Comparatively, deep learning methods outperform traditional methods across various evaluation metrics, with deep convolutional neural networks (CNNs) being a prime example. [4] By leveraging its weight sharing and local connectivity, CNNs effectively reduce redundant computation in computer vision tasks, becoming the dominant network architecture in the field. Traditional convolutional operations only consider the information interaction between pixels within the convolutional kernel window, ignoring the correlation between windows. Although CNNs can still achieve a larger receptive field by stacking multiple convolutional layers, shallow features acquired in earlier layers may be lost as the network deepens. Recently, with the emergence of the Transformer architecture... [5]With success in Natural Language Processing (NLP), many visual Transformer architectures have been proposed to solve related computer vision tasks. Traditional Transformer block architectures consist of two parts: a self-attention mechanism and a feedforward network. The self-attention mechanism directly learns the dependencies of global information, overcoming the limitation of convolutional networks that only compute local information. The feedforward network, composed of multilayer perceptrons, projects the feature map into a high-dimensional space for processing through scaling factors, aiming to further refine and filter the output of the self-attention mechanism. Although the self-attention mechanism can be computed in parallel on GPUs using matrix multiplication, its computational complexity increases exponentially with increasing spatial image resolution. Considering that high-resolution images are more common in digital cameras, reducing computational complexity has become a research focus. Since the Transformer architecture can compute global attention better and outperforms most CNN-based methods in various evaluation metrics, our proposed CF-Former is also based on the Transformer architecture. However, existing Transformer-based low-light enhancement methods mainly focus on the impact of brightness on the image, rarely considering the changes in color before and after enhancement. Therefore, when RGB or RAW format images are directly input into a deep network, the resulting enhancements often suffer from color distortion. To address this issue, we propose a feature extraction module. This module first converts the RGB image to HSV and LAB color spaces, then uses a cascaded Transformer structure to extract shallow features and fuse the three color spaces. Experiments demonstrate that this module enables the network to generate more realistic and natural colors.

[0004] LLFormer proposed by Wang et al. [6] An axis-based multi-head self-attention (A-MSA) mechanism is adopted, which considers global information and achieves linear computational complexity. A-MSA calculates self-attention separately in the height and width dimensions of the feature map, fully considering the interaction between pixels in both dimensions. However, this method has potential drawbacks: (1) the mechanism cannot calculate the information correlation of the channel dimension of the feature map; (2) equal attention to the information of each channel can lead to information redundancy or loss of important data. To solve this problem, we introduce a channel attention mechanism (CAM). [7]This allows the network to adaptively focus on more important channel information and effectively suppresses noise generated during network forward propagation and continuous downsampling. Specifically, we added CAM after the third and fourth downsampling steps of the U-shaped Former structure; the choice of this location is based on theoretical predictions and experimental verification.

[0005] Low-light image enhancement (LLIE) aims to optimize low-light images captured in low-brightness and low-contrast environments, making the enhanced images more consistent with human visual perception and the true characteristics of images in natural scenes. However, most existing methods often suffer from color distortion, noise amplification, and loss of detail during the enhancement process. Summary of the Invention

[0006] Therefore, a low-light image enhancement method based on feature fusion and attention embedding is proposed. This method uses the Transformer Block as the basic unit and includes three modules: a Feature Extraction Module (FEM), a U-shaped Former module, and a Refinement Module (RM). First, the Feature Extraction Module uses three color spaces with different color gamuts to extract shallow features, thereby preserving rich color and detail information. Furthermore, a channel attention mechanism is introduced into the U-shaped Former structure to compensate for the lack of spatial dimension information interaction, prevent the loss of channel dimension information, and suppress noise amplification. Finally, to address the problem of insufficient model expressive power caused by the use of only the L1 loss function in most existing methods, four loss functions are selected and trained on the LOL dataset. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on various benchmark datasets.

[0007] Its advantages are:

[0008] Deep neural networks achieve better models by iteratively minimizing the loss function; therefore, the design and selection of the loss function play a crucial role in guiding the optimization of network parameters. Full-reference loss functions optimize the model by calculating the difference between the input image and the labeled image, while non-reference loss functions utilize specific theories or prior knowledge to constrain and guide the network optimization direction. Currently, most low-light image enhancement methods only use the L1 correlation loss function, which only calculates the difference between individual pixels and cannot perceive the image's structure and high-level semantic information. Therefore, we adopted four loss functions, including three full-reference loss functions and one non-reference loss function, considering both the similarity of individual pixels and high-level semantic information. Experiments demonstrate that the non-reference loss function can generate more natural enhancement results. Our main contributions are as follows:

[0009] (1) A low-light image enhancement method based on color space fusion and channel attention embedding is proposed. The method uses Transformer Block as the basic unit and includes three modules: Feature Extraction Module (FEM), U-shaped Former Module, and Refinement Module (RM).

[0010] (2) To address the color deviation problem in existing methods, a feature extraction module is proposed. Because low-light images have low brightness, they hide a lot of color information. How to accurately recover the true colors in the original image has always been an urgent problem to be solved in this field.

[0011] (3) Add a channel attention mechanism after downsampling in the U-Former. The axis self-attention mechanism reduces the amount of computation and fully integrates the spatial information of image height and width, but it loses more important brightness information in the channel dimension. After the third and fourth downsampling in the U-Former encoder, the channel attention mechanism is added to enable the backbone network to pay better attention to the channel information and suppress noise amplification.

[0012] (4) Four loss functions were used to guide the network in generating more realistic results. Although the Transformer-based method has a significant advantage in model expressiveness, a single loss function cannot accurately guide the direction of network parameter optimization. After analyzing and experimenting with various loss functions, we finally selected the SmoothL1 loss function, the structural similarity loss function, and the spatial consistency loss function. [8] And the perceptual loss function. Furthermore, the proposed method is applicable to LOL. [9] LRSW

[10] It has achieved state-of-the-art results on multiple benchmark datasets. Attached Figure Description

[0013] Figure 1 This is a diagram of the CF type UFormer structure.

[0014] Figure 2 This is a structural diagram of a multi-level fusion unit.

[0015] Figure 3 Compare single images from the LOL dataset.

[0016] Figure 4 Compare single images in the LIME dataset.

[0017] Figure 5 Comparison of methods for the MEF dataset.

[0018] Figure 6 for Figure 1 A magnified view of the left side.

[0019] Figure 7 for Figure 1 A magnified view of the central part.

[0020] Figure 8 for Figure 1 A magnified view of the right side. Detailed Implementation

[0021] 1.1 Traditional Methods

[0022] Traditional methods can be categorized into histogram equalization (HE), frequency domain transform-based, and model optimization-based methods. Histogram equalization algorithms, depending on the segmented region, primarily include global histogram equalization.

[11] Local histogram equalization

[12] Adaptive histogram equalization with limited contrast

[13] Adaptive Local Region Stretching Histogram Equalization

[14] Methods such as these are used. However, these methods ignore the positional relationships between pixels and do not consider the impact of image segmentation, leading to unrealistic enhancement results or checkerboard effects. Frequency domain transformation-based methods utilize two-dimensional Fourier transform or wavelet transform to transform the image to the frequency domain before filtering, such as those used by Zhang et al.

[15] Zhang et al.

[16] Han et al.

[17] Zaheeruddin et al.

[18] However, the enormous computational cost of Fourier transform limits the availability and real-time performance of the model.

[0023] Model optimization-based methods using Retinex theory

[18] Primarily, the prior theory of dark channels

[20] The illumination component is widely used in image dehazing, so we mainly introduce traditional methods based on Retinex theory. Retinex theory posits that an image can be decomposed into an illumination component and a reflection component. The illumination component reflects lighting information such as brightness, while the reflection component contains inherent object properties such as texture information. If these two components can be accurately separated, the influence of low illumination on the image can be removed. However, simultaneously estimating two unknown variables in an equation is an ill-posed problem. Therefore, methods based on Retinex theory utilize different prior assumptions to estimate the illumination component. Jobson et al.

[21] (Single Scale Retinex, SSR) uses a single-scale Gaussian blur kernel for convolution operations to simulate illumination components, which cannot represent diverse variations in illumination; Rahman et al.

[22] Multi-Scale Retinex (MSR) uses a multi-scale Gaussian kernel to address this issue, but the image will exhibit color distortion; Jobson et al. (Multi-Scale Retinex with Color Restoration, MSRCR)

[23] By introducing a color restoration factor, image colors are made more realistic, but this method requires manually setting a large number of parameters; Guo et al.

[24] The proposed LIME considers image fidelity and structural smoothness, transforming the problem of estimating illumination components into a convex optimization problem, which is then solved using the augmented Lagrange multiplier method; Fu et al.

[25] To balance improving quality and computational efficiency, the proposed SRIE introduces a weighted variational model to optimize the regularization term.

[0024] 1.2 Deep Learning-Based Methods

[0025] While traditional methods have achieved significant results, they rely too heavily on specific theoretical models or prior assumptions, making them unsuitable for complex real-world scenarios. In contrast, deep neural network models possess powerful nonlinear representation capabilities, enabling them to autonomously and accurately learn the latent features of the input data distribution.

[0026] Deep convolutional neural networks possess two inductive biases: locality and shift invariance, making them a mainstream model for computer vision tasks. Lore

[26] The Low Light Net (LLNet) proposed by Chen et al. uses a sparse denoising autoencoder as the backbone network. This model has good nonlinear expressive power, but the method of synthesizing datasets is too singular and cannot generalize to different real-world scenarios.

[27] Inspired by fully convolutional networks, U-net...

[28] As the default network architecture, although the original RAW format dataset contains more detailed information, it also increases the computational complexity of the model. (Shen et al.)

[29] The proposed MSRNet applies the traditional MSR algorithm to a deep convolutional neural network. This method selects 10,000 low-light images from multiple image datasets to generate colorful images, but banded shadows appear at the edges. (Wei et al.) [9] Introducing Residual Connections in RetinexNet

[30] Avoid gradient vanishing and use the BM3D algorithm

[31] The model denoises the reflection component and then reconstructs the image using Retinex theory. This method effectively improves image brightness, but artifacts appear in local image regions. (Zhang et al.)

[32] The proposed KinD model, based on Retinex theory, is designed as a decomposition module, a reflection component enhancement module, and an illumination adjustment module, with a specific loss function applied to each module. The deeper network results in more natural color and brightness in the enhanced image. However, this model cannot effectively handle the impact of uneven illumination on the image. (Hai et al.)

[10] The proposed R2R method utilizes Retinex theory to decompose the image, and then introduces a large number of learnable parameters into the denoising and enhancement modules to enhance the model's expressive power. However, this method has slow inference speed and color deviation in the enhancement results. Ma et al.

[33] The proposed SCI model uses a multi-layer cascaded structure to estimate illumination components. The authors introduce a self-calibration block in each module, requiring only the computational complexity of a single stage during inference. This model achieves an effective balance between computational efficiency and image quality. However, the overall brightness of the enhanced result is too dark, and the color information of the original image is lost. Jiang et al.

[34] The proposed EnlightenGAN was the first to use GAN networks for low-light image enhancement. This model takes a low-light image as input and uses a U-Net architecture with an attention mechanism as the backbone network of the generator, enhancing dark areas while preserving texture details. However, the images generated by this no-reference training method lack realism. (Guo et al.) [8] The proposed Zero-DCE views low-light image enhancement as a problem of learning a transformation function. By designing various no-reference loss functions to guide training, it achieves results similar to supervised training without using paired datasets. (Fu et al.)

[35] A data-driven approach was adopted, which adaptively learns latent priors from pairs of low-light images. However, as an unsupervised learning method, it cannot achieve the performance of supervised learning methods on full-reference evaluation metrics.

[0027] The Transformer architecture was first applied in the field of natural language processing. Its self-attention mechanism can not only learn long-distance dependencies in language sequences but also achieve parallelization of model computation. (Carion et al.)

[36] The proposed DETR was the first to introduce the Transformer into object detection tasks, achieving the same accuracy and runtime as Faster R-CNN.

[37] Similar results. Dosovitskiy et al.

[38] The proposed ViT transfers the original Transformer to classification tasks, outperforming CNN-based methods when trained on larger datasets. (Chen et al.)

[39] The proposed IPT was the first to apply the Transformer to low-level computer tasks, achieving breakthroughs in denoising, deraining, and super-resolution tasks. Considering the quadratic computational complexity of self-attention mechanisms, which makes them difficult to apply to high-resolution images, Wang et al.

[40] The proposed Uformer reduces computational complexity by utilizing the LeWinTransformer Block to compute self-attention only within the window, making it efficient for low-level computer tasks. Similarly, to achieve linear computational complexity, Zamir et al....

[41] The self-attention mechanism in the Restormer method employs depthwise separable convolution operations, and the feedforward network uses a gating mechanism, achieving excellent results in various image restoration tasks.

[0028] 2. Methods

[0029] The goal of low-light image enhancement is to enhance low-light images acquired in underexposed environments (e.g., nighttime, indoors, backlit) to achieve results consistent with human visual perception (e.g., high brightness, strong contrast, sharp edges, and color fidelity). To address the common problems of color deviation, noise amplification, and detail loss in existing methods, we propose CF-UFormer.

[0030] 2.1 Overall Framework

[0031] like Figure 1As shown, CF-UFormer is an end-to-end Transformer-based deep autoencoder architecture, mainly consisting of a feature extraction module, a multi-level fusion unit (MFU), and a U-Former structure. Specifically, the low-light image serves as the network input. First, it passes through three branches to convert the original image to RGB, HSV, and LAB color spaces, thus obtaining information from three color spaces, including RGB. Then, cascaded Transformer Blocks fuse these three information sources to obtain T0, T1, and T2. To prevent information loss, the three information sources are input separately to the multi-level fusion unit (MFU). The MFU uses skip-layer connections to prevent image detail loss while achieving multi-input cross-layer information fusion. The fused information is then concatenated channel-wise and input into the U-former Former structure. The encoder part of the U-former Former structure consists of four alternating downsampling and Transformer Block operations, while the decoder part consists of four alternating upsampling and three Transformer Block operations. This process fully fuses the multi-scale features of the image. It's important to note that the channel dimension of the feature map increases continuously after multiple downsampling iterations. Since the axis-based attention mechanism only interacts with information in the image's height and width dimensions, a channel attention mechanism is introduced after the third and fourth downsampling iterations to allow the network to adaptively focus on dimensions with more important information. This suppresses noise amplification that occurs during continuous downsampling. Finally, the output of the U-shaped Former structure is added to F1 and input into the detail recovery module. This module still employs a cascaded Transformer structure with skip connections and multi-level fusion units to prevent detail loss in the final stage of the network.

[0032] 2.2 Feature Extraction Module

[0033] 2.2.1 Limitations of the RGB color space

[0034] The RGB color space uses a linear combination of red, green, and blue color components to represent colors. However, when the image color changes, the magnitude of the change in each component is not uniform, making it impossible to accurately estimate the change in each component. Therefore, the RGB color space is not suitable for direct application to image processing tasks. Images acquired in natural environments are easily affected by natural lighting, occlusion, and shadows. To eliminate these effects, the separation of luminance information is crucial, but the RGB color space does not have a separately separated luminance component.

[0035] Unlike the RGB color space, the HSV color space includes lightness, hue, and saturation channels, separating chroma and lightness information. This allows for targeted processing of low-light images, and the separated hue channel can guide models to generate more realistic images. The LAB color space has a wider color gamut, greatly preserving image information after conversion. It also separates the lightness channel, making it effectively suitable for low-light image enhancement.

[0036] 2.2.2 Feature Extraction Module (FEM)

[0037] To address the color distortion issue in existing methods, a feature extraction module is proposed. This module processes the image in different color spaces to achieve more realistic color enhancement. The feature extraction module comprises three parallel branches, each converting the original image to RGB, HSV, and LAB color spaces respectively. Considering that low-light images are significantly affected by illumination intensity, the most frequently used RGB color space, along with the HSV and LAB color spaces containing a luminance channel, are selected as the processing targets.

[0038] Figure 1 The structural details of each branch are shown, where F represents the color space transformation function. The subsequent three Transformer Blocks with skip connections perform preliminary processing on the transformed information. This is done to fully utilize the powerful global modeling capabilities of the Transformer to obtain shallow features from each color space. To avoid loss of image details in the early stages, a Multi-Level Fusion Unit (MFU) is used to fuse the features of the outputs of the three Transformer Blocks. Because the MFU module does not perform downsampling on the image, it only achieves accurate color space and shallow feature extraction, thus avoiding the loss of image details.

[0039] 2.3 U-shaped Former

[0040] 2.3.1 U-shaped Former Structure

[0041] U-Net is a classic model for semantic segmentation using fully convolutional networks. Its name comes from its U-shaped architecture, and it has significantly influenced the design of numerous pixel-intensive tasks, including semantic segmentation. Based on convolutional neural networks, U-Net fully utilizes neighborhood information of pixels. Furthermore, continuous downsampling combined with skip connections enables information preservation while achieving multi-scale image fusion.

[0042] Given the advantages of the U-Net network, our proposed backbone network also adopts a U-shaped design, but replaces convolutional operations with Transformer Blocks, which greatly enhances the model's expressive power. Specifically, the result F0 after fusing three color spaces is used as the input to the U-shaped Former. The encoder part uses downsampling and Transformer Blocks alternately connected four times to obtain a feature map with a resolution reduced by a factor of 16. Each downsampling reduces the image spatial resolution by a factor of 2 and increases the channel dimension by a factor of 2. The number of Transformer Blocks in each stage is {2, 4, 8, 16}, and the number of attention heads is {1, 2, 4, 8}. After the third and fourth downsampling operations, the feature map dimension increases by a factor of 8 and 16, respectively. The semantic information contained in each channel differs significantly, and the network may ignore some important information during forward propagation. We introduce a channel attention mechanism to address this issue. The decoder operates in the opposite manner to the encoder, consisting of four upsampling operations and three stages. Each upsampling increases the image spatial resolution by a factor of two and reduces the number of channel dimensions by a factor of two. The number of Transformer Blocks in each stage is {2, 4, 8}, and the number of attention heads is {1, 2, 4}.

[0043] The channel attention mechanism projects the feature map into a high-dimensional space for processing and then maps it back to its original size to obtain an attention map. This attention map has the same dimensions as the input feature map, and it is used as the attention weights, multiplied element-wise with the feature map to obtain the final result. This process enables the network to adaptively learn attention to different dimensions, which will help the network pay more attention to brightness-related information, thus obtaining more realistic results.

[0044] 2.3.2 Multilevel Fusion Unit (MFU)

[0045] When obtaining feature information from different stages from each branch, pixel-level addition or multiplication is a common fusion approach. However, such simple processing cannot fully fuse multi-path information and may even cause irreversible loss of detail. To address this issue, multi-level fusion units (MFUs) are used to preserve and fuse different information. For example... Figure 2As shown, the input from N stages undergoes a reshape operation followed by self-attention computation, similar to the operation in the Transformer Block. The multi-level fusion unit only fuses the branch dimensions, without destroying the height and width information of each branch. The MFU in the feature extraction module aims to fuse the low-level features extracted by the three Transformer Blocks, ensuring better information preservation of the feature map in each color space before it is input into the U-shaped Former structure. The MFU in the detail recovery module further processes the output of the U-shaped Former structure to recover the detail information lost due to downsampling. Experiments demonstrate that this fusion mechanism is significantly superior to simple pixel addition or multiplication schemes.

[0046] 2.4 Transformer Block

[0047] The Transformer Block is the foundation of the Transformer structure and its biggest difference from convolutional neural networks. The following sections will introduce the axis self-attention mechanism and dual-gated feedforward network used in this technical solution. The improved Transformer Block proposed in this technical solution can be represented as:

[0048]

[0049] Where F in For the input of TransformerBlock, F' represents the output of the axis self-attention mechanism with skip connections, LN(.) represents the layer normalization operation, A-MSA(.) represents the axis self-attention mechanism, and DGFN(.) represents the dual-gated feedforward network.

[0050] 2.4.1 Axial Self-Attention Mechanism (A-MSA)

[0051] Traditional visual Transformers divide images into blocks, treating each block as a single word in a language model and calculating the correlation between each block and other blocks. However, when the input is a high-resolution image, the number of blocks increases quadratically with the image's height and width, significantly increasing the computational cost of the self-attention mechanism. To effectively reduce computational complexity, several efficient Transformer methods have been proposed, such as recursive connections.

[42] Sparse self-attention

[43] Low-rank decomposition

[44] and linear self-attention

[45] The linear self-attention mechanism achieves linear computational complexity by changing the order of matrix multiplication.

[0052] This technical solution employs an axis-based self-attention mechanism. Compared to the original ViT method, which performs self-attention calculations across all pixels in the image, this method only calculates along the height and width directions, significantly reducing the computational load of the attention mechanism. The reason why calculating only along two axes remains effective for image enhancement tasks is twofold: 1. The two chosen directions are perpendicular to each other, allowing us to obtain global dependencies between pixels after one round of calculation. 2. The pixel neighborhood information in natural images is highly similar; we utilize this prior knowledge to eliminate a large amount of redundant computation.

[0053] While axial attention plays a crucial role in multi-scale information interaction in images, it lacks specific improvements in the channel dimension. Converting images to HSV and LAB color spaces alters the meaning of channels; the luminance channel significantly impacts low-light images. Therefore, channel dimension information reflects the influence of different luminance levels on image features. Initially, luminance features are not effectively separated, making self-attention calculation in the channel dimension unsuitable. However, after U-shaped Form downsampling by 8x and 16x, different features are separated into more channels, and small-scale features are sufficiently learned and separated. Introducing a channel attention mechanism at this point compensates for the lack of spatial dimension information interaction in the axial attention mechanism.

[0054] The multi-axis self-attention component includes cascaded operations on height, width, and channel dimensions. Figure 4 This demonstrates the case where computation is performed using the channel dimension as the principal axis, with the same operation applied to other dimensions. First, the input feature map is mapped to Q, K, and V through 1×1 convolutions and depthwise separable convolutions in three branches. The depthwise separable convolution operation better focuses on local image information, and combined with the self-attention mechanism's ability to compute global information dependencies, this structure can simultaneously consider both global and local information. Next, Q and K are reshaped and multiplied to obtain the attention weight matrix. This attention matrix has the same height and width as the original feature map's channel number. Finally, this attention weight matrix is ​​multiplied by V along the channel dimension to obtain an output of the same size as the original feature map. 2.4.2 Dual-Gated Feedforward Network (DGFN)

[0055] Gating mechanisms have been proven effective for filtering and transforming input data. Restormer achieved good results by introducing this mechanism into a feedforward network. However, unidirectional gating inevitably filters out useful information during training. To address this issue, LLFomer proposed a dual-gated feedforward network, which performs secondary filtering of information and fully utilizes the results of the self-attention mechanism. This technical solution employs the same operation as LLFomer.

[0056] Specifically, firstly, depthwise separable convolutions and GELU are applied and multiplied on two parallel paths. This operation filters out features with less information. The reason for using two paths is to avoid losing effective information in a single filtering. Next, the outputs of the two branches are summed to fuse the filtered information. Finally, a 1×1 convolution operation combined with residual connections ensures the integrity of the information.

[0057] 2.5 Loss Function

[0058] To achieve higher PSNR values, most supervised learning methods employ the L1 correlation loss function. However, the L1 loss function only considers the Euclidean distance between individual pixels in an image, failing to measure the structural information and high-level semantic similarity between two images, resulting in a lack of realism in the generated images. To address this issue, a comprehensive weighted loss function is proposed. Three full-reference loss functions guide the network to generate results that are more similar to the labels in terms of single pixels and structure, while a non-reference loss function utilizes prior knowledge to optimize and adjust the image, thereby generating results that are more natural and closer to human visual perception. This loss function consists of the SmoothL1 loss function, the structural similarity loss function, the spatial consistency loss function, and the perceptual loss function.

[0059] 2.5.1 SmoothL1 Loss

[0060] SmoothL1 loss is an improved version of the L1 loss function. SmoothL1 uses a smoother L2 loss for regions with smaller parameter values, resulting in smaller gradients in these areas; while for regions with larger parameters, it still uses the L1 loss to avoid gradient explosion. The SmoothL1 loss function is expressed as follows:

[0061]

[0062] Where a is the balance coefficient and x is the model parameter. L SL1 The SmoothL1 loss function accurately calculates the similarity between individual pixels in an image. It combines the advantages of L1 and L2 loss functions and dominates the overall loss function.

[0063] 2.5.2 Structural Similarity Loss

[0064] The similarity between two images is not only reflected in the differences between individual pixels, but also in the brightness, contrast, and structural information between multiple pixels, which better aligns with the human eye's judgment of image similarity. Based on this consideration, a structural similarity loss function L is introduced. ssim :

[0065]

[0066] in, x i and y i Let represent the i-th pixel in the test image and the label image, respectively, and N represent the number of pixels in the image. Constants C1 and C2 are added to prevent the denominator from being zero in the formula. The mean μ in the formula measures the global brightness of the image, while the variance σ reflects the difference between different pixel values ​​and can measure contrast information in the visual domain. Covariance σ xy Measure the pixel difference between the test image and the label image; L ssim This represents the structural similarity loss function. Therefore, the introduction of the structural similarity loss function will make the generated images appear more visually similar.

[0067] 2.5.3 Spatial Consistency Loss

[0068] In everyday natural images or scenes perceived by the human eye, the neighborhood information of pixels is highly similar. This is a key reason for the success of deep convolutional neural networks. Based on this prior knowledge, a spatial consistency loss function is introduced. The spatial consistency loss function L... spa It does not require a label image as a reference, and the generated image is smooth in small local areas, which avoids the checkerboard effect and artifacts that occur in most methods.

[0069]

[0070] Where K is the number of local regions, Ω(i) are the four adjacent regions centered at pixel i, j represents the j-th pixel within Ω(i), Y and I represent the enhanced image and the low-light image, respectively, and L spa This represents the calculated result of the spatial consistency loss function. This no-reference loss function guides the network to generate smoother images and effectively reduces abrupt noise.

[0071] 2.5.4 Perceived Loss

[0072] The three loss functions mentioned above all measure low-level semantic similarity. However, humans can quickly extract features from multiple images and use them to determine similarity. In this technical solution, a pre-trained VGG model is used to extract high-level semantic features from low-light images and enhanced images respectively to calculate the similarity L between them. per :

[0073]

[0074] Where j represents the j-th layer of the network, φ(.) represents the VGG pre-trained network, and C j H j W j L represents the size of the feature map at layer j. perThis represents the perceptual loss function.

[0075] 2.5.5 Total Loss

[0076] The method in this technical solution uses the weighted sum of the above four loss functions as the total loss function L. total Its expression is:

[0077] L total =L SL1 +αL ssim +βL spa +γL per (6)

[0078] Where α, β, and γ are balance coefficients, all of which are taken as 0.1 in this experiment.

[0079] 3. Experiment

[0080] 3.1 Implementation Details

[0081] CF-UFormer was trained for 1000 epochs on the League of Legends benchmark dataset using an RTX 3090 GPU with a batch size of 8. Low-light images were randomly cropped into 128×128 image patches and used as model input, with horizontal and vertical flipping used for data augmentation. The Adam optimizer was employed with an initial learning rate of 10⁻⁴, which was subsequently decayed to 10⁻⁶ using cosine annealing. The number of Transformer Blocks in the four stages of the U-Net encoder were {2, 4, 8, 16}, and the number of self-attention heads were {1, 2, 4, 8}, respectively; the number of Transformer Blocks in the three stages of the decoder were {2, 4, 8}, and the number of self-attention heads were {1, 2, 4}, respectively.

[0082] 3.2 Method Comparison

[0083] 3.2.1 Subjective Evaluation

[0084] Figure 3 This is a demonstration of single-image enhancement results from the League of Legends (LOL) dataset, comparing eight classic deep learning-based low-light image enhancement methods. (Observation) Figure 3 The red clothes in the picture reveal KinD

[32] EnlightenGAN

[34] R2R

[10] and Uformer

[40] Both exhibited significant color deviations, which will have a substantial impact on subsequent advanced computer vision tasks; ZeroDCE [8] and SCI

[33] The enhancement effect is poor, with low overall image brightness and slight color deviation; RetinexNet [9]With LLFomer [6] While achieving good color fidelity, it contains significant noise and exhibits localized blurring. A checkerboard effect is observed in the LLFomer image when the wall area is magnified. Observing the coat hanger portion of the image, other methods all suffer from edge blurring or amplified noise, while the proposed CF-UFormer preserves the original image's color information while avoiding detail loss.

[0085] To demonstrate the generalization ability of CF-UFormer, tests were conducted on the LIME and MEF benchmark datasets. Figure 4 The results of single-image augmentation on randomly selected LIME images are shown, and eight deep learning-based augmentation methods are compared. EnlightenGAN is one such example.

[34] Significant color deviation still exists; KinD

[32] With RetinexNet [9] Large areas of artifacts appear, and a significant amount of edge information is lost, making the image look unrealistic; R2R

[10] The enhancement effect is the weakest, the overall image brightness is dark, and the model does not restore the detail information in dark areas; ZeroDCE [8] and SCI

[33] Severe noise appears, making areas of the same color look uneven and affecting the visual experience; Uformer

[40] With LLFomer [6] The image is divided into blocks and inter-pixel self-attention is calculated, but the influence of edge neighborhood information on color is not considered, resulting in a noticeable cardioid effect in the image. By observing magnified local images, CF-UFormer successfully suppresses the amplification of noise while preserving sufficient detail.

[0086] like Figure 5 As shown, six images were randomly selected from the MEF dataset for testing. Through comparison, the problems existing in different current methods were summarized. (RetinexNet) [9] Overly thickening the outlines of objects in the image results in unrealistic generated results; EnlightenGAN

[34] There is a significant color deviation, especially with red and blue appearing to be interchanged; ZeroDCE [8] R2R

[10] and SCI

[33] The enhancement capabilities are weak; observing the wall area in the image, all three methods exhibit black shadows; Uformer

[40] With LLFomer [6] A noticeable checkerboard effect (block artifacts) appears in smooth areas of the image. CF-UFormer's powerful low-light enhancement capabilities produce more realistic results.

[0087] 3.2.2 Objective Evaluation

[0088] To ensure the fairness of the comparison method, in LOL... [9] Training on benchmark datasets and in LOL and LSRW

[10] For testing on the datasets, the LOL dataset contains 500 image pairs (485 training images and 15 test images), and the LSRW dataset contains images taken with Huawei phones (30 test images) and Nikon cameras (20 test images) as subsets. We use four objective evaluation metrics: PSNR, SSIM, NIQE, and LPIPS, aiming to evaluate both pixel-level similarity and high-level semantic similarity, and to measure the merits of various methods from different perspectives. To characterize the model's generalization ability, we randomly selected 75 test images from the DICM (20 images), ExDark (20 images), LIME (10 images), MEF (15 images), and VV (10 images) test datasets. These datasets do not have corresponding labeled images, and we use the unreferenced evaluation metric NIQE as the objective evaluation metric.

[0089] Table 1 compares the test results of 6 traditional methods and 8 deep learning-based methods on the LOL dataset. Experiments demonstrate that our method achieves the best results in all four objective evaluation metrics. Table 2 shows the test results of these methods on the LSRW dataset. Table 3 presents the NIQE evaluation results from different benchmark datasets. CF-Former achieves the best results on the DICM and MEF datasets, the second best results on VV, and the best results among the average values ​​of the five datasets.

[0090] Table 1 compares the results on the LOL dataset.

[0091]

[0092]

[0093] Table 2 compares the results on the LSRW dataset.

[0094]

[0095] Table 3 shows the test results on five benchmark datasets.

[0096] Model DICM LIME MEF VV ExDark mean Input 3.729 4.248 4.818 4.265 4.264 4.704 2018-RetinexNet-BMVC 6.101 6.200 5.847 3.476 3.476 5.020 2019-KinD-ACMMM 3.749 4.982 3.905 2.634 3.725 3.799 2020-ZeroDCE-CVPR 3.724 4.665 3.992 3.162 4.510 4.011 2021-EnlightGAN-TIP 3.570 4.966 3.869 3.839 4.599 4.169 2022-SCI-CVPR 4.812 5.090 4.511 3.188 4.767 4.474 2022-Uformer-CVPR 3.701 4.194 3.913 3.731 3.691 3.846 2022-LLFormer-AAAI 3.853 4.478 3.920 4.291 4.519 4.212 2023-R2R 4.525 3.797 4.322 3.410 4.238 4.058 2023-PairLIECVPR 3.929 4.982 4.480 3.541 5.396 4.446 CF-Former 3.297 4.335 3.843 3.137 3.747 3.672

[0097] 3.3 Ablation Experiment

[0098] To verify the effectiveness of the submodule, ablation experiments were conducted on the color space, loss function, and channel attention mechanism.

[0099] 3.3.1 Color Space

[0100] Most low-light image enhancement methods only process images in the RGB color space and do not consider the HSV and LAB color spaces with luminance channels. Experiments have shown that using multiple color spaces can improve color fidelity to some extent. Table 4 shows the impact of not using the HSV and LAB color spaces on model performance. It can be found that the loss of a certain color space leads to a decrease in all objective evaluation indicators of the model.

[0101] Table 4. The impact of different color spaces on model performance

[0102]

[0103] 3.3.2 Loss Function

[0104] To verify that the various loss functions used in this technical solution improve model performance, ablation experiments were conducted on each loss function, and the results are shown in Table 5.

[0105] Table 5. Impact of different loss functions on model performance

[0106]

[0107] 3.3.3 Channel Attention Mechanism

[0108] To compensate for the Transformer Block's inability to adaptively focus on channel dimensions, a channel attention mechanism was introduced. However, when the number of channel dimensions is small, introducing CA leads to a significant loss of information. Experiments showed that placing the CA module after the third and fourth downsampling steps of the U-shaped Former module yielded the best results, as the number of channel dimensions was large at this point, resulting in better separation of feature information. Table 6 illustrates the impact of adding the channel attention module at different positions in the network on model performance.

[0109] Table 6. Impact of different channel attention mechanisms on model performance

[0110] Model PSNR↑ SSIM↑ LPIPS↓ NIQE↓ Remove CA 21.713 0.848 0.087 3.708 Outside U-Former 22.932 0.849 0.082 3.592 CF-Former 23.671 0.847 0.079 3.503

[0111] 4. Conclusion

[0112] This technical solution proposes a low-light image enhancement method called CF-UFormer. This method is based on the Transformer architecture and includes a Feature Extraction Module (FEM), a U-Former, and a Multi-Level Fusion Unit (MFU). It fuses color space information from RGB, HSV, and LAB and embeds a channel attention mechanism. Addressing the low brightness characteristic of low-light images, CF-UFormer separates brightness information into different channels for targeted processing. This avoids generating overexposed or underexposed images and effectively enhances color fidelity while avoiding detail loss caused by downsampling. Furthermore, traditional Transformer Blocks do not consider channel-level information interaction, while CF-UFormer introduces a channel attention mechanism after downsampling, guiding the network to focus on more important channel information and suppressing noise amplification. Although this method is trained in a supervised manner on the LOL dataset, we introduce four loss functions to guide network training, including three full-reference loss functions and one no-reference loss function. Therefore, CF-UFormer achieves state-of-the-art performance on multiple benchmark datasets, demonstrating strong generalization ability.

[0113] 5 References

[0114] [1] Krizhevsky A, Sutskever I, Hinton G E. Image classification on Imagenet dataset using deep convolutional neural networks [J]. Communications of the ACM, 2017, 60(6): 84-90.

[0115] [2] Liu W, Anguelov D, Erhan D, et al. Single-shot multi-box object detection [C] / / ComputerVision–ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October11–14,2016,Proceedings,Part I 14.Springer International Publishing,2016:21-37.

[0116] [3] Badrinalayanan V, Kendall A, Cipolla R. A deep convolutional autoencoder structure for image segmentation [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.

[0117] [4] LeCun Y, Boser B, Denker JS, et al. Backpropagation technique applied to handwritten postal code recognition [J]. Neural computation, 1989, 1(4): 541-551.

[0118] [5] Vaswani A, Shazeer N, Parmar N, et al. All you need is an attention mechanism [J]. Advances in neural information processing systems, 2017, 30.

[0119] [6]Wang T,Zhang K,Shen T,et al. Ultra-high resolution low-light image enhancement: a benchmark dataset and a Transformer approach [C] / / Proceedings of the AAAI Conference on Artificial Intelligence.2023,37(3):2654-2662.

[0120] [7] Hu J, Shen L, Sun G. Compression and excitation networks [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018:7132-7141.

[0121] [8] Guo C, Li C, Guo J, et al. Zero-reference depth curve estimation for low-light image enhancement techniques [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:1780-1789.

[0122] [9]Wei C,Wang W,Yang W,et al. Depth Retinex decomposition for low-light image enhancement[J].arXiv preprint arXiv:1808.04560,2018.

[0123]

[10] Hai J, Xuan Z, Yang R, et al. Low-light image enhancement from low-light to normal-light images [J]. Journal of Visual Communication and Image Representation, 2023, 90: 103712.

[0124]

[11] Cheng HD,Shi X J. A simple and effective image enhancement method based on histogram equalization[J].Digital Signal Processing,2004,14(2):158-170.

[0125]

[12] Stark, J, Alex. An adaptive image contrast enhancement method using histogram equalization [J]. IEEE Transactions on Image Processing, 2000, 9(5): 889-889.

[0126]

[13] Reza A M. Implementation of real-time image enhancement based on contrast-limited adaptive histogram equalization [J]. Journal of VLSI signal processing systems for signal, image and video technology, 2004, 38: 35-44.

[0127]

[14] Srinivasan S, Balram N. Adaptive contrast enhancement algorithm using local stretching [C] / / Proceedings of the 9th Asian symposium on information display. 2006:152-155.

[0128]

[15] Zhang Y, Xie M. Color image enhancement algorithm based on HIS color space and local homomorphic filtering [J]. Computer Applications and Software, 2013, 30(12): 303-307.

[0129]

[16] Zhang C, Liu W, Xing W. Color image enhancement based on local spatial homomorphic filtering and gradient domain variance-guided image filtering [J]. Journal of Electronic Imaging, 2018, 27(6): 063026-063026.

[0130]

[17] Han L,Xiong J,Geng G,et al. HSV spatial true color image enhancement algorithm based on two-channel homomorphic filtering[J].Computer Engineering and Applications,2009,45(27):18-20.

[0131]

[18] Zaheeruddin S, Suganthi K. A method for enhancing fuzzy parameter transformation based on homomorphic filtering [J]. Procedia Computer Science, 2019, 165: 166-172.

[0132]

[19] Land EH, McCann JJ. Luminance and Retinex Theory [J]. Josa, 1971, 61(1): 1-11.

[0133]

[20] He K, Sun J, Tang X. Single image dehazing method based on dark channel prior [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(12): 2341-2353.

[0134]

[21] Jobson DJ, Rahman Z, Woodell G A. Characteristics and performance of the center-surround Retinex algorithm [J]. IEEE transactions on image processing, 1997, 6(3): 451-462.

[0135]

[22] Rahman Z, Jobson DJ, Woodell G A. Multiscale Retinex algorithm for color image enhancement [C] / / Proceedings of 3rd IEEE international conference on image processing.IEEE, 1996, 3: 1003-1006.

[0136]

[23] Jobson DJ, Rahman Z, Woodell GA. A multiscale Retinex method to bridge the gap between color images and human scene observations [J]. IEEE Transactions on Image Processing, 1997, 6(7): 965-976.

[0137]

[24] Guo X, Li Y, Ling H. Low-light image enhancement by estimating illumination maps, 2016, 26(2):982-993.

[0138]

[25] Fu X, Zeng D, Huang Y, et al. A weighted variational model for reflectivity and illumination pattern estimation [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:2782-2790.

[0139]

[26] Lore KG, Akintayo A, Sarkar S. Natural low-light image enhancement method based on deep autoencoder [J]. Pattern Recognition, 2017, 61: 650-662.

[0140]

[27] Chen C,Chen Q,Xu J,et al. Learning in the dark [C] / / Proceedings of the IEEEconference on computer vision and pattern recognition.2018:3291-3300.

[0141]

[28] Ronneberger O, Fischer P, Brox T. Convolutional Networks for Biomedical Image Segmentation [C] / / Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015:18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer International Publishing, 2015:234-241.

[0142]

[29] Shen L, Yue Z, Feng F, et al. Low-light image enhancement algorithm based on deep convolutional network [J]. arXiv preprint arXiv:1711.02488,2017.

[0143]

[30] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778.

[0144]

[31] Dabov K, Foi A, Katkovnik V, et al. Image denoising algorithm based on sparse 3D transform domain cooperative filtering [J]. IEEE Transactions on image processing, 2007, 16(8): 2080-2095.

[0145]

[32] Zhang Y, Zhang J, Guo X. Igniting the darkness: A practical low-light image enhancement model [C] / / Proceedings of the 27th ACM international conference on multimedia.2019:1632-1640.

[0146]

[33] Ma L, Ma T, Liu R, et al. A fast, flexible and robust low-light image enhancement algorithm [C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and PatternRecognition.2022:5637-5646.

[0147]

[34] Jiang Y,Gong X,Liu D,et al. Unsupervised deep low-light image enhancement algorithm[J].IEEE transactions on image processing,2021,30:2340-2349.

[0148]

[35] Fu Z, Yang Y, Tu X, et al. Learning lightweight enhancement models from paired low-light images [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:22252-22261.

[0149]

[36] Carion N,Massa F,Synnaeve G,et al. End-to-end object detection algorithm using transformer [C] / / Computer Vision–ECCV 2020:16th European Conference,Glasgow,UK,August23–28,2020,Proceedings,Part I 16.Springer International Publishing,2020:213-229.

[0150]

[37] Ren S, He K, Girshick R, et al. Real-time object detection using region proposal networks [J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39(6): 1137-1149.

[0151]

[38] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is 1600 word vectors: Transformer algorithm for large-scale image recognition [J]. arXiv preprint arXiv:2010.11929,2020.

[0152]

[39] Chen H,Wang Y,Guo T,et al. Pre-trained Transformer models for image processing[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:12299-12310.

[0153]

[40] Wang Z, Cun X, Bao J, et al. A general U-shaped Transformer network for image restoration [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:17683-17693.

[0154]

[41] Zamir SW, Arora A, Khan S, et al. Efficient Transformer algorithm for high-resolution image restoration [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:5728-5739.

[0155]

[42] Dai Z, Yang Z, Yang Y, et al. Attention-based language models beyond fixed-length contexts [J]. arXiv preprint arXiv:1901.02860,2019.

[0156]

[43] Child R, Gray S, Radford A, et al. Generation of long sequences based on sparse Transformer [J]. arXiv preprint arXiv:1904.10509,2019.

[0157]

[44] Wang S, Li BZ, Khabsa M, et al. Self-attention mechanism with linear complexity [J]. arXiv preprint arXiv:2006.04768,2020.

[0158]

[45] Katharopoulos A, Vyas A, Pappas N, et al. Fast autoregressive Transformer algorithm with linear attention [C] / / International Conference on Machine Learning.PMLR,2020:5156-5165.

Claims

1. A low-light image enhancement method based on feature fusion and attention embedding, characterized in that... The method comprises the following steps: It consists of three modules: a feature extraction module, a U-shaped Former module, and a detail restoration module. First, the feature extraction module utilizes three color spaces with different color gamuts to extract shallow features, thereby preserving rich color and detail information. Furthermore, a channel attention mechanism is introduced into the U-shaped Former structure to compensate for insufficient spatial dimension information interaction, prevent the loss of channel dimension information, and suppress noise amplification. Finally, to address the problem of insufficient model expressive power caused by using only the L1 loss function in most existing methods, four loss functions are selected and trained on the LOL dataset. Loss functions include SmoothL1 loss function, structural similarity loss function, spatial consistency loss function, and perceptual loss function; SmoothL1 loss is an improved version of the L1 loss function. The expression for SmoothL1 is: Where a is the balance coefficient, x is the model parameter; L SL1 The result is the calculation result of the SmoothL1 loss function, which accurately calculates the similarity between single pixels in an image. It combines the advantages of L1 and L2 loss functions and dominates the overall loss function. Structural similarity loss function L ssim : in, x i and y i Let represent the i-th pixel of the image under test and the label image, respectively; N represents the number of pixels in the image; constants C1 and C2 are added to prevent the denominator from being zero in the formula; the mean μ in the formula can measure the global brightness of the image, and the variance σ reflects the difference between different pixel values; in the field of vision, it can measure the contrast information of a single image; covariance σ xy Measure the pixel difference between the test image and the label image; L ssim This represents the structural similarity loss function; therefore, introducing the structural similarity loss function will make the generated images visually more similar. Spatial consistency loss function L spa No label image is needed as a reference; the generated image has smooth surfaces in small local areas. Where K is the number of local regions, Ω(i) are the four adjacent regions centered at pixel i, j represents the j-th pixel within Ω(i), Y and I represent the enhanced image and the low-light image, respectively, and L spa This represents the calculation result of the spatial consistency loss function; A pre-trained VGG model was used to extract high-level semantic features from the low-light image and the enhanced image, respectively, to calculate the similarity L between them. per : Where j represents the j-th layer of the network, φ(.) represents the VGG pre-trained network, and C j H j W j L represents the size of the feature map at layer j. per This represents the calculation result of the perceptual loss function; This method uses the weighted sum of the above four loss functions as the total loss function L. total Its expression is: L total =L SL1 +αL ssim +βL spa +γL per (6) Where α, β, and γ are balance coefficients.

2. The low-light image enhancement method based on feature fusion and attention embedding according to claim 1, characterized in that... Includes the following steps: The specific steps are as follows: CF-UFormer is an end-to-end Transformer-based deep autoencoder architecture, comprising a feature extraction module, a multi-level fusion unit (MFU), and a U-shaped Former structure. Specifically, the low-light image serves as the network input. First, it passes through three branches to convert the original image to RGB, HSV, and LAB color spaces, obtaining information from all three color spaces, including RGB. Then, cascaded Transformer Blocks fuse this information to obtain T0, T1, and T2. To prevent information loss, the three inputs are fed into the MFU separately. The MFU uses skip connections to prevent loss of image details while achieving multi-input processing. Cross-layer information fusion is performed, and the fused information is concatenated by channel dimension and then input into a U-shaped Former structure. The encoder part of the U-shaped Former structure consists of four alternating downsampling and Transformer Block operations, while the decoder part consists of four alternating upsampling and three Transformer Block operations. To allow the network to adaptively focus on the more important dimensions of information, a channel attention mechanism is introduced after the third and fourth downsampling operations. Finally, the output of the U-shaped Former structure is added to F1 and then input into the detail recovery module. The detail recovery module still adopts a cascaded Transformer mechanism with skip connections and multi-level fusion units to avoid the loss of details in the final stage of the network. The feature extraction module processes the image in different color spaces to obtain more realistic color enhancement results. The feature extraction module includes three parallel branches, each of which converts the original image to RGB, HSV and LAB color spaces respectively. The most frequently used RGB color space and the HSV and LAB color spaces with luminance channels are used as processing targets. In the structure of each branch, F represents the color space transformation function. The subsequent three Transformer Blocks with skip connections will perform preliminary processing on the transformed information. In order to avoid the loss of image details in the early stage, a multi-level fusion unit (MFU) is used to perform feature fusion on the output of the three Transformer Blocks. TransformerBlocks were used instead of convolution operations. The result F0 after fusing the three color spaces was used as the input to the U-shaped Former. The encoder part used downsampling and TransformerBlocks alternately four times to obtain a feature map with a resolution reduced by a factor of 16. Each downsampling reduced the image spatial resolution by a factor of 2 and increased the channel dimension by a factor of 2. The number of TransformerBlocks in each stage was {2, 4, 8, 16}, and the number of attention heads was {1, 2, 4, 8}. After the third and fourth downsampling operations, the feature map dimension increased by a factor of 8 and a factor of 16, respectively. The semantic information contained in each channel is different, and the network will ignore some important information during forward propagation. Therefore, a channel attention mechanism was introduced to solve this problem. The decoder operates in the opposite way to the encoder, including four upsampling operations and three stages. Each upsampling increases the image spatial resolution by a factor of two and reduces the number of channel dimensions by a factor of two. The number of TransformerBlocks in each stage is {2,4,8} and the number of attention heads is {1,2,4}. The channel attention mechanism projects the feature map into a high-dimensional space for processing and then maps it back to the original size to obtain an attention map. This attention map has the same dimensions as the input feature map and is used as the attention weight to multiply the feature map element by element to obtain the final result. Multi-level fusion units (MFUs) are employed to preserve and fuse different information. After reshaping the input from N stages, self-attention is calculated, similar to the operation in the Transformer Block. The MFUs only fuse branch dimensions, without destroying the height and width information of each branch. The MFU in the feature extraction module aims to fuse the low-level features extracted by the three Transformer Blocks, ensuring better information preservation of the feature maps in each color space before inputting them into the U-shaped Former structure. The MFU in the detail recovery module further processes the output of the U-shaped Former structure to recover the detail information lost due to downsampling. The improved Transformer Block proposed in this technical solution is represented as follows: Where F in For the input of TransformerBlock, LN represents the layer normalization operation, A-MSA(.) represents the axis self-attention mechanism, F′ represents the output of the axis self-attention mechanism with skip connections, and DGFN(.) represents the dual-gated feedforward network; F out This is the output of TransformerBlock; This technical solution employs an axis self-attention mechanism. Compared to the original ViT, which performs self-attention calculations between all pixels in the image, this method only performs calculations along the height and width directions, which greatly reduces the computational load of the attention mechanism. The multi-axis self-attention part includes cascaded operations on height, width, and channel dimensions. First, the input feature map is mapped to Q, K, and V through 1×1 convolutions and depthwise separable convolutions in three branches. Next, Q and K are reshaped and multiplied to obtain the attention weight matrix. The height and width of the attention matrix are the same as the number of channels in the original feature map. Finally, this attention weight matrix is ​​multiplied with V in the channel dimension to obtain an output of the same size as the original feature map. The dual-gated feedforward network (DGFN) first applies depthwise separable convolution and GELU on two parallel paths and then multiplies them. Next, it sums the outputs of the two branches. Finally, a 1×1 convolution operation combined with residual connections ensures the integrity of the information. A comprehensive weighted loss function is proposed. Three full-reference loss functions guide the network to generate results that are more similar to the labels in terms of single pixels and structure, while another no-reference loss function uses prior knowledge to optimize and adjust the image.

Images