A low-light image enhancement network based on efficient transform and normalized flow fusion

Abstract

The application discloses a low-light image enhancement network based on efficient transformation and normalized flow fusion, and belongs to the field of image processing. The method comprises the following steps: acquiring a low-light image, and extracting multi-scale global-local conditional features; constructing a reversible generation network of a multi-head transpose attention and a gated deep convolution feedforward network, inputting a standard normal distribution and the conditional features into the reversible network, and outputting an enhanced normal-light image. The application uses zero-element guided channel dimension attention modules to realize global context modeling across channels, uses zero-element gated local feedforward enhancement modules to realize controllable feature transformation, and uses a progressive learning strategy to improve the processing capability of high-resolution images. Compared with existing methods, the application can generate low-light enhanced images with uniform illumination, natural colors, rich details and no noise artifacts.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing and computer vision technology, specifically relating to a low-light image enhancement network based on the fusion of efficient Transformer and normalized flow. Background Technology

[0002] Images captured in low-light environments typically suffer from low brightness, poor contrast, color distortion, and severe noise, which not only affect subjective visual perception but also interfere with the performance of subsequent advanced visual tasks (such as object detection and semantic segmentation). Existing low-light enhancement methods are mainly divided into two categories: traditional methods based on Retinex theory and data-driven methods based on deep learning.

[0003] In recent years, methods based on convolutional neural networks (CNNs) have made significant progress, such as LLNet, RetinexNet, and KinD. However, CNNs are limited by their local receptive fields, making it difficult to model long-distance pixel dependencies, and their fixed kernel weights prevent them from adapting to the input content. Transformer models can capture global context through self-attention mechanisms, but their computational complexity increases quadratically with spatial resolution, making them difficult to apply directly to high-resolution images.

[0004] To address this, the efficient Transformer U-shaped conditional coding network applies self-attention to the channel dimension through a zero-element guided channel-dimensional attention module, achieving linear complexity. Furthermore, a zero-element gated local feedforward enhancement module is designed to implement controllable feature transformation, achieving optimal performance in tasks such as image deraining, deblurring, and denoising. On the other hand, LLFlow (AAAI2022) utilizes normalized flow to learn a one-to-many mapping between low-light and normal-light images, and trains an invertible network using negative log-likelihood loss, generating results that are more consistent with human perception.

[0005] However, existing methods have not effectively combined the efficient Transformer architecture with the ability to generate normalized flows. This invention aims to integrate the advantages of both to construct a low-light enhancement network that can capture global-local dependencies and model conditional distributions, while maintaining efficient processing capabilities for high-resolution images. Summary of the Invention

[0006] The purpose of this invention is to provide a low-light image enhancement network based on the fusion of a high-efficiency Transformer U-shaped conditional coding network and a normalized flow, which solves the problem that existing methods cannot simultaneously take into account global dependency modeling, distribution modeling and high-resolution efficient processing, and significantly improves the visual quality and objective indicators of the enhanced image.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a low-light image enhancement network based on the fusion of efficient Transformer and normalized flow, comprising the following steps:

[0008] Step 1: Data preprocessing;

[0009] Acquiring low-light images ∈ Optionally, compute its zero-element graph M∈ :

[0010] ,

[0011] will with The input to the conditional coding network is concatenated with M along the channel dimension (if a zero-element graph is not used, only the input is used). ).

[0012] Step 2: The network consists of two main modules: a conditional coding network and a reversible generative network (based on normalized flow, incorporating a zero-element guided channel-dimensional attention module).

[0013] 2.1 Conditional Coding Network

[0014] It adopts an efficient Transformer U-shaped conditional coding network structure with a total of 4 layers. Each layer contains several Transformer blocks, and each Transformer block is composed of a multi-head transposed attention network and a gated depthwise convolutional feedforward network concatenated together.

[0015] Zero-element guided channel-dimensional attention module: Input feature Y∈ First, pixel-level cross-channel context is aggregated using 1×1 convolution, then channel-level spatial context is encoded using 3×3 depthwise convolution to obtain the query Q-key K value V. Then, Q and K are reshaped to obtain Q^∈ 、K^∈ Calculate the transposed attention map The output V^⋅A is then subjected to a 1×1 convolution. The computational complexity of the zero-element guided channel-dimensional attention module is O(n). ,when It has a linear complexity.

[0016] Zero-element gated local feedforward enhancement module: Employs a gating mechanism with two paths: one activated via GELU, and the other directly passing through, performing element-wise multiplication and introducing a 3×3 depthwise convolution. The channel expansion factor is set to 2.66 to maintain parameter balance.

[0017] The encoder downsamples step-by-step through pixel-unshuffle, doubling the number of channels; the decoder upsamples through pixel-shuffle and fuses the encoder features via skip connections. The final output consists of multi-scale conditional features Ft1, Ft2, and Ft3, each corresponding to a different spatial resolution.

[0018] 2.2 Reversible Generative Networks

[0019] The reversible network adopts a three-level architecture, with each level containing one scaling layer and eight identical reversible transformation blocks. Each reversible transformation block consists of the following sub-layers in the following order:

[0020] Reversible activation layer: Performs channel-wise affine transformation on features, and the parameters are learnable.

[0021] Reversible 1×1 convolutional layer: channels are mixed through reversible convolutional kernels.

[0022] Affine injection layer: Taking the conditional feature Ft as input, the parameter extraction network (composed of three convolutional layers) outputs translation and scaling coefficients to perform affine transformation on the feature.

[0023] Cross-affine coupling layer: The features are divided into odd and even halves in the channel dimension, and affine transformation is performed in two stages. The transformation coefficients are generated by concatenating the other half of the features with the conditional features and then passing them through the parameter extraction network.

[0024] Improvement: Embed a zero-element guided channel-dimensional attention module after the cross-affine coupling layer or in the affine injection layer (without changing the reversibility, because the zero-element guided channel-dimensional attention module itself is a linear transformation plus residual, which can be used as a supplement to the reversible layer in the flow model, and the overall reversibility is guaranteed by combining it with reversible convolution) to enhance the global interaction between channels.

[0025] The scaling layer changes the feature size through reversible pixel rearrangement operations: in the forward direction, the spatial size is halved and the number of channels is increased to 4 times; in the reverse direction, it is restored.

[0026] Step 3: Training and Reasoning

[0027] Training phase: Using pairs of low-light-normal-light images (IL, IGT). The input conditional encoding network is then used to obtain... Simultaneously, the IGT is input into the reversible network to perform a forward transformation (mapping the normal light image to the latent space) and calculate the latent variables. Negative log-likelihood loss is used for training.

[0028] ,

[0029] in To enhance reconstruction details, L1 loss is incorporated:

[0030] ,

[0031] in For the network in inference mode (sampling) The enhanced images generated are used. A progressive learning strategy is adopted: the initial training image patch is 128×128, with a batch size of 64; after 92k iterations, it is increased to 160×160, with a batch size of 40; then it is gradually increased to 384×384, with a batch size of 8, until convergence.

[0032] Inference phase: Given a low-light image IL, extract Ft through a conditional coding network, sampling from a standard normal distribution. ,Will Inputting the image into an invertible network and performing an inverse transformation yields an efficient global model of the enhanced image: .

[0033] The advantages of this invention compared to existing technologies are as follows:

[0034] Efficient global modeling: Introducing a zero-element guided channel-dimensional attention module of the efficient Transformer U-shaped conditional coding network to compute attention in the channel dimension, achieving linear complexity, capable of processing high-resolution low-light images, and capturing long-distance pixel dependencies.

[0035] Distribution modeling capability: Based on normalized flow learning of one-to-many mappings, it avoids the blurring and artifacts produced by deterministic models, and the generated results are more consistent with the real illumination distribution.

[0036] Controllable feature transformation: The gating mechanism of the zero-element gated local feedforward enhancement module and the depth convolution effectively filter useful information and enhance local structure recovery.

[0037] Progressive training: enables the model to learn global statistical properties at different resolutions, thereby improving its generalization ability.

[0038] Experiments demonstrate that on datasets such as LOL and VE-LOL, the method of this invention outperforms existing state-of-the-art methods (including standalone efficient Transformer U-shaped conditional coding network and standalone LLFlow) in terms of PSNR, SSIM, and LPIPS, enhancing image brightness to be natural, details to be clear, and noise to be low. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall workflow of the method of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of the high-efficiency Transformer U-shaped conditional coding network of the present invention;

[0041] Figure 3This is an internal structural diagram of the zero-element guided channel attention module of the present invention;

[0042] Figure 4 This is an internal structural diagram of the zero-element gated local feedforward enhancement module of the present invention;

[0043] Figure 5 This is a diagram showing the overall structure of the reversible generative network of this invention.

[0044] Figure 6 This is a structural diagram of the reversible transform block of the present invention, showing the connection relationship of the reversible activation layer, 1×1 convolution, affine injection layer, and cross-affine coupling layer enhanced by the zero-element guided channel attention module;

[0045] Figure 7 This is a detailed calculation diagram of the cross-affine coupling layer of the present invention. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. This embodiment uses the LOL-v2-real dataset as an example, with an image size of 400×600.

[0048] Example 1: Network Parameter Settings

[0049] Conditional coding network: encoder has 4 levels, number of channels: 48, 96, 192, 384; number of Transformer blocks per level: [4, 6, 6, 8]; number of zero-element guided channel dimension attention module heads: [1, 2, 4, 8]; expansion factor γ=2.66 for zero-element gated local feedforward enhancement module.

[0050] Downsampling uses pixel inverse rearrangement (factor 2), and upsampling uses pixel rearrangement. Skip connections: Encoder features and decoder features are concatenated at the same resolution and then halved in channels by 1×1 convolution (1×1 convolution is not used at the top layer to preserve details).

[0051] Reversible generative networks:

[0052] It has 3 levels, with 8 reversible transformation blocks in each level.

[0053] The parameters of the reversible activation layer are learnable, with an initial scaling factor of 1 and a translation factor of 0.

[0054] Reversible 1×1 convolution: Stable inversion is achieved using LU decomposition.

[0055] The parameter extraction network NN for the affine injection layer and the cross-affine coupling layer consists of 3 convolutional layers (3×3, stride=1, padding=1 → 1×1 → 3×3), with channels (conditional feature channel, 64, 64, current feature channel), and ReLU is used in between.

[0056] A zero-element guided channel dimension attention module is added after the cross-affine coupling layer, with the input and output channels remaining unchanged and the number of heads being 4.

[0057] Training parameters:

[0058] Optimizer: AdamW, learning rate 3e-4, weight decay 1e-4.

[0059] The total number of iterations was 300k, and the learning rate was adjusted to 1e-6 using cosine annealing.

[0060] Progressive learning path: 128²(64 batches) → 160²(40) → 192²(32) → 256²(16) → 320²(8) → 384²(8), with switching iterations of 92k, 156k, 204k, 240k, and 276k respectively.

[0061] L1 loss balance coefficient λ = 0.1.

[0062] Example 2: Training Process

[0063] Load the LOL-v2-real training set (689 pairs), randomly crop it to 128×128, and randomly flip it horizontally / vertically. Input the low-light image into the conditional encoding network to obtain conditional features at three scales. Input the corresponding normal-light image into the reversible generative network and perform a forward transformation: sequentially pass through three levels of scale transformation and 8 reversible transformation blocks at each level, using the conditional features of the corresponding scale at each level, and finally output the latent variables.

[0064] Calculate the negative log-likelihood loss:

[0065] Simultaneously, the network is placed in inverse mode, sampling N(0,I), and inversely generating enhanced images along with conditional features to calculate the L1 loss. The total loss is backpropagated to update the parameters. The image patch size and batch size are adjusted according to the progressive learning plan.

[0066] Example 3: Test Results

[0067] On the LOL-v2-real test set (100 pairs), the average PSNR of this invention is 25.83 dB, SSIM is 0.942, and LPIPS is 0.094. Compared with the standalone efficient Transformer U-shaped conditional coding network (adapted for low-light tasks, PSNR 24.31 dB) and LLFlow alone (24.06 dB), it represents improvements of 1.52 dB and 1.77 dB, respectively. In terms of subjective visual effects, the enhanced image has uniform brightness, good recovery of details in dark areas, and no obvious noise amplification or color deviation.

[0068] Example 4: Zero-element guided variant

[0069] In the preprocessing stage, an additional zero-element map was computed and stitched together to form a 4-channel input conditional coding network. The results show that the PSNR was further improved by 0.2 dB, especially in the recovery of extremely dark regions (regions with dense zero elements).

[0070] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A low-light image enhancement network based on the fusion of efficient Transformer and normalized flow, characterized in that, Includes the following steps: Step 1, Data Preprocessing; Acquiring low-light images And extract its multi-scale features or zero-element map as auxiliary input: Step 2: Construct a conditional coding network and a reversible generative network to form an overall augmentation network and complete training. The conditional coding network is based on the Transformer architecture and includes multiple cascaded multi-head transposed attention modules and a gated deep convolutional feedforward network to extract multi-scale conditional features from low-light images. The reversible generative network consists of multi-level scale transformation layers and reversible transformation blocks. Each reversible transformation block includes a reversible activation layer, a 1×1 convolutional layer, an affine injection layer, and a cross-affine coupling layer that fuses zero-element guided channel-dimensional attention modules. Step 3: Input the low-light image into the conditional coding network to extract conditional features. Simultaneously, latent variables are sampled from the standard normal distribution. ,Will With conditional features A common input reversible generative network generates enhanced images through inverse transformation. .

2. The method according to claim 1, characterized in that, The conditional coding network adopts a 4-level encoder-decoder structure. Each level contains multiple Transformer blocks. Each Transformer block is composed of a zero-element guided channel-dimensional attention module and a zero-element gated local feedforward enhancement module connected in series. The zero-element guided channel-dimensional attention module aggregates pixel-level cross-channel context and channel-level spatial context through 1×1 convolution and 3×3 depthwise convolution, respectively, and then calculates the transposed attention map across channels. The zero-element gated local feedforward enhancement module adopts a dual-path gating mechanism. One path is activated by GELU and multiplied element-wise with the other path, and depthwise convolution is introduced to encode local spatial information.

3. The method according to claim 1, characterized in that, The reversible generative network introduces a multi-head transpose attention module, which embeds the zero-element guided channel dimension attention module before the cross-affine coupling layer or in the affine injection layer, to enhance the global dependency between channels during the flow transformation process. Specifically, in the cross-affine coupling layer, the features after affine transformation are input into the zero-element channel dimension attention module, and then concatenated with the conditional features into the input parameter extraction network.

4. The method according to claim 1, characterized in that, The overall network is trained using a progressive learning strategy: small image patches and large batches are used in the early stages of training, and the image patch size is gradually increased and the batch size is decreased as the number of iterations increases.

5. The method according to claim 1, characterized in that, The loss function during training is a weighted sum of the negative log-likelihood loss and the pixel-level reconstruction loss: ， The first term is the regularization term of the latent space, the second term is the logarithm of the Jacobian determinant of the invertible transformation, the third term is the L1 reconstruction loss, and λ is the balance coefficient.

6. The method according to claim 1, characterized in that, The conditional coding network introduces jumpers between encoder-decoder layers and concatenates encoder features and decoder features at the same spatial resolution level, then performs 1×1 convolution for dimensionality reduction to preserve detailed information.

7. The method according to claim 1, characterized in that, Each level of the reversible generative network contains eight identical reversible transformation blocks, each of which consists of, in order: a reversible activation layer, a reversible 1×1 convolutional layer, an affine injection layer, and a cross-affine coupling layer enhanced by a zero-element guided channel dimension attention module.

8. The method according to claim 1, characterized in that, The preprocessing of the low-light image also includes calculating a zero-element map, concatenating the zero-element map with the low-light image in the channel dimension, and inputting the concatenated map into a conditional coding network to guide the network to focus on areas with severe information loss.

Images