A low-light image enhancement method based on reversible neural network

Abstract

The present application relates to the technical field of low-light image, and particularly relates to a low-light image enhancement method based on reversible neural network. The method comprises the following steps: step one, building a reversible neural network; step two, working process and training of the reversible neural network; step three, application: inputting an image taken under low-light conditions into the trained reversible neural network for enhancement processing. Compared with the prior art, the present application improves the enhancement effect and interpretability of the model, increases the decomposition accuracy and generalization ability of the model, introduces a memory mechanism to learn the brightness information of the entire data set and guide the recovery process of the image in terms of brightness, thereby improving the enhancement effect of the model.

Description

Technical Field

[0001] The present invention relates to the field of low-light image technology, and in particular to a low-light image enhancement method based on a reversible neural network. Background Art

[0002] Images captured under low-light conditions often suffer from visual degradation problems such as low brightness, low contrast, and severe noise pollution. These visual degradation problems can cause the loss of image information, leading to malfunction or failure of computer vision systems. To address this challenge, low-light image enhancement (LLIE) has become a highly sought-after research direction in the field of computer vision. LLIE aims to improve the visibility of low-light images and effectively suppress noise and artifacts caused by low light in the shooting environment. Through low-light image enhancement technology, various computer vision applications can benefit from improved low-light images and enhance system performance and robustness.

[0003] In the early stages of computer vision development, low-light image enhancement primarily relied on traditional image processing techniques. These traditional low-light image enhancement methods can be divided into two categories: those based on image priors and those based on physical models. Methods based on image priors rely on prior knowledge of image processing and computer vision. For example, histogram equalization is a common method based on image priors. It adjusts the global intensity distribution in a linear or nonlinear manner to improve image brightness and contrast. However, these methods often require manual parameter tuning, are sensitive to noise, and have poor generalization capabilities when handling complex scenes. On the other hand, methods based on physical models use imaging theory to guide image enhancement. For example, Retinex theory decomposes an image into illumination and reflectance components, then processes the reflectance component to achieve enhancement. These methods have strong generalization capabilities and can adapt to diverse scenes. However, these methods perform poorly in scenes with complex lighting conditions.

[0004] In recent years, deep learning techniques have been widely used in low-light image enhancement (LLIE) tasks, achieving excellent performance. Deep learning-based LLIE methods improve low-light image enhancement performance by improving the model's structural design. Compared to traditional methods, deep LLIE methods leverage the powerful capabilities of convolutional neural networks (CNNs) and Transformer neural networks to achieve superior results. Summary of the Invention

[0005] Purpose of the present invention:

[0006] Most learning-based LLIE methods learn the mapping relationship between low-light images and normal-light images by using some paired data. However, in real scenes, one low-light image can correspond to multiple normal-light images with different brightness. The model trained based on paired data tends to fuse multiple possible results as output, resulting in problems such as uneven brightness enhancement, residual noise removal, and image artifacts. Some methods propose to learn the bidirectional mapping relationship between low-light images and normal-light images, and use the auxiliary information of the reverse process to guide the learning of the forward process. However, these methods simply use reversible networks without specific design for the task.

[0007] In addition, in order to further improve the image enhancement quality, many methods introduce the Retinex theory into the model. Since the Retinex theory simulates the color perception of human vision, the deep learning method based on Retinex often achieves better results and better generalization ability. However, the illumination component and the reflection component obtained by Retinex decomposition lack ground truth, and cannot guarantee the accuracy and reasonableness of the decomposition result. These works solve this problem by introducing additional priors or regularizers, but these manually designed regularizers are difficult to apply to all scenes. Excessive constraints in joint optimization also lead to lack of adaptability and efficiency.

[0008] The present application mainly proposes optimization for the following problems of low-light image enhancement:

[0009] (1) For the specific task of low-light image enhancement, the present application integrates the Retinex theory into the reversible neural network, making the whole process have better interpretability.

[0010] (2) Further, Retinex decomposition can improve the low-light image enhancement result, but the illumination component and the reflection component lack ground truth, and cannot guarantee the accuracy and reasonableness of the decomposition result. In order to accurately decompose and improve the generalization ability, the present application uses the cross-color ratio imaging theory to guide the Retinex decomposition.

[0011] (3) Further, the present application introduces a memory mechanism to improve the brightness recovery effect, learns the brightness information of the whole dataset, and uses this information to guide the low-light image enhancement.

[0012] Technical scheme:

[0013] A low-light image enhancement design method based on reversible neural network, characterized in that it comprises:

[0014] Step 1: Build a reversible neural network;

[0015] Step 2: Working process and training of reversible neural network;

[0016] Step three application: input the image taken under low light conditions into the above trained reversible neural network for enhancement processing.

[0017] Compared with the prior art, the present application has the following beneficial effects:

[0018] (1) Enhance the interpretability of the process: the present application redesigns the reversible network structure, so that the reversible network enhancement process can be explained by Retinex theory, improving the enhancement effect and interpretability of the model.

[0019] (2) Generalization of image decomposition: the present application introduces a cross-color ratio physical model to guide the image decomposition process based on Retinex theory, increasing the accuracy of decomposition and the generalization ability of the model.

[0020] (3) Global guidance of brightness information: the present application introduces a memory mechanism to learn the brightness information of the entire data set, guiding the brightness recovery process of the image and improving the enhancement effect of the model. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is the overall framework flowchart of the present application;

[0022] Figure 2 is the detailed description of the feature enhancement module in Figure 1

[0023] Figure 3 is the visual comparison result on the LOL-v1 dataset;

[0024] Figure 4 is the visual comparison result on the LOLv2-real dataset;

[0025] Figure 5 is the visual comparison result on the SID dataset;

[0026] Figure 6 is the visual comparison result on the SMID dataset;

[0027] Figure 7 is the visual comparison result on the SDSD-in dataset;

[0028] Figure 8 is the visual comparison result on the SDSD-out dataset. DETAILED DESCRIPTION

[0029] The present application will be described in detail below in conjunction with the drawings and specific examples. The present embodiment is implemented on the premise of the technical solution of the present application, and gives a detailed implementation and specific operation process, but the protection scope of the present application is not limited to the following examples. ​

[0030] The present application proposes a low-light image enhancement method based on reversible neural network, which combines Retinex theory with reversible neural network to better learn the one-to-one mapping between low-light images and normal light images. Specifically, the present application redesigns the forward process and reverse process of reversible neural network, so that the forward process can be represented as image enhancement based on Retinex theory, and the reverse process can be represented as image degradation based on Retinex theory. Further, the present application introduces a cross color ratio (CCR) physical model to guide the decomposition results of Retinex, improve the accuracy of decomposition and the generalization ability of the model. In addition, the present application introduces a memory mechanism to learn the brightness information of the data set, further improving the brightness recovery effect of low-light images.

[0031] As shown in Figure 1 , Figure 2 , a low-light image enhancement design method based on reversible neural network is introduced in detail as follows:

[0032] Step one: build reversible neural network

[0033] The reversible neural network is composed of a feature encoding module φ(·), a feature decoding module ψ(·), and n stacked reversible modules θ(·). The feature encoding module φ(·) and the feature decoding module ψ(·) are connected in parallel, and then sequentially connected with the stacked reversible modules θ(·).

[0034] The feature encoding module φ(·) and the feature decoding module ψ(·) are both composed of two convolutional layers, where the size of the convolution kernel is 3x3 and the step is 1.

[0035] The reversible module θ(·) is composed of four feature enhancement modules G1(·), H1(·), G2(·), and H2(·), which are connected in the manner shown in Figure 1 They have the same network structure.

[0036] The feature decoding module ψ(·) is also composed of two convolutional layers, where the size of the convolution kernel is 3x3 and the step is 1.

[0037] More specifically,

[0038] As shown in Figure 2 ​​(a) As shown, the feature enhancement modules G1(·), H1(·), G2(·), H2(·) are each composed of a residual dual-branch attention module (DBAM) and a residual feed-forward neural network (FFN), wherein the feed-forward neural network (FFN) is implemented by a 3x3 convolution layer.

[0039] As shown in Figure 2 (b) As shown, the dual-branch attention module (DBAM) is composed of a Max Pooling layer, a Transformer branch, a convolutional neural network (CNN) branch (referred to as: CNN branch), a channel concatenation layer (Concatenate), a 3x3 convolution layer, and an interpolation layer (Interpolate). The Transformer branch and the CNN branch are connected in parallel and sequentially connected with other layers.

[0040] Wherein:

[0041] The CNN branch is composed of a dual attention unit (DAU) with a residual. The network structure of the DAU is as shown in Figure 2 (b) As shown by the dashed box, it also contains two branches. The first branch is composed of two layers of 1x1 convolution layers with a scaled exponential linear unit (SELU) activation function, and the second branch adds two layers of 1x1 convolution layers with a SELU activation function after a global average pooling layer (GAP). The results of the two branches are added and then normalized by a Sigmoid function. The normalized result is used as a weight to weight the second input of the DAU.

[0042] The Transformer branch is sequentially composed of a cross-attention module and a dual attention unit (DAU). The cross-attention module is composed of a 1x1 convolution layer and a Softmax function, and additionally introduces a global variable M and an interpolation layer (Interpolate).

[0043] As mentioned above, the dual attention unit (DAU) of the Transformer branch has the same network structure as the dual attention unit (DAU) of the CNN branch, but the input data is different.

[0044] Step two Working process and training of reversible neural network

[0045] Design steps including forward process

[0046] Step 1.1. Calculate cross color ratio image.

[0047] Low-light image I low As the input of the reversible neural network forward process, the cross color ratio is first calculated by the cross color ratio calculation formula f ccr (·) to calculate the cross color ratio image

[0048]

[0049] Step 1.2. Feature extraction.

[0050] The low-light image I low and the cross color ratio image are input into the feature encoders φ(·) and respectively to extract the respective features and

[0051]

[0052] Step 1.3. Feature conversion.

[0053] The features and are input into the n stacked reversible modules θ(·) to perform feature conversion, obtaining the converted features and corresponding to the enhanced image features and the enhanced image cross color ratio image features. For the i-th reversible module θ i (·), the input features and are converted based on the feature enhancement modules G1(·), H1(·), G2(·), H2(·). The key innovation is that the feature conversion process is combined with the Retinex theory. In addition, the cross color ratio image is also used to guide the decomposition result of Retienx. Based on the Retienx theory, the input features and can be explained as:

[0054]

[0055] where represents the reflection component, represents the illumination component, and represent the noise component. Further, the output of the feature enhancement module G2(·) is regarded as the noise component The output of the feature enhancement module H2(·) Considered as light component Then the reversible module θ i (·) can be achieved by Remove noise components and add lighting components To convert it into features

[0056]

[0057] Similarly, by taking the output of the feature enhancement module G1(·) Considered as noise component The output of the feature enhancement module H1(·) Considered as light component The reversible module θ i (·) Further by removing the noise component and remove the lighting component The features Convert to Features

[0058]

[0059]

[0060] The feature enhancement modules G1(·), H1(·), G2(·), and H2(·) are used to enhance the input features, as shown in the following example: Figure 2 (a) shows the structure. Taking G1(·) as an example, the input feature x is processed as follows:

[0061] G1(x)=f FFN (f DBAM (x)+x)+(f DBAM (x)+x)

[0062] where f DBAM (·) represents the dual-branch attention module (DBAM), f FFN (·) represents the feedforward neural network (FNN).

[0063] like Figure 2 As shown in (b), the dual-branch attention module (DBAM) transforms the input feature x into feature x ′ The input feature x first passes through the maximum pooling layer f MP (·) downsamples to reduce the computational complexity of the model. Then it is split into features x along the channel dimension. cnn and feature x trans, used as input to the CNN branch and the Transformer branch:

[0064] {x cnn ,x trans}=f MP (x).

[0065] The CNN branch is mainly composed of dual attention units (DAU) with residuals, which are used to cnn To enhance:

[0066] x′ cnn =f DAU (x cnn ,x cnn )+x cnn .

[0067] like Figure 2 (b) As shown in the dotted box, the dual attention unit (DAU) accepts two features as input. The first feature x cnn After the dual-branch structure is enhanced, the second feature x cnn Weighted:

[0068]

[0069] The Transformer branch is composed of a cross-attention module and a dual attention unit (DAU) in sequence. For the cross-attention module, first, trans Calculate the key and value in the CNN branch and use the result x′ cnn It is used as a query to calculate the cross attention. In addition, it is observed that the existing model is poor in restoring image brightness information. Only by modulating the global brightness information of the true value image can the reconstruction effect and index be effectively improved. Therefore, the cross attention module introduces a memory mechanism and uses the global variable M∈R 1×C×3×3 To learn the brightness information of the dataset and modify the size of M using interpolation, then Q Modulate for better reconstruction:

[0070] {x K ,x V}=f conv (x trans ),x Q =x′ cnn ×f interp (M)

[0071]

[0072] Enhanced feature x′ transFurther input to the dual attention unit (DAU) for processing. The dual attention unit (DAU) has the same network structure as the dual attention unit (DAU) in the CNN branch, but the input features are different to realize the weighting of x' cnn :

[0073]

[0074] Wherein:

[0075]

[0076] The results x" cnn of the CNN branch and the results x' trans of the Transformer branch are spliced along the channel dimension, then input to a 3x3 convolution for processing, and restored to the input size by interpolation:

[0077]

[0078] Step 1.4 image reconstruction.

[0079] The features of the enhanced image and the features of the cross-color ratio image of the enhanced image are input to the feature decoding module ψ(·) to obtain the enhanced image I' norm :

[0080]

[0081] And the design steps including the reverse process

[0082] Step 2.1 Calculate the cross-color ratio image.

[0083] The normal image I norm is input as the input of the reversible neural network reverse process, and the cross-color ratio image is first calculated by the cross-color ratio calculation formula f ccr (·):

[0084]

[0085] Step 2.2 Feature extraction.

[0086] The normal image I norm and the cross-color ratio image are input to the feature encoders φ(·) and respectively, and the respective features and

[0087]

[0088] Step 2.3 Feature Inverse Conversion.

[0089] Feature and Input into the n stacked reversible modules θ(·) for feature inverse conversion, to get the inverse converted feature and Corresponding to the degraded image feature and the feature of the degraded image cross-color ratio graph. For the i-th reversible module θ i (·), input feature and Based on the feature enhancement module G1(·), H1(·), G2(·), H2(·), the feature inverse conversion operation is performed. Based on the Retienx theory, the input feature and Can be explained as:

[0090]

[0091] Wherein represents the reflection component, represents the illumination component, and represents the noise component. As in the forward process, the output of the feature enhancement module G1(·) is regarded as the noise component The output of the feature enhancement module H1(·) is regarded as the illumination component Then the reversible module θ i (·) can convert the feature and add the illumination component and add the noise component to convert it into the feature

[0092]

[0093] Similarly, by regarding the output of the feature enhancement module G2(·) as the noise component The output of the feature enhancement module H2(·) is regarded as the illumination component The reversible module θ i (·) further converts the feature by removing the illumination component and adding the noise component to convert it into the feature

[0094]

[0095]

[0096] The feature enhancement modules G1(·), H1(·), G2(·), and H2(·) are used to enhance the input features, and have the structure shown in (a). Figure 2

[0097] Step 2.4 Image reconstruction.

[0098] Features of the degraded image Features of the cross-color ratio image of the degraded image Input into the feature decoding module ψ(·) to obtain the degraded image I' low :

[0099]

[0100] and the design step of the loss function

[0101] In the training phase, the L1 distance is used as the loss function of the method. For the forward and backward processes, we have:

[0102]

[0103] Since using only the L1 loss may cause image smoothing, a perceptual loss is further used for constraint to improve the perceptual quality of the image:

[0104]

[0105] where VGG i (i) represents the i-th layer of the pre-trained VGG-19 on ImageNet, i∈{1,3,5,9,13}. In addition, ω i is the weight corresponding to the i-th layer. In the present application, ω i is set to 1. In addition, the forward process needs to denoise the low-light image, so an edge-aware smoothing loss is introduced to improve the denoising effect:

[0106]

[0107] where and represent the gradients in the horizontal and vertical directions. Therefore, the total loss of the present application is the weighted sum of the above losses:

[0108]

[0109] where λ i represents the weight, and λ1=λ4=0.05, λ2=λ3=1.​

[0110] Step three application

[0111] Based on the above design, the image taken under low light conditions is input into the above trained reversible neural network for enhancement processing.

[0112] The present application achieves good results on multiple public data sets. For detailed description, see the examples.

[0113] Key points of the technical solution of the present application

[0114] (1) Reversible neural network based on Retinex theory.

[0115] In order to make the reversible neural network more suitable for low light image enhancement task, and improve the explainability of the network, the structure of the reversible network is redesigned, so that the feature transformation process of the reversible neural network can be explained by Retinex theory. Specifically, the forward process can be explained as enhancing the illumination component while removing the noise component, and the inverse process can be explained as degrading the illumination component while adding the noise component. Based on this design, the process of image enhancement by reversible neural network can be well explained, and the reversible neural network can more specifically handle the low light image enhancement task.

[0116] (2) Retinex decomposition under the guidance of cross color ratio map.

[0117] In order to solve the problem of lack of effective constraint in Retinex decomposition, and improve the generalization ability of the model, the present application uses cross color ratio map, an imaging theory, to guide the decomposition process. Specifically, Retinex theory decomposes the image into illumination component and reflectance component, but lacks the corresponding decomposition result truth value for constraint. In imaging theory, cross color ratio map represents illumination invariant gradient, and its value is related to reflectance. Therefore, cross color ratio map can be used to guide the reflectance component, and thus effectively constrain and guide the result of image decomposition.

[0118] (3) Dataset brightness learning based on memory mechanism.

[0119] In order to encode the overall brightness information of the dataset and improve the ability of the model to recover the brightness information of the image, the present application uses a feature enhancement module based on memory mechanism. Specifically, the reversible neural network realizes the feature conversion process through the feature enhancement module G / H, and each feature enhancement module is realized through a complex neural network. The present application uses a model design combining CNN and Transformer to extract global and local information. On this basis, the present application further introduces a memory mechanism to learn the brightness information M of the entire dataset and modulate the Q component of the Transformer.

[0120] Advantages of the present application

[0121] (1) Most methods only learn the mapping relationship from low-light images to normal-light images, but this mapping relationship is ill-posed, and the model will tend to learn the mean of multiple mappings, thereby affecting the effect of the model. Some methods propose to learn the bidirectional mapping relationship between low-light images and normal-light images, and use the auxiliary information of the inverse process to guide the learning of the forward process. These methods simply use reversible networks and lack improvements for specific tasks. The present application redesigns the structure of the reversible neural network for the low-light image enhancement task, so that the enhancement process of the reversible network can be explained by the Retinex theory, thereby improving the performance of the model.

[0122] (2) Since the Retinex theory simulates human color perception, many methods have introduced it into the low-light image enhancement task and have achieved good results. However, the illumination component and the reflection component obtained by Retinex decomposition lack ground truth constraints, and cannot guarantee the accuracy and reasonableness of the decomposition results. Some methods introduce additional priors or regularizers to solve this problem, but lack generalization ability and are difficult to apply to more scenes. The present application introduces the cross-color ratio theory, which uses this illumination invariant to guide the reflection component of Retinex decomposition, to obtain better decomposition results and generalization ability.

[0123] (3) Existing methods have poor effects on brightness recovery, and the model updates based on limited data each time, which cannot effectively learn the brightness information of the entire dataset. The present method introduces a memory mechanism that can receive information from all images in the dataset during training, so it can be used to learn the brightness information of the dataset, and then guide the brightness recovery of low-light image enhancement.

[0124] Embodiments

[0125] Dataset

[0126] Experiments were conducted on commonly used low-light image enhancement benchmark datasets, including LOL-v1, LOL-v2, SID, SMID, SDSD, and LSRW. To demonstrate the generalization ability of the model, the present application also tested on five datasets without ground truth, including DICM, LIME, MEF, NPE, and VV.

[0127] Method application

[0128] A method for low-light image enhancement based on a reversible neural network, comprising:

[0129] Forward process description: first, based on the CCR theory f ccr (·), using a low-light image Ilow cross-color ratio map Then, the low-light image I low and the cross-color ratio map are input into the feature encoding module φ(·) and extracted features of the low-light image and the corresponding cross-color ratio map Then, the features and are converted by inputting into n reversible modules θ(·) to obtain enhanced features and Finally, the features and are input into the feature decoding module ψ(·) to obtain the enhanced image I' norm .

[0130] Reverse process description: first, based on the CCR theory f ccr (·), the normal image I norm is used to calculate the cross-color ratio map Then, the normal image I norm and the cross-color ratio map are input into the feature encoding module φ(·) and extracted features of the normal image and the corresponding cross-color ratio map Then, the features and are converted by inputting into n reversible modules θ(·) to obtain degraded features and Finally, the features and are input into the feature decoding module ψ(·) to obtain the degraded image I' low .

[0131] The parts not described in detail are the same as in the foregoing and Figures 1-2 network structure, training algorithm (loss function). In the embodiments, they will not be described again. The present application is based on the Pytorch framework and is trained on an NVIDIA 4090 GPU. The present application is trained using the Adam optimizer, where β1 is set to 0.9 and β2 is set to 0.999. The initial learning rate is set to 3×e-4 and is decayed to 1×e-6 using the cosine annealing scheme. In addition, the batch size and patch size are set to 5 and 256×256 respectively, and the number of iterations is set to 2.5×10 5 . Random augmentation is performed on the training data, including horizontal flipping, vertical flipping, and 90° rotation.

[0132] Implementation effect comparison

[0133] For ease of comparison, several state-of-the-art methods are selected for comparison, which are all improved for the low-light image enhancement task. These methods include KinD, LLFlow, SNR-Net, LLFormer, PyDiff, UHD Four and DiffLL.

[0134] First, the Peak Signal-to-Noise Ratio (PSNR) and structural similarity (SSIM) indicators are compared on multiple paired datasets. There are two test methods for existing methods, one is direct testing, and the other is adjusted testing using the average value of the ground truth image. In order to compare fairly, we implement both test methods on all methods, and compare the corresponding full-reference indicators respectively. Among them, the indicators without true value adjustment are represented as PSNR and SSIM, and the indicators with true value adjustment are represented as PSNR-G and SSIM-G. For methods with pre-trained weights, directly test using the weights. For methods without pre-trained weights, keep the original parameters unchanged and train for 300,000 iterations. Tables 1-9 show the performance of our method and the comparison methods on 6 datasets and the average results. Our method achieves the best or sub-optimal results on multiple datasets, while achieving the best average result. Then we compare the generalization ability of the model on five non-paired datasets. We uniformly use the weights trained on the LOLv1 dataset for testing. Table 10 shows the Natural Image Quality Evaluator (NIQE) indicators of the method of the present application and the comparison method. The method of the present application achieves the best or sub-optimal result on multiple datasets, and has a sub-optimal average result, indicating that the method of the present application has good generalization ability.

[0135] Table 1 Comparison of experimental results of the present application and other models on LOL-v1 dataset

[0136]

[0137] Table 2 Comparison of experimental results of the present application and other models on LOL-v2-real dataset

[0138]

[0139]

[0140] Table 3 Comparison of experimental results of the present application and other models on SID dataset

[0141]

[0142] Table 4 Comparison of experimental results of the present invention and other models on SMID dataset

[0143]

[0144] Table 5 Comparison of experimental results of the present invention and other models on SDSD-in dataset

[0145]

[0146] Table 6 Comparison of experimental results of the present invention and other models on SDSD-out dataset

[0147]

[0148] Table 7 Comparison of experimental results of the present invention and other models on LSRW-Huawei dataset

[0149]

[0150]

[0151] Table 8 Comparison of experimental results of the present invention and other models on LSRW-Nikon dataset

[0152]

[0153] Table 9 Comparison of average experimental results of the present invention and other models on multiple datasets

[0154]

[0155] Table 10 Quantitative results of NIQE on DICM, LIME, MEF, NPE and VV datasets

[0156]

[0157]

[0158] Figures 3-8 For the visual comparison results of the present invention and previous methods on multiple datasets:

[0159] Figure 3 The visualization results on LOLv1 dataset are shown. KinD performs image enhancement based on Retinex theory and reduces color difference, but the brightness recovery is limited. Although SNR-Net, LLFormer, and UHD-Four methods further restore the brightness and contrast of the image, there are obvious noise and artifacts.

[0160] Both LLFlow method and PyDiff method are based on generative model to implement low-light image enhancement. Although the visual effect is good, the recovered image is too smooth and loses a lot of detailed information. DiffLL method has obvious chessboard artifacts because it finally uses wavelet inverse transform to obtain the image. In contrast, the present invention not only removes noise artifacts, but also retains detailed information, with the best visual effect. We also visualize the test results on LOL-v2-real, SID, SMID, SDSD-in and SDSD-out datasets, as shown in Figures 4-8 Fig. 6.

Claims

1. A low-light image enhancement design method based on reversible neural network, characterized in that: include: Step 1: Build a reversible neural network; Step 2: The working process and training of reversible neural network; Step 3: Apply the image captured under low-light conditions to the trained reversible neural network for enhancement. The step 1 builds a reversible neural network: The reversible neural network consists of a feature encoding module φ(·) and And the stacked n reversible modules θ(·), and the feature decoding module ψ(·), the feature encoding module φ(·) and After parallel connection, it is sequentially connected with the stacked reversible module θ(·) and feature decoding module ψ(·), where: The feature encoding module φ(·) and They are all composed of two convolutional layers; the reversible module θ(·) is composed of four feature enhancement modules G1(·), H1(·), G2(·), and H2(·), which have the same network structure; the feature decoding module ψ(·) is also composed of two convolutional layers; The feature enhancement modules G1(·), H1(·), G2(·), and H2(·) are composed of a dual-branch attention module DBAM with residual and a feedforward neural network FFN with residual; The dual-branch attention module DBAM consists of a maximum pooling layer, a Transformer branch, a CNN branch, a channel concatenation layer, a 3x3 convolution layer, and an interpolation layer. The Transformer branch and the CNN branch are connected in parallel and sequentially connected to other layers. in: The CNN branch consists of a dual attention unit (DAU) with residuals. The dual attention unit (DAU) contains two branches. The first branch consists of two 1x1 convolutional layers with a scaled exponential linear unit (SELU) activation function. The second branch adds two 1x1 convolutional layers with a SELU activation function after the global average pooling layer (GAP). The results of the two branches are added together and normalized by the Sigmoid function. The normalized result is used as the weight to weight the input of the dual attention unit (DAU). The Transformer branch consists of a cross-attention module and a dual attention unit DAU in sequence. The cross-attention module consists of a 1x1 convolutional layer and a Softmax function, and additionally introduces a global variable M and an interpolation layer Interpolate. Step 2 is the working process and training of the reversible neural network. Includes design steps for the forward process; and design steps including the reverse process; and design steps including loss function; The design steps of the forward process include: Step 1.1 Calculate the cross color ratio map Low-light image I low As the input of the forward process of the reversible neural network, the cross color ratio formula f is first calculated ccr (·) Calculate the cross color ratio map Step 1.2 Feature extraction Low-light image I low and cross color ratio images are input into the feature encoder φ(·) and In the above example, we extract their respective features. and Step 1.3 Feature transformation feature and Input into the n stacked reversible modules θ(·) for feature conversion to obtain the converted features and Corresponding to the enhanced image features and the features of the enhanced image cross color ratio map; for the i-th reversible module θ i (·), input features and Based on the feature enhancement modules G1(·), H1(·), G2(·), H2(·), feature conversion operation is performed; based on Retienx theory, the input feature and Explained as: in represents the reflection component, Represents the illumination component, and Represents the noise component; further, the output of the feature enhancement module G2(·) Considered as noise component The output of the feature enhancement module H2(·) Considered as light component Then the reversible module θ i (·) By Remove noise components and add lighting components To convert it into features Similarly, by taking the output of the feature enhancement module G1(·) Considered as noise component The output of the feature enhancement module H1(·) Considered as light component The reversible module θ i (·) Further by removing the noise component and remove the lighting component The features Convert to Features The feature enhancement modules G1(·), H1(·), G2(·), and H2(·) are used to enhance the input features. Taking G1(·) as an example, the input feature x processing process is as follows: G1(x)=f FFN (f DBAM (x)+x)+(f DBAM (x)+x) where f DBAM (·) represents the dual-branch attention module DBAM, f FFN (·) represents the feedforward neural network FNN; The dual-branch attention module DBAM transforms the input feature x into feature x ′ ;The input feature x first passes through the maximum pooling layer f MP (·) downsamples to reduce the computational complexity of the model; then splits into features x along the channel dimension cnn and feature x trans , used as input to the CNN branch and the Transformer branch: {x cnn ,x trans }=f MP (x) The CNN branch is mainly composed of a dual attention unit DAU with residuals, which is used to cnn To enhance: x′ cnn =f DAU (x cnn ,x cnn )+x cnn The dual attention unit DAU accepts two features as input; the first feature x cnn After the dual-branch structure is enhanced, the second feature x cnn Weighted: The Transformer branch consists of a cross-attention module and a dual-attention unit DAU sequence; For the crisscross attention module, we first start from x trans Calculate the key and value in the CNN branch and use the result x′ cnn Used as a query to calculate the cross attention; introduce a memory mechanism in the cross attention module and use the global variable M∈R 1×C×3×3 To learn the brightness information of the dataset and modify the size of M using interpolation, then Q Modulate for better reconstruction: {x K ,x V }=f conv (x trans ),x Q =x′ cnn ×f interp (M) Enhanced feature x′ trans It is further input into the dual attention unit DAU for processing; the dual attention unit DAU has the same network structure as the dual attention unit DAU in the CNN branch, but the input features are different to achieve x′ cnn Weighted: in: The result x″ of the CNN branch cnn and the result x′ of the Transformer branch trans Splicing is performed along the channel dimension, then input into the 3x3 convolution for processing, and restored to the input size by interpolation: Step 1.4 Image reconstruction Enhance image features and the features of the cross color ratio map of the enhanced image Input to the feature decoding module ψ(·) to obtain the enhanced image I′ norm : The design steps of the reverse process include: Step 2.1 Calculate the cross color ratio map Normal image I norm As the input of the reversible neural network reverse process, first calculate the formula f through the cross color ratio ccr (·) Calculate the cross color ratio map Step 2.2 Feature extraction The normal image I norm and cross color ratio images are input into the feature encoder φ(·) and In the above example, we extract their respective features. and Step 2.3 Feature inverse transformation feature and Input into the n stacked reversible modules θ(·) for feature inverse transformation to obtain the inverse transformed feature and Corresponding to the features of the degraded image and the features of the cross-color ratio map of the degraded image; for the i-th reversible module θ i (·), input features and Based on the feature enhancement modules G1(·), H1(·), G2(·), H2(·), feature inverse conversion operation is performed; based on Retienx theory, the input feature and Explained as: in represents the reflection component, Represents the illumination component, and Represents the noise component; Same as the forward process, the output of the feature enhancement module G1(·) Considered as noise component The output of the feature enhancement module H1(·) Considered as light component Then the reversible module θ i (·) By and add lighting components and adding noise components To convert it into features Similarly, by taking the output of the feature enhancement module G2(·) Considered as noise component The output of the feature enhancement module H2(·) Considered as light component The reversible module θ i (·) Further by removing the illumination component and adding noise components The features Convert to Features Step 2.4 Image reconstruction Characteristics of degraded images and the features of the cross color ratio map of the degraded image Input to the feature decoding module ψ(·) to obtain the degraded image I′ low : The design steps of the loss function are: During the training phase, L1 distance is used as the loss function. For the forward and backward processes, we have: Perceptual loss is further used for constraints to improve the perceptual quality of the image: Among them VGG i (·) represents the i-th layer of VGG-19 pre-trained on ImageNet, i∈{1,3,5,9,13}; In addition, ω i is the weight corresponding to the i-th layer; In the forward process, the low-light image is denoised and edge-aware smoothing loss is introduced to improve the denoising effect: in and Represents gradients in horizontal and vertical directions; In summary, the total loss is the weighted sum of the above losses: where λ i Represents the weight, take λ1=λ4=0.05, λ2=λ3=1.

Images