Weak light image enhancement method based on multi-branch progressive deep network

Abstract

The application discloses a weak light image enhancement method based on a multi-branch progressive type, and comprises the following steps: step 1, preparing data; step 2, building a multi-branch progressive enhancement model; step 3, training the data prepared in step 1 according to the structure built in step 2 to obtain a trained multi-branch progressive enhancement model; and step 4, testing data based on the trained multi-branch progressive enhancement model obtained in step 3. The weak light image degradation problem is overcome by the multi-branch enhancement network of different scales.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to a low-light image enhancement method based on a multi-branch progressive deep network. Background Technology

[0002] Researchers have conducted studies on low-light enhancement in various aspects, which can be broadly categorized into spatial and frequency domain methods, Retinex theory-based methods, fusion methods, and learning-based methods. Early research primarily focused on spatial and frequency domain methods, transforming images into a specific space and modifying pixel distribution and dynamic range to enhance the image. Later, Edwin H. Land established Retinex theory, and some methods determined the true characteristics of objects by eliminating the influence of illumination light in the image. In addition, image fusion methods use different sensors to acquire many images of the same scene, extracting as much useful information as possible from each image to synthesize a high-quality image.

[0003] In recent years, the powerful fitting ability of neural networks has demonstrated outstanding performance in establishing mapping relationships between high- and low-quality images. Deep learning-based image enhancement methods have emerged in large numbers, showing great potential in learning complex end-to-end mappings from low-light images to enhanced high-quality images, and exhibiting excellent performance in color and detail restoration. To accomplish the task of low-light image enhancement, many existing methods build end-to-end networks to handle these complex degradations, while others process them step-by-step in a cascaded manner. For example, in the field of deep learning, the first method for low-light image enhancement produced satisfactory results through end-to-end learning. Some methods design multiple networks to predict different components of the target image using the low-light image, and finally integrate them to obtain the desired enhanced image; the most representative is the method based on Retinex theory. Recently, some methods have focused on designing models with multiple branches to enhance low-light images. These models offer strong flexibility and good performance, especially in complex visual tasks. Some literature proposes using multiple identical branches to simultaneously enhance low-light images, fusing the output images of different branches to obtain satisfactory results. However, this method of directly fusing images also introduces a significant amount of noise and artifacts into the target image. Summary of the Invention

[0004] The purpose of this invention is to provide a low-light image enhancement method based on a multi-branch progressive deep network. This method overcomes the low-light image degradation problem by using multi-branch enhancement networks of different scales.

[0005] The technical solution adopted in this invention is a low-light image enhancement method based on a multi-branch progressive deep network, which specifically includes the following steps:

[0006] Step 1, prepare the data;

[0007] Step 2: Build a multi-branch progressive enhancement model;

[0008] Step 3: Train the data prepared in Step 1 based on the structure built in Step 2 to obtain a trained multi-branch progressive augmentation model.

[0009] Step 4: Test the trained multi-branch progressive augmentation model obtained in Step 3 using data.

[0010] The invention is further characterized by:

[0011] The specific process of step 1 is as follows:

[0012] Step 1.1, Prepare paired low-light image pairs This indicates a low-light image that needs enhancement. The ground truth represents a high-quality image with normal brightness that corresponds one-to-one with the low-light image, where i represents the i-th data pair and N represents the number of data pairs in the dataset.

[0013] Step 1.2: Divide the image pairs obtained in Step 1.1 into two parts, namely the training set D. train and test set D test The training set The test set accounts for 97% of all data. This accounts for 3% of all data;

[0014] Step 1.3, for the training set D partitioned in Step 2 train The images are randomly cropped, retaining only 80% of the pixel data from the original images in the training set for training. Then, bicubic interpolation is used to pair the training images with {I}. L ,I H The size of the label image block I was adjusted to 256×256, and then the same bicubic interpolation method was used to resize the 256×256 label image block I. H Downsampling by 1 / 2 and 1 / 4 yields label image blocks I of 128×128 and 64×64 respectively. H2 and I H4 .

[0015] In step 2, the multi-branch progressive enhancement model includes a multi-branch enhancement structure and a progressive enhancement structure;

[0016] The multi-branch enhancement structure includes an encoder module, a multi-branch module, and a decoder module;

[0017] The encoder module consists of six convolutional and ReLU activation function layers, receiving the input low-light image I. LEnhance input From the initial low-light image I L With two enhanced images The input data is concatenated along the channel dimension to obtain the enhanced image during initialization. From the original low-light image I L Instead, therefore, the model's input Represented as:

[0018]

[0019] Will Input encoder to extract feature representations of different dimensions and scales.

[0020] The encoder is represented as:

[0021]

[0022] in, It is the input of the encoder. It is the feature extractor for the j-th branch. It is the depth feature representation output by the j-th encoder, where [W,H,C] represents the size and number of channels of the input low-light image;

[0023] The multi-branch module includes four branches, which are represented as follows:

[0024]

[0025] in, This represents the k-th enhancement branch. and These represent the input and output of the k-th enhancement branch, respectively. The output channel of the fourth branch is half the size of the input, while the input and output sizes of the other three branches are exactly the same.

[0026] The decoder module receives the outputs from four branches, fuses the enhanced features from different branches, and compresses them into three channels to generate the final result. The decoder fusion process is represented as follows:

[0027]

[0028] Among them, F up This indicates that the feature map is upsampled by 2X, and the number of channels in the feature map is compressed to half its original size, F. gz This involves compressing the fused feature map to the RGB channels to obtain an image similar to the input low-light image I. L Enhanced images of the same size

[0029] The progressive enhancement architecture includes four convolutional LSTM modules of different scales, which are embedded in a multi-branch enhancement architecture. In each enhancement branch of the multi-branch enhancement architecture, a convolutional LSTM unit is located between convolutional layers. Each convolutional LSTM unit consists of four components: a forget gate, an input gate, an output gate, and a storage unit C. t The forget gate, input gate, and output gate are represented by the following formulas (5) to (7):

[0030] C t =C t-1 ×δ(W f [h t-1 ,x t ]) (5);

[0031] C t =C t +δ(W i [h t-1 ,x t ])×Tanh(W g [h t-1 ,x t ]) (6);

[0032] h t =δ(W o [h t-1 ,x t ])×Tanh(C t (7);

[0033] Among them, h t This represents the hidden state vector of the LSTM at time step t; [h t-1 ,x t [W] is the connection channel feature map of the output at time step (t-1) and the input at time step t. f W i W g , and W o These are independent convolutions; δ and Tanh are the sigmoid activation function and the Tanh activation function, respectively.

[0034] The specific process of step 3 is as follows:

[0035] Step 3.1, preprocess the data I L With two enhanced images By concatenating the data along the channel dimension, we obtain enhanced input data;

[0036] Step 3.2: The enhanced input data obtained in Step 3.1 is processed by an encoder to extract feature representations at different scales, resulting in outputs at four scales.

[0037] Step 3.3: Before enhancing the feature representations for different branches, channel attention is used for the input features of different branches. By assigning different weight parameters along the channel dimension, the channel attention module is represented as follows:

[0038]

[0039] in, It is the original feature map extracted by the encoder. It is the output weighted feature representation, F sq This represents the global average pooling operation, where W1 and W2 are the weight parameters of the channel attention.

[0040] Step 3.4: After assigning feature weights to different channels, the weighted feature representations at the four scales are... j = 1, 2, 3, 4 are assigned to four different branches, and the feature maps are propagated forward within the corresponding branches, so that the four branches enhance the feature representations of different sizes, resulting in multiple enhanced feature representations;

[0041] Step 3.5: Use the decoder fusion module to fuse the enhanced feature representations of multiple branches into a three-channel RGB image;

[0042] Step 3.6: In the enhancement stage of the multi-branch enhancement structure, while enhancing the feature representation, the state C of the convolutional LSTM module unit of the progressive enhancement structure is also enhanced. t The output h of the LSTM at the current time step t It will be updated;

[0043] Step 3.7: After the multi-branch augmentation model completes one augmentation task, it will obtain the augmented image at the current time step.

[0044] Step 3.8: Calculate the loss between the enhancement results generated by the multi-branch enhancement structure and the progressive enhancement structure and the labeled image. The calculation method is as follows:

[0045] L=λ1L Pixel +λ2L Multi +λ3L Adv +λ4L Grad +λ5L Color (9);

[0046] Among them, L Pixel For pixel loss, L Multi For multi-scale sensing loss, L Adv To combat losses, L Grad For gradient loss, L ColorFor color loss, λ1, λ2, λ3, λ4, and λ5 are weight parameters that balance each term in the final loss function;

[0047] Step 3.9: Based on the loss obtained in Step 3.8, backpropagate the loss and complete the training after all epochs.

[0048] The specific process of step 4 is as follows:

[0049] Step 4.1: Load the trained multi-branch progressive augmentation model;

[0050] Step 4.2, Load the test dataset D divided during the data preparation phase. test ;

[0051] Step 4.3: Keeping the size of the low-light test images unchanged, input each test image into the trained multi-branch progressive enhancement model to enhance the target image. With input low-light image I L Same size;

[0052] Step 4.4, process the obtained enhanced image and label image I H The PSNR / SSIM test metric is calculated as follows:

[0053]

[0054] In the formula, L represents the maximum pixel value in the image, and I H This represents the actual label image, where N represents the number of pixels. Represents the reconstructed image;

[0055]

[0056] In the formula, μ x and μ y Reconstructed images and real image I H The average value, σ x and σ y They are respectively and I H The variance, σ xy for and I H The covariance of , with parameters C1 and C2 being constants.

[0057] The beneficial effects of this invention are as follows:

[0058] 1. This invention employs a divide-and-conquer approach in the process of enhancing low-light images, decomposing the enhancement process into four different scales and performing the enhancement process simultaneously at each scale. This avoids the noise amplification or contrast loss problems caused by cascaded networks enhancing low-light images sequentially, and can still recover more details and brightness in extremely low-brightness environments.

[0059] 2. This invention proposes a coarse-to-fine progressive enhancement strategy. It uses a convolutional LSTM structure to decompose the low-light image enhancement problem vertically, dividing the enhancement process from low-light image to normal brightness image into multiple stages. The fusion output of each multi-branch network is fed back into the multi-branch network. The LSTM structure is used to combine the enhancement effects of multiple stages, refining the fitting problem between the complex low-light image and the high-quality image.

[0060] 3. This invention sets up a rich joint loss function. Simulation results show that, compared with existing low-light image enhancement algorithms, this invention can recover more image details, produce clearer image edges and textures, and avoid obvious image distortion and artifacts. Attached Figure Description

[0061] Figure 1 This invention relates to a low-light image enhancement method based on a multi-branch progressive deep network, which employs a low-light image enhancement structure based on a multi-branch network.

[0062] Figure 2 This invention is a low-light image enhancement method based on a multi-branch progressive deep network and a progressive enhancement strategy structure based on convolutional LSTM.

[0063] Figure 3 This is a visualization of the progressive enhancement strategy at each stage of the low-light image enhancement method based on a multi-branch progressive deep network, as described in this invention.

[0064] Figure 4 (a) to (j) are visual comparison results of the low-light image enhancement method based on multi-branch progressive deep networks in this invention and existing algorithms after testing on a public dataset;

[0065] Figure 5 (a) to (i) are visual comparison results of the low-light image enhancement method based on multi-branch progressive deep network of the present invention and existing algorithms after testing images taken by mobile phones.

[0066] Figure 6 This is a comparison chart of the low-light image enhancement method based on multi-branch progressive deep networks of this invention and existing algorithms in terms of model parameter statistics and test metrics on public datasets. Detailed Implementation

[0067] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0068] This invention presents a low-light image enhancement method based on a multi-branch progressive deep network. The technical approach is as follows: a training set is constructed using a large number of natural or artificially synthesized low-light images and labeled images. Each time a low-light image is input, it is cropped to 80% of its original size to augment the dataset. Four branches with different structures are designed to perform feature enhancement and fusion at different scales and levels. A convolutional LSTM is used to alleviate the training difficulties caused by excessive network depth. The mapping relationship from low-light images to target images is learned through the multi-branch progressive deep network. (Refer to...) Figure 1 and Figure 2 The implementation steps of the multi-branch progressive low-light image enhancement method mainly include four stages: data preparation stage, building the multi-branch enhancement structure and progressive enhancement model stage, training stage, and testing stage.

[0069] (I) Data Preparation Stage

[0070] Step 1: Constructing the dataset of image pairs for training the multi-branch enhancement and progressive enhancement stages: First, prepare paired low-light image pairs. This indicates a low-light image that needs enhancement. The ground truth represents a high-quality image with normal brightness that corresponds one-to-one with the low-light image, where i represents the i-th data pair and N represents the number of data pairs in the dataset.

[0071] Step 2, Data Partitioning: Divide the image data obtained in Step 1 into two parts, namely the training set D. train and test set D test The training set The test set accounts for 97% of all data. It accounts for 3% of all data.

[0072] Step 3, for the training set D divided in Step 2 train Data preprocessing: For the training set D train The images in the dataset are randomly cropped, retaining only 80% of the pixel data from the original images in the training set for training. Then, bicubic interpolation is used to pair the training images with {I... L ,I H The size of the label image block I was adjusted to 256×256, and then the same bicubic interpolation method was used to resize the 256×256 label image block I. H Downsampling by 1 / 2 and 1 / 4 yields label image blocks I of 128×128 and 64×64 respectively. H2 and I H4 .

[0073] (ii) Construct a multi-branch progressive enhancement model, including a multi-branch enhancement structure and a progressive enhancement structure;

[0074] The multi-branch reinforcement structure proposed in this invention is as follows: Figure 1 As shown, it includes three main parts: an encoder module, a multi-branch module, and a decoder module.

[0075] (1) The encoder module consists of six convolutional and ReLU activation function layers, receiving the input low-light image I. L Enhance input From the initial low-light image I L With two enhanced images The input data is concatenated along the channel dimension to obtain the enhanced image during initialization. From the original low-light image I L Therefore: the model's input It can be represented as:

[0076]

[0077] After that Input encoder to extract feature representations of different dimensions and scales.

[0078] An encoder can be represented as:

[0079]

[0080] in, It is the input of the encoder. It is the feature extractor for the j-th branch. This is the depth feature representation output by the j-th encoder, where [W, H, C] represents the size and number of channels of the input low-light image. For j = 1, 2, 3, 4, The dimensions and number of channels are [W,H,64], and

[0081] (2) Multi-branch modules are Figure 1Branches 1, 2, 3, and 4 are defined in the model. These four branches enhance the features extracted by the encoder at different depths. Inspired by U-Net and Res-Net, this invention designs four branches with different receptive fields to learn depth representations at different scales. More specifically, the first branch is used to handle relatively simple tasks, and therefore has the simplest structure compared to the other three branches. The structure of the second branch is similar to U-Net, but it has unique differences in skip connections and feature map size. Correspondingly, the structure of the third branch is similar to Res-Net, but it removes the batch normalization layer and retains only a small number of res-blocks to reduce the number of model parameters. The fourth branch is responsible for performing relatively complex enhancement tasks, which integrates deeper regular layers and four dilated convolutional layers to learn more informational feature representations. Formally, these four branches are represented as:

[0082]

[0083] in, This represents the k-th enhancement branch. and These represent the input and output of the k-th enhancement branch, respectively. The output channel of the fourth branch is half the size of the input, while the input and output sizes of the other branches are exactly the same.

[0084] (3) The decoder module receives the outputs from the four branches, fuses the enhanced features from different branches, and compresses them into three channels to generate the final result. The decoder fusion process can be represented as follows:

[0085]

[0086] Among them, F up This indicates that the feature map is upsampled by 2X, and the number of channels in the feature map is compressed to half its original size, F. gz This involves compressing the fused feature map to the RGB channels to obtain an image similar to the input low-light image I. L Enhanced images of the same size

[0087] The progressive enhancement structure proposed in this invention is as follows: Figure 2 As shown: This invention constructs four convolutional LSTM modules of different scales and embeds them into the branch network. In each enhancement branch, the convolutional LSTM unit is located between convolutional layers. Each convolutional LSTM unit consists of four components: a forget gate (Equation (5)), an input gate (Equation (6)), an output gate (Equation (7)), and a storage unit C. t The details are as follows:

[0088] C t =Ct-1 ×δ(W f [h t-1 ,x t ]) (5);

[0089] C t =C t +δ(W i [h t-1 ,x t ])×Tanh(W g [h t-1 ,x t ]) (6);

[0090] h t =δ(W o [h t-1 ,x t ])×Tanh(C t (7);

[0091] Among them, h t This represents the hidden state vector of the LSTM at time step t; [h t-1 ,x t [ ] is the connection channel feature map between the output at time step (t-1) and the input at time step t. W f W i W g , and W o These are independent convolutions; δ and Tanh are the sigmoid activation function and the Tanh activation function, respectively. At the start of training, C... t and h t All elements are initialized to 1 and 0 respectively. G1, G2, and G3 are multi-branch models with the same structure and weights, but different time steps during iteration. At consecutive time steps in the network, adjacent sub-networks are connected by convolutional LSTM units. For example... Figure 3 As shown: Progressive enhancement results at different time steps, with the enhanced image at the last time step. To ultimately enhance the results

[0092] (III) Training Phase

[0093] Step 1, firstly, process the preprocessed data I from the data preparation stage. L With two enhanced images splicing in the channel dimension We obtain enhanced input data.

[0094] Step 2, then the enhanced input data is passed through the encoder. Extract feature representations at different scales. This yields outputs at four scales. j = 1, 2, 3, 4.

[0095] Step 3: Before enhancing the feature representation for different branches, channel attention is used for the input features of different branches. By assigning different weight parameters along the channel dimension, the channel attention module is represented as follows:

[0096] in, j = 1, 2, 3, 4 are the original feature maps extracted by the encoder. j = 1, 2, 3, 4 are the weighted feature representations of the output, F sq This represents the global average pooling operation, where W1 and W2 are the channel attention weights.

[0097] Step 4: After assigning feature weights to different channels, the weighted feature representations at the four scales are obtained. The j = 1, 2, 3, 4 are assigned to four different branches. The feature maps are propagated forward within the corresponding branches, which enhances the feature representations of different sizes in the four branches, resulting in multiple enhanced feature representations.

[0098] Step 5: Use the decoder fusion module to fuse the enhanced feature representations of multiple branches into a three-channel RGB image.

[0099] Step 6: In the enhancement stage of the multi-branch enhancement structure, while enhancing the feature representation, the state C of the convolutional LSTM module unit in the progressive enhancement stage... t The output h of the LSTM at the current time step t It will be updated.

[0100] Step 7: After the multi-branch augmentation model completes one augmentation task, it will obtain the augmented image at the current time step. Use the updated version C t h t Repeat steps 1 through 5 of the training phase three times.

[0101] Step 8: Calculate the loss between the augmentation results generated by the multi-branch network and the progressive network and the labeled image. The calculation method is as follows:

[0102] L=λ1L Pixel +λ2L Multi +λ3L Adv +λ4L Grad +λ5L Color (9);

[0103] The five losses are pixel loss L Pixel Multi-scale sensing loss L MultiCombating Losses L Adv Gradient loss L Grad Color loss L Color λ1, λ2, λ3, λ4, and λ5 are the weight parameters for each term in the final loss function. Here, λ1, λ2, λ3, λ4, and λ5 are set to 1.5, 1, 0.001, 1, and 0.5, respectively.

[0104] The calculation process for each loss function is as follows:

[0105] L Pixel The loss is represented as:

[0106]

[0107] Among them, I H It is the ground truth. It is the i-th enhanced image of the progressive network, L MSE It is a metric that measures the difference between two images, where N is the number of iterations in progressive enhancement;

[0108] L Multi The loss is represented as:

[0109]

[0110] Among them, L VGG (·) represents the MSE of the VGG19 network between the feature maps of the predicted image and its target image in layers 2 and 7; I H ,I H2 ,I H4 These are the original, 1 / 2, and 1 / 4 scale reference ground truths, respectively. These are the original, 1 / 2, and 1 / 4 scale enhanced images, respectively, with weight constants γ2 and γ4 set to 0.8 and 0.4, respectively.

[0111] Adverse loss L Adv Represented as:

[0112] L Adv =L MSE (D(I L ,G(I L ;θ)),[1]) (12);

[0113] L Grad The calculation process is expressed as follows:

[0114]

[0115] in, and Represents gradient operators along the horizontal and vertical directions;

[0116] L Color The calculation process is expressed as follows:

[0117]

[0118] Where ∠ represents the cosine similarity between the pixel-level color vectors of the enhanced image and its target image, and p is the p-th pixel of the enhanced image or its target image.

[0119] Step 9: After calculating the overall loss, backpropagate the loss. Here, the Adam optimizer is used, with the learning rate set to 0.0004 for the first 50 epochs and then decreasing to 0.0002. The process is repeated 200 times, and training is complete after all epochs have been completed.

[0120] (IV) Testing Phase

[0121] Step 1: Load the trained multi-branch progressive augmentation model.

[0122] Step 2, Load the test dataset D divided during the data preparation phase. test .

[0123] Step 3: Keeping the size of the low-light test images unchanged, input each test image into the trained multi-branch progressive enhancement model to enhance the target image. With input low-light image I L Same size.

[0124] Step 4, process the obtained enhanced image and label image I H The calculation method for the test index PSNR (Precision Performance Index) / SSIM (Precision Performance Index) is as follows:

[0125]

[0126] In the formula: L represents the maximum pixel value in the image, I H This represents the actual label image, where N represents the number of pixels. This represents the reconstructed image. The smaller the pixel difference between the two images, the higher the PSNR value.

[0127]

[0128] Where: μ x and μ y Reconstructed images and real image I H The average value, σ x and σ y They are respectively and I HThe variance, σ xy for and I H The covariance of the SSIM is calculated using constants C1 and C2 to prevent the denominator from being zero. The SSIM value ranges from [0, 1]. Generally, the larger the SSIM value, the more similar the two images are.

[0129] For low-light images where it is impossible to acquire label images, allow I H It does not exist; it only generates an enhanced target image. Test metrics are not calculated.

[0130] The effectiveness of the present invention is verified through simulation experiments.

[0131] Simulation content:

[0132] (1) On the same training set and test images, a comparative experiment was conducted, selecting seven representative low-light image enhancement methods and comparing them with the simulation results of this invention to verify the effectiveness of this invention. For example... Figure 4 As shown in (a) to (j), seven representative low-light image enhancement methods are: the method proposed by Lv et al., abbreviated as MBLLEN; the method proposed by Wang et al., abbreviated as DeepUPE; the method proposed by Moran et al., abbreviated as DeepLPF; the method proposed by Zamir et al., abbreviated as MIRNet; the method proposed by Zhao et al., abbreviated as RetinexDIP; the method proposed by Yang et al., abbreviated as SGM-Net; and the method proposed by Wang et al., abbreviated as LLFlow.

[0133] (2) To further verify the versatility of the present invention, low-light images were captured using a smartphone under different brightness and noise conditions, and the enhancement results obtained by different methods were compared, such as... Figure 5 Shown in (a)~(i).

[0134] (3) To compare the number of parameters of the models with their low-light image enhancement performance, the number of parameters of the seven models and the test metrics on a public dataset were calculated and compared, such as... Figure 6 As shown.

[0135] For specific simulation conditions, please refer to the description of each embodiment.

[0136] Example 1

[0137] The invention was trained and tested against seven other methods on three widely used databases (LOL, SYS, and MIT), and the results are as follows: Figure 4 As shown: Figure 4 (a) is the input low-light image. Figure 4 (b) shows the enhanced results of the MBLLEN method. Figure 4(c) shows the enhanced result of the DeepUPE method. Figure 4 (d) shows the enhanced results of the DeepLPF method. Figure 4 (e) shows the enhanced result of the MIRNet method. Figure 4 (f) shows the enhanced result of the RetinexDIP method. Figure 4 (g) shows the enhanced result of the SGM-Net method. Figure 4 (h) is the enhanced result of the LLFlow method. Figure 4 (i) is the enhanced result of this method. Figure 4 (j) is the actual label image.

[0138] Figure 4 Simulation results show that, comparing the results of the first and second rows on the MIT database, most comparison methods produce relatively satisfactory results. In contrast, this invention performs better in generating finer texture details (see [link to simulation results]). Figure 4 The first line in the text), and shows better correction in sky color (see the first line). Figure 4 (See the second row in the database). Comparing our method to unnatural low-light images in the SYS database, referring to the third and fourth rows, we find that our method produces the most convincing results among the compared algorithms. In particular, the fourth row of sunsets generated by our invention most closely resembles the target image, exhibiting better human-eye perceptual features than other methods. In contrast, other methods produce very severe color shifts (see...). Figure 4 The third row (d), (f), (g), (h)) and the blur effect (see Figure 4 The images in the third row (b), (c), and (e)). The images in the LOL database were taken indoors, are very dark and noisy, which is very challenging for many low-light image enhancement methods. This can be seen from the comparison results (see...). Figure 4 (See lines 5 and 6). This invention produces the most realistic results with minimal artifacts and noise in the main regions. Furthermore, the brightness, contrast, and color of the enhanced results are most realistic to the target image. In contrast, other comparative methods tend to produce significant amounts of unexpected halos in edge and textured areas (see lines 5 and 6). Figure 4 In the last two lines (b) to (e), some methods even produce noticeable color distortion throughout the entire reconstruction result (see...). Figure 4 (c), (e), and (h) in the last two lines). Thanks to its carefully designed multi-branch structure and the applied progressive enhancement strategy, the proposed method can handle a variety of complex degradations in very dark environments and under low-light conditions, demonstrating sufficient enhancement in terms of brightness, color, contrast, and noise.

[0139] Example 2

[0140] Low-light images under varying brightness and noise conditions were captured using a smartphone, and the enhancement results obtained by different methods were intuitively evaluated. The results are as follows: Figure 5 As shown in (a)~(i): Figure 5 (a) is the input low-light image. Figure 5 (b) shows the enhanced results of the MBLLEN method. Figure 5 (c) shows the enhanced result of the DeepUPE method. Figure 5 (d) shows the enhanced results of the DeepLPF method. Figure 5 (e) shows the enhanced result of the MIRNet method. Figure 5 (f) shows the enhanced result of the RetinexDIP method. Figure 5 (g) shows the enhanced result of the SGM-Net method. Figure 5 (h) is the enhanced result of the LLFlow method. Figure 5 (i) is an enhanced result of the present invention.

[0141] from Figure 5 The simulation results in (a) to (i) show that the present invention is significantly superior to other methods in all aspects: 1) it exhibits satisfactory performance in restoring brightness and color, 2) it perfectly preserves more details, and 3) it has excellent robustness to various noises.

[0142] Example 3

[0143] Achieving a better balance between performance and computational complexity is key to evaluating model efficiency. Sorting the model by the number of parameters in ascending order helps assess the performance and computational efficiency of different methods, such as... Figure 6 As shown, the horizontal axis represents network parameters (MB), and the vertical axis represents the average PSNR value across the three databases.

[0144] from Figure 6 The simulation results show that the present invention leads other methods in terms of performance enhancement while maintaining a moderate scale of computational cost.

[0145] The results of the three experiments above show that, compared with existing low-light image enhancement methods, the present invention has stronger image enhancement capabilities, can obtain more image details, and the restored image has a better visual perception effect, without obvious distortion and artifacts, and is closer to the real image.

Claims

1. A weak light image enhancement method based on a multi-branch progressive deep network, characterized in that: Specifically comprising the following steps: Step 1, preparing data; Step 2, building a multi-branch progressive enhancement model; in the step 2, the multi-branch progressive enhancement model comprises a multi-branch enhancement structure and a progressive enhancement structure; The multi-branch enhancement structure comprises an encoder module, a multi-branch module and a decoder module; The encoder module includes six convolutions and an activation function layer that receives the input low-light image , enhances the input from the initial low-light image with two enhanced images concatenated in the channel dimension, resulting in enhanced input data, which is initialized with the enhanced images from the original low-light image instead, so the input to the model is represented as: （1）； Will Input encoder to extract feature representations of different dimensions and scales. ; The encoder is expressed as: （2）； wherein, is an input to the encoder, is a feature extractor of the th branch, is a deep feature representation of the th encoder output, denotes the size and number of channels of the input low-light image; The multi-branch module comprises four branches, which are expressed as: （3）； wherein, represents the first enhanced branch, and represent the input and output of the first enhanced branch, respectively, the output channel of the fourth branch is half of the input, while the input and output dimensions of the remaining three branches are exactly the same; The decoder module receives the outputs of the four branches, fuses the enhanced features of different branches, and compresses them into three channels to generate the final result, and the process of decoder fusion is expressed as: （4）； wherein, represents upsampling the feature map and compressing the channel number of the feature map to half of its original size, is an operation of compressing the fused feature map to channels, obtaining an enhanced image of the same size as the input weak light image ;​ The progressive enhancement structure comprises four convolution LSTM modules of different scales, the four convolution LSTM modules of different scales are embedded into a multi-branch enhancement structure, in each enhancement branch of the multi-branch enhancement structure, a convolution LSTM unit is located between convolution layers, and each convolution LSTM unit is composed of four components, i.e., a forgetting gate, an input gate, an output gate and a storage unit The forgetting gate, the input gate and the output gate are shown in the following formulas (5)-(7): （5）； （6）； （7）； wherein, denotes the hidden state vector of the LSTM at the time step; is the output of the LSTM at the time step and the concatenated channel feature maps of the input at the time step, are independent convolutions; and are an activation function and an activation function, respectively;​​​ Step 3, training the data prepared in step 1 according to the structure built in step 2 to obtain a trained multi-branch progressive enhancement model; Step 4, testing data based on the trained multi-branch progressive enhancement model obtained in step 3.

2. The method of claim 1, wherein the multi-branch progressive deep network is a multi-branch residual network. The specific process of the step 1 is: Step 1.

1. Prepare pairs of low-light image , denotes a low-light image that needs to be enhanced, denotes a high-quality image of normal brightness that corresponds to the low-light image ground truth, denotes the pair of data, denotes the number of data pairs in the dataset; Step 1.2, divide the image pair data obtained in step 1.1 into two parts, which are training set and test set , wherein the training set accounts for 97% of the total data, and the test set accounts for 3% of the total data; Step 1.3, for the training set divided in Step 2 The images are randomly cropped, retaining only 80% of the pixel data from the original images in the training set for training. Then, bicubic interpolation is used to pair the training images. Size adjustment Then, using the same bicubic interpolation method, respectively... Tag image blocks Downsampling by 1 / 2 and 1 / 4 yielded the following results: and Tag image blocks and . 3.The weak light image enhancement method based on the multi-branch progressive deep network according to claim 2, characterized in that: The specific process of the step 3 is: Step 3.1, data after pre-processing with two enhanced images concatenated in the channel dimension, resulting in enhanced input data; Step 3.2, the enhanced input data obtained in step 3.1 is passed through an encoder to extract different scale feature representations, resulting in four scales of output ; Step 3.3, before enhancing the feature representation of different branches, using channel attention for the input features of different branches Different weight parameters are assigned in the channel dimension, and the channel attention module is represented as: （8）； wherein, is the original feature map extracted by the encoder, is the output weighted feature representation, denotes a global average pooling operation, wherein: and are the weight parameters of the channel attention. Step 3.4, after giving different channel characteristic weights, the weighted feature representation of four scales The feature map is respectively given to four different branches and forward propagated in the corresponding branch, so that the four branches enhance the feature representation of different sizes, and a plurality of branch-enhanced feature representations are obtained. Step 3.5, using the decoder fusion module to fuse the enhanced feature representations of multiple branches into a three-channel RGB image; Step 3.6, while enhancing the representation of the multi-branch enhanced structure at the enhancement stage of the enhanced structure, the convolutional LSTM module unit state of the progressive enhanced structure and the output of the current time step LSTM will be updated; Step 3.7, after the multi-branch enhancement model completes an enhancement task once, an enhanced image of the current time step is obtained ; Step 3.8, calculating the loss between the enhanced results generated by the multi-branch enhancement structure and the progressive enhancement structure and the label image, and the calculation method is as follows: （9）； wherein, is a pixel loss, is a multi-scale perception loss, is an adversarial loss, is a gradient loss, is a color loss, wherein , , , , and are weight parameters balancing each term in the final loss function. Step 3.9, according to the loss obtained in step 3.8, the loss is back propagated, and the training is completed after all epochs are iterated.

4. The weak light image enhancement method based on the multi-branch progressive deep network according to claim 3, characterized in that: The specific process of the step 4 is: Step 4.1, loading the trained multi-branch progressive enhancement model; Step 4.2, loading data preparation phase divided test data set ; Step 4.3, the size of the tested low-light image remains unchanged, and the trained multi-branch progressive enhancement model is input with the test image one by one to obtain the enhanced target image is the same as the input low-light image size; Step 4.4, the obtained enhanced image is subjected to a histogram equalization process and the label image The test index PSNR / SSIM is calculated in the following manner: （10）； wherein denotes the maximum value of a pixel in the image, denotes the real label image, denotes the number of pixels, denotes the reconstructed image; （11）； wherein and are the average values of the reconstructed image and the real image respectively, and are the variances of the reconstructed image and the real image respectively, is the covariance of the reconstructed image and the real image and the parameters C1, C2 are constants.

Images