A multi-branch low-light image enhancement method based on Retinex
By employing a Retinex-based multi-branch image enhancement method, which utilizes a decomposition network with shared parameters and a multi-branch enhancement network, the problems of shadows and color distortion caused by uneven illumination in low-light image enhancement are solved, achieving image enhancement effects with uniform brightness and faithful color reproduction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2023-05-22
- Publication Date
- 2026-06-02
AI Technical Summary
Existing low-light image enhancement methods are prone to producing shadow areas and color distortion when dealing with varying light intensities, and deep learning methods have difficulty distinguishing between excessively dark or excessively bright areas, resulting in unclear enhancement results.
A Retinex-based multi-branch image enhancement method is adopted. By sharing parameters between normal light and low light image decomposition networks, the reflection and illuminance components of the image are decomposed using feature extraction and enhancement modules. A color loss function is added to the multi-branch enhancement network to optimize the training process and improve the enhancement effect.
It achieves brightness uniformity and color fidelity of images under uneven lighting conditions, avoids shadow areas and color distortion, and improves the quality of image enhancement.
Smart Images
Figure CN116797474B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image enhancement technology, and in particular to a multi-branch low-light image enhancement method based on Retinex. Background Technology
[0002] In nighttime or low-light environments, images captured by cameras often suffer from low brightness, low contrast, and high noise due to the minimal light reflection from objects, making it difficult to obtain clear, bright, and high-quality image information. This is a significant issue for many applications, such as night vision, security, medical imaging, drones, and rockets.
[0003] Current methods for enhancing low-light images mainly fall into two categories. The first is traditional methods, specifically those based on histogram equalization (HUE) and Retinex theory. The former homogenizes the image's histogram to improve contrast, while the latter divides the image into reflectance and illuminance components based on prior knowledge; the enhanced image is obtained by removing the illuminance component. These methods produce poor image quality and have low universality. The second category utilizes deep learning techniques to learn the transformation patterns from low-light to normal-light images from massive datasets. A classic example is LLNET (Adeep autoencoder approach to natural low-light image enhancement). It was the first network to apply deep learning to image enhancement, proposing a stacked sparse denoising autoencoder for low-light image enhancement.
[0004] However, most existing deep learning-based methods cannot distinguish between excessively dark or excessively bright areas. If they are uniformly trained and enhanced, the enhanced results will easily show shadow areas and lose some color information, making the enhanced results unclear. Summary of the Invention
[0005] The purpose of this invention is to overcome the defects of the prior art by providing a Retinex-based multi-branch low-light image enhancement method. This invention solves the problem of uneven illumination intensity affecting the enhancement result in the prior art, so that the enhanced image has no shadow areas and uniform brightness.
[0006] The objective of this invention can be achieved through the following technical solutions:
[0007] This invention provides a Retinex-based multi-branch low-light image enhancement method, wherein the low-light image to be enhanced is input into a pre-constructed and trained image enhancement network to obtain the enhanced image;
[0008] The image enhancement network includes:
[0009] Normal light image decomposition network: The input image is a normal light image, which is used to decompose the reflection component and illuminance component of the normal light image;
[0010] Low-light image decomposition network: The input image is a low-light image corresponding to the normal light image, which is used to decompose the reflection component and illuminance component of the low-light image. The network parameters of the low-light image decomposition network and the normal light image decomposition network are shared.
[0011] Multi-branch enhancement network: The input image is a low-light image. The low-light image decomposition network trained on the low-light image decomposition network outputs a low-light image containing only the reflection component and the corresponding normal light image. This is used to enhance the low-light image containing only the reflection component, resulting in an enhanced image.
[0012] Preferably, the normal light image decomposition network and the weak light image decomposition network have the same network structure, both including:
[0013] Feature extraction module: used to extract features from the input image to obtain the first feature image;
[0014] Feature enhancement module: used to enhance the features of the first feature image to obtain the second feature image;
[0015] Component decomposition module: used to process the second feature image, realize reflectance and illumination mapping, and then obtain the reflectance component and illumination component of the input image.
[0016] Preferably, the feature enhancement module includes three 5x5 convolutional layers and three deconvolutional layers connected in sequence.
[0017] Preferably, the component decomposition module includes multiple 3*3 convolutional layers with ReLU activation functions connected in sequence, a 3*3 convolutional layer and a sigmoid function layer.
[0018] Preferably, the normal light image decomposition network and the low light image decomposition network are optimized and trained by the invariance of the two reflection components and the smoothness of the illuminance component.
[0019] Preferably, the invariance of the reflection component is represented by the Smooth L1 loss function, and the smoothness of the illuminance component is represented by the total variation minimization constraint.
[0020] Preferably, the multi-branch enhancement network includes multiple convolutional layers, the output of each convolutional layer is connected to a sub-network branch, and the outputs of all the sub-network branches are connected to a merged network branch;
[0021] The low-light image containing only the reflection component is sequentially input into each convolutional layer. The output of each convolutional layer is used as the input of the corresponding sub-network branch and the input of the next convolutional layer. The outputs of all sub-network branches are merged through a merging network branch and used as the output of the multi-branch enhancement network. The output is the enhanced image of the low-light image and the normal light image corresponding to the low-light image containing only the reflection component, which serves as the label of the multi-branch enhancement network.
[0022] Preferably, the sub-network branch includes a 3*3 convolutional layer, three 5*5 convolutional layers, and three 5*5 deconvolutional layers connected in sequence.
[0023] Preferably, the loss function used to optimize the multi-branch augmentation network includes a structure loss function, a region loss function, and a color loss function.
[0024] Preferably, the training process of the image enhancement network is as follows:
[0025] Obtain a training set, which includes normal light images and low light images. Input the normal light images and low light images into the pre-constructed normal light image decomposition network and the low light image decomposition network respectively for training. During the training process, the network parameters of the two image decomposition networks are shared.
[0026] The low-light images in the training set are input into the trained low-light image decomposition network, which outputs a low-light image containing only the reflection component. The low-light image containing only the reflection component and the normal light image corresponding to the low-light image are input into a pre-constructed multi-branch enhancement network for training, and the trained multi-branch enhancement network is obtained.
[0027] The low-light images in the training set are input into the trained low-light image decomposition network. The output low-light images containing only the reflection component are directly input into the trained multi-branch enhancement network for joint training, resulting in the final trained image enhancement network.
[0028] Compared with the prior art, the present invention has the following beneficial effects:
[0029] (1) This embodiment provides a Retinex-based multi-branch low-light image enhancement method. By using a low-light image decomposition network to decompose the low-light image into a fixed reflection component, and then using a multi-branch enhancement network to enhance the reflection component, the influence of uneven light intensity on the enhancement result is solved, so that the enhanced image has no shadow area and uniform brightness.
[0030] (2) This embodiment provides a Retinex-based multi-branch low-light image enhancement method. By sharing network parameters between the normal light image decomposition network and the low-light image decomposition network, and adding a feature enhancement module to the two decomposition networks, the image features are enhanced, making the decomposition results more accurate.
[0031] (3) This embodiment provides a Retinex-based multi-branch low-light image enhancement method. By adding a color loss function to the multi-branch enhancement network, the enhanced image is protected from color distortion. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the image enhancement network structure for a Retinex-based multi-branch low-light image enhancement method provided in this embodiment.
[0033] Figure 2 for Figure 1 The training flowchart of the embodiment shown.
[0034] Figure 3 To adopt Figure 1 The image shown in the embodiment is a comparison diagram between the original image and the obtained image. Figure 3 (a) is a low-light image. Figure 3 (b) is adopted Figure 1 The image obtained from the illustrated embodiment. Detailed Implementation
[0035] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0036] This embodiment provides a Retinex-based multi-branch low-light image enhancement method, including the following steps:
[0037] refer to Figure 2 As shown, S1: Obtain the training set.
[0038] The training set includes an artificially synthesized dataset and a real dataset. The artificially synthesized dataset is obtained through gamma correction, which adjusts the brightness of the image. The brightness of the normal light image is reduced to obtain the corresponding low-light image. These two images are a pair of paired data in the dataset. The real dataset is obtained by taking photos with a camera. The camera is fixed in one place, and photos are taken at 12 noon and when the sun is about to set. The image taken at 12 noon is the normal light image, and the image taken when the sun is about to set is the low-light image.
[0039] To increase the diversity of the dataset, the camera position can be changed, such as indoors or outdoors. A total of one thousand images were collected as the dataset, and each image was cropped to 256*256 pixels.
[0040] S2: Construct an image enhancement network;
[0041] refer to Figure 1 As shown in the figure, S normal S represents a normal light image. low Represents a low-light image. R normal I normal R represents the reflectance and illuminance components obtained from the decomposition of a normal light image. low I low This represents the reflectance and illuminance components obtained from the decomposition of a low-light image.
[0042] Image enhancement networks include:
[0043] Normal light image decomposition network: The input image is a normal light image, which is used to decompose the reflection component and illuminance component of the normal light image;
[0044] Low-light image decomposition network: The input image is a low-light image corresponding to the normal light image, which is used to decompose the reflection component and illuminance component of the low-light image. The network parameters of the low-light image decomposition network and the normal light image decomposition network are shared.
[0045] Multi-branch enhancement network: The input image is a low-light image. The low-light image decomposition network trained on the low-light image decomposition network outputs a low-light image containing only the reflection component and the corresponding normal light image. This is used to enhance the low-light image containing only the reflection component, resulting in an enhanced image.
[0046] The normal light image decomposition network and the low light image decomposition network have the same network structure, both including:
[0047] Feature extraction module: used to extract features from the input image to obtain the first feature image;
[0048] Feature enhancement module: used to enhance the features of the first feature image to obtain the second feature image;
[0049] Component decomposition module: used to process the second feature image, realize reflectance and illumination mapping, and then obtain the reflectance component and illumination component of the input image.
[0050] Specifically, in this embodiment, the feature extraction module includes a 3*3 convolutional layer, the feature enhancement module includes three 5*5 convolutional layers and three deconvolutional layers connected in sequence, and the component decomposition module includes eight 3*3 convolutional layers with ReLU activation functions, a 3*3 convolutional layer and a sigmoid function layer connected in sequence.
[0051] The input image first passes through a 3*3 convolutional layer to extract features, then through three 5*5 convolutional layers and three deconvolutional layers to enhance the features. Then, eight 3*3 convolutional layers with ReLU activation functions are used to process the feature image to achieve reflectivity and illumination mapping. Finally, a 3*3 convolutional layer and a sigmoid function are used to constrain the reflectivity and illumination components within the range of [0,1].
[0052] The multi-branch augmentation network consists of multiple convolutional layers, the output of each convolutional layer is connected to a sub-network branch, and the outputs of all sub-network branches are connected to a merged network branch.
[0053] A low-light image containing only the reflection component is sequentially input into each convolutional layer. The output of each convolutional layer serves as the input to the corresponding sub-network branch and the input to the next convolutional layer. The outputs of all sub-network branches are merged through a merging network branch and used as the output of a multi-branch enhancement network. The output is the enhanced image of the low-light image and the normal light image corresponding to the low-light image containing only the reflection component, which serves as the label of the multi-branch enhancement network.
[0054] Specifically, in this embodiment, there are 10 convolutional layers. The sub-network branches include a 3*3 convolutional layer, three 5*5 convolutional layers and three 5*5 deconvolutional layers connected in sequence. The network branches are merged into a 1*1 convolutional kernel.
[0055] The input image is passed through 10 3x3 convolutional layers. Each layer produces an output, which is then used as the input to another sub-network. This sub-network consists of one 3x3 convolutional layer, three 5x5 convolutional layers, and three 5x5 deconvolutional layers. Its function is to progressively enhance the input image. Finally, the outputs of the ten sub-networks are merged through a 1x1 convolutional kernel, and the result is used as the final output of the entire multi-branch enhancement network.
[0056] S3: Train the normal light image decomposition network and the low light image decomposition network;
[0057] The training set in S1 is divided into normal light images and corresponding low light images. The normal light images and low light images are simultaneously input into the normal light image decomposition network and the low light image decomposition network for training. The parameters of the two image decomposition networks are shared, resulting in the trained normal light image decomposition network and low light image decomposition network.
[0058] During the training of the two decomposed networks, the networks are optimized by considering the invariance of the two reflection components and the smoothness of the illuminance component. The invariance of the reflection components is represented by the Smooth L1 loss function. The smoothness of the illuminance component is represented by the total variation minimization constraint.
[0059] S4: Input the low-light images from the training set of S1 into the low-light image decomposition network trained in S3 to obtain low-light images containing only the reflection component. Input the low-light images containing only the reflection component into the multi-branch enhancement network for training. During the training process, the normal light image corresponding to the low-light image is used as the Ground Truth. The loss function composed of structural loss, region loss and YCbCr-based color loss is used as the loss function of the multi-branch enhancement network to optimize the multi-branch enhancement network.
[0060] S5: Joint Training: To further improve the overall network performance, the low-light image decomposition network and the multi-branch enhancement network are connected together for a final training (the initial parameters of both modules are the parameters of the previously trained model). The input image set is the low-light images from the training set in S1. During training, the low-light images are input from the low-light image decomposition network, and the reflection components obtained by the low-light image decomposition network are directly input into the multi-branch enhancement network. Finally, the output is obtained from the multi-branch enhancement network. This entire process constitutes one training iteration. Normal light images are used as the Ground Truth. The loss function is the same as the loss function in S4.
[0061] S6: Input the low-light image to be enhanced into the trained low-light image decomposition network to obtain the corresponding reflection component, and input the reflection component into the multi-branch enhancement network to obtain the final enhanced image.
[0062] In summary, this invention proposes a multi-branch low-light image enhancement method based on Retinex. It improves upon existing Retinex decomposition modules by adding a small sub-network composed of convolutions and deconvolutions, enhancing the image to a certain extent before decomposition, which facilitates more accurate subsequent decomposition. Furthermore, it uses the Smooth L1 loss function to represent the invariance of the reflection component. The Smooth L1 loss maintains a small gradient value, reducing the risk of gradient explosion. Through normal light image decomposition networks and low-light image decomposition networks, the influence of different light intensities can be removed. Then, multi-branch enhancement is applied to the reflection component. In this multi-branch enhancement network, a loss function based on the YCbCr color space is added to avoid color distortion in the enhanced image.
[0063] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A Retinex-based multi-branch low-light image enhancement method, characterized in that, The image enhancement method inputs the low-light image to be enhanced into a pre-constructed and trained image enhancement network to obtain the enhanced image; The image enhancement network includes: Normal light image decomposition network: The input image is a normal light image, which is used to decompose the reflection component and illuminance component of the normal light image; Low-light image decomposition network: The input image is a low-light image corresponding to the normal light image, which is used to decompose the reflection component and illuminance component of the low-light image. The network parameters of the low-light image decomposition network and the normal light image decomposition network are shared. The normal light image decomposition network and the weak light image decomposition network have the same network structure, both including: Feature extraction module: used to extract features from the input image to obtain the first feature image; Feature enhancement module: used to enhance the features of the first feature image to obtain the second feature image; the feature enhancement module includes three 5*5 convolutional layers and three deconvolutional layers connected in sequence; Component decomposition module: used to process the second feature image, realize reflectance and illumination mapping, and then obtain the reflectance component and illumination component of the input image; Multi-branch enhancement network: The input image is a low-light image. The low-light image decomposition network trained on the low-light image decomposition network outputs a low-light image containing only the reflection component and the corresponding normal light image. This is used to enhance the low-light image containing only the reflection component to obtain the enhanced image. The multi-branch enhancement network includes multiple convolutional layers, the output of each convolutional layer is connected to a sub-network branch, and the output of all the sub-network branches is connected to a merged network branch. The low-light image containing only the reflection component is sequentially input into each convolutional layer. The output of each convolutional layer serves as the input to the corresponding sub-network branch and the input to the next convolutional layer. The outputs of all sub-network branches are merged through a merging network branch and become the output of the multi-branch enhancement network. The output is the enhanced image of the low-light image, and the normal light image corresponding to the low-light image containing only the reflection component serves as the label of the multi-branch enhancement network. The sub-network branch includes a 3*3 convolutional layer, three 5*5 convolutional layers, and three 5*5 deconvolutional layers connected in sequence.
2. The Retinex-based multi-branch low-light image enhancement method according to claim 1, characterized in that, The component decomposition module includes multiple 3x3 convolutional layers with ReLU activation functions connected in sequence, a 3x3 convolutional layer and a sigmoid function layer.
3. The Retinex-based multi-branch low-light image enhancement method according to claim 1, characterized in that, The normal light image decomposition network and the low light image decomposition network are optimized and trained by taking advantage of the invariance of the two reflection components and the smoothness of the illuminance component.
4. The Retinex-based multi-branch low-light image enhancement method according to claim 3, characterized in that, The invariance of the reflection component is represented by the Smooth L1 loss function, and the smoothness of the illuminance component is represented by the total variation minimization constraint.
5. The Retinex-based multi-branch low-light image enhancement method according to claim 1, characterized in that, The loss functions used to optimize the multi-branch augmentation network include a structural loss function, a region loss function, and a color loss function.
6. The Retinex-based multi-branch low-light image enhancement method according to claim 1, characterized in that, The training process of the image enhancement network is as follows: Obtain a training set, which includes normal light images and low light images. Input the normal light images and low light images into the pre-constructed normal light image decomposition network and the low light image decomposition network respectively for training. During the training process, the network parameters of the two image decomposition networks are shared. The low-light images in the training set are input into the trained low-light image decomposition network, which outputs a low-light image containing only the reflection component. The low-light image containing only the reflection component and the normal light image corresponding to the low-light image are input into a pre-constructed multi-branch enhancement network for training, and the trained multi-branch enhancement network is obtained. The low-light images in the training set are input into the trained low-light image decomposition network. The output low-light images containing only the reflection component are directly input into the trained multi-branch enhancement network for joint training, resulting in the final trained image enhancement network.