Weak light image enhancement method and system based on illumination decomposition

Abstract

The invention discloses a weak light image enhancement method and system based on illumination decomposition, and belongs to the technical field of artificial intelligence and computer vision. A weak light image is preliminarily enhanced through an illumination decomposable weak light enhancement model based on coordinate attention, and then the enhanced image and an original weak light image are simultaneously input into an unsupervised denoising enhancement model for re-enhancement and denoising; the unsupervised denoising enhancement model improves the generalization and anti-noise performance of the network, can better process the noise from various environments, improves the performance of the network under complex conditions, and further improves the performance of the method in a weak light image enhancement task.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and specifically relates to a low-light image enhancement method and system based on illumination decomposition. Background Art

[0002] Low-light images are usually taken or captured under conditions of insufficient lighting. Insufficient lighting can greatly reduce the visual quality of the image, such as loss of detail, low contrast, and increased noise. This not only affects the subjective experience of viewing the image, but also the existing visual systems are often designed for normal lighting images, which also affects the processing performance of many computer vision tasks. Factors that contribute to the low quality of low-light images include insufficient ambient lighting, poor photosensitivity of low-end imaging equipment, and unreasonable exposure configuration of the camera equipment. In order to restore image details and improve image quality, researchers have proposed a variety of low-light image enhancement and denoising methods.

[0003] In recent years, the low-light enhancement technology based on deep learning has attracted the attention of many people. The method based on deep learning has better effect than the traditional low-light image enhancement method, and has good generalization and adaptability to various scenes. The RetinexNet model includes a decomposition network (Decom) for decomposition and an enhancement network for lighting adjustment (Relight). First, the decomposition network decomposes the low-light image into a reflectance map and an illumination map; then the enhancement network enhances and adjusts the image. However, the enhanced image lacks details, the network model is large, and the calculation is time-consuming. The KinD model is similar to Retinex-Net. It decomposes the image into illumination component and reflection component, and uses convolutional neural network to process the illumination component and reflection component respectively, which improves the problems in RetinexNet, but its model size is much larger than RetinexNet. The SSIENet model decomposes the image based on the Retinex theory, uses the result after histogram equalization as a reference, and believes that the maximum channel of the reflectance is consistent with the maximum channel of the low-light image and has the largest entropy. On this basis, a simple network can achieve better results, but the enhanced image has color difference and detail loss. Inspired by retinex, Liu et al. first established an enhancement network by unfolding the optimization process of the Retinex-inspired model; then introduced a priori module search space based on distillation pool; finally, proposed a cooperative two-layer search strategy to discover the required illumination estimation and noise removal architecture. The network requires a very small model size, floating point calculation number and time. Although the image details are enhanced, visible noise also appears, which damages the image quality. Based on the above low-light image enhancement models, they have a large number of parameters and the problem of noise in the enhanced images, which makes them difficult to be directly applied to mobile terminals or to provide guarantees for back-end tasks. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a low-light image enhancement method and system based on illumination decomposition, which solves the problem that the existing low-light image enhancement method has a large model, a large number of parameters, and cannot be directly used for mobile terminals or back-end tasks.

[0005] The present invention is achieved through the following technical solutions:

[0006] A low-light image enhancement method based on illumination decomposition comprises the following steps:

[0007] Step 1: Construct an illumination decomposable weak light enhancement model based on coordinate attention, which includes an illumination decomposition and learning module and a reflection component self-calibration module;

[0008] The illumination decomposition learning module embeds a coordinate attention mechanism in the feature extraction layer, the feature extraction layer obtains a feature map of the original low-light image, and uses the coordinate attention mechanism to embed the position information of the original image into the feature map to obtain an illumination map of the original low-light image, and divides the original low-light image by the illumination map to obtain a reflectance map of the original low-light image;

[0009] The reflection component self-calibration module obtains the reflection component of the reflection image, combines the reflection component with the original low-light image, and obtains an enhanced low-light image;

[0010] The enhanced low-light image is input into the illumination decomposition learning module, and the reflectance component self-calibration module is repeatedly executed until the number of iterations is reached to obtain the reflectance map of the preliminary enhanced image;

[0011] Step 2: Build an unsupervised denoising enhancement model;

[0012] The unsupervised denoising enhancement model uses upsampling and downsampling to denoise the final reflectance map output by the illumination decomposition learning module, and jump-connects the feature maps obtained from different sampling layers to fuse the feature maps from different sampling layers to obtain the final enhanced image.

[0013] Preferably, the method for extracting the illumination map of the original low-light image in step 1 is as follows:

[0014] The illumination decomposition learning module includes at least four convolutional layers;

[0015] The first convolutional layer contains at least one 2D convolution, relu activation function and coordinate attention;

[0016] The second convolutional layer has the same structure as the third convolutional layer, including at least one 2D convolution, normalization and relu activation function;

[0017] The fourth convolutional layer contains at least one 2D convolution and a sigmoid activation function.

[0018] Preferably, the method for extracting the illumination map of the original low-light image in step 1 is as follows:

[0019] Each channel of the feature sequence of the low-light image is encoded, and then the obtained position code is concat-operated and transformed to obtain a feature map. The feature map is normalized and relu activated, and then decomposed into two separate tensors along the spatial dimension. The two tensors are transformed into tensors with the same number of channels, keeping the dimension of the input tensor unchanged, and the sigmoid activation function is performed on the two obtained tensors to obtain two attention weight tensors and use them to weight the input tensor. The two weighted input tensors are multiplied with the initial input to obtain the illumination map.

[0020] Preferably, the number of channels of the tensor transformation is reduced using a reduction ratio.

[0021] Preferably, the reflection component self-calibration module comprises at least six convolutional layers;

[0022] The first convolutional layer includes a 2D convolution, normalization, relu activation function and coordinate attention;

[0023] The last convolutional layer includes a two-dimensional convolution and a sigmoid activation function;

[0024] The structure of the first convolutional layer is the same as that of the multiple convolutional layers between the last convolutional layer, including two sets of 2D convolutions, normalization, and relu activation functions.

[0025] Preferably, a loss function is constructed and a back-propagation algorithm is used to train the illumination decomposable low-light enhancement model and the unsupervised denoising enhancement model until the model converges to obtain a trained lightweight model for low-light image enhancement;

[0026] The loss function includes fidelity loss, smoothness loss, color constancy loss and reflection loss.

[0027] Preferably, the loss function is expressed as follows:

[0028] L total =1.5·L f +L s +2·L color +L R

[0029] Among them, L f is the fidelity loss, L s is the smoothing loss, L color is the color constancy loss, L R is the reflection loss.

[0030] Preferably, the expression of the fidelity loss is as follows:

[0031]

[0032] Among them, T is the total number of stages, y+s t-1 Input lighting, x t Output lighting;

[0033] The expression of the smoothing loss is as follows:

[0034]

[0035] Where N is the total number of pixels, i is the i-th pixel, N(i) represents the neighboring pixels of i in the window, ω i,j represents weight;

[0036]

[0037] Where c is the image channel in the YUV color space, σ is the standard deviation of the Gaussian kernel;

[0038] The expression of the color constancy loss is as follows:

[0039]

[0040] The expression of the reflection loss is as follows:

[0041]

[0042] Where F(x) is the histogram equalization operator of image x, λ is the weight parameter, is the gradient operator.

[0043] A system for a weak-light image enhancement method based on illumination decomposition, comprising:

[0044] An illumination decomposable weak light enhancement model construction module is used to construct an illumination decomposable weak light enhancement model based on coordinate attention, which includes an illumination decomposition and learning module and a reflection component self-calibration module;

[0045] The illumination decomposition learning module embeds a coordinate attention mechanism in the feature extraction layer, the feature extraction layer obtains a feature map of the original low-light image, and uses the coordinate attention mechanism to embed the position information of the original image into the feature map to obtain an illumination map of the original low-light image, and divides the original low-light image by the illumination map to obtain a reflectance map of the original low-light image;

[0046] The reflection component self-calibration module obtains the reflection component of the reflection image, combines the reflection component with the original low-light image, and obtains an enhanced low-light image;

[0047] The enhanced low-light image is input into the illumination decomposition learning module, and the reflectance component self-calibration module is repeatedly executed until the number of iterations is reached to obtain the reflectance map of the preliminary enhanced image;

[0048] Unsupervised denoising enhancement model building module, used to build an unsupervised denoising enhancement model;

[0049] The unsupervised denoising enhancement model uses upsampling and downsampling to denoise the final reflectance map output by the illumination decomposition learning module, and jump-connects the feature maps obtained from different sampling layers to fuse the feature maps from different sampling layers to obtain the final enhanced image.

[0050] Compared with the prior art, the present invention has the following beneficial technical effects:

[0051] The present invention provides a low-light image enhancement method based on illumination decomposition, which adopts a staged reflection component self-calibration learning technology to ensure the effectiveness of the low-light image enhancement network. Through this technology, the network can ensure that the outputs of different stages can converge to the same state during the training stage, thereby improving the convergence of the network. At the same time, the self-calibration module is independent of the prediction stage, which greatly reduces the number of parameters required for the network model and makes the network more lightweight. In addition, in order to further improve the generalization and anti-noise performance of the network, the present invention designs an unsupervised anti-noise enhancement module. Through this module, the network can learn to suppress noise and enhance low-light images without clear labels, thereby further improving the performance of the low-light image enhancement of the present invention. Finally, the loss function of the present invention introduces a calibrable color deviation loss term, so that the enhanced image color is more realistic and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the low-light image enhancement method based on illumination decomposition of the present invention;

[0053] Figure 2 It is a flow chart of the lightweight learning network of the present invention;

[0054] Figure 3 This is the structural diagram of the coordinate attention module of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below in conjunction with the accompanying drawings, which are intended to explain the present invention rather than to limit it.

[0056] See also Figure 1 , a low-light image enhancement method based on illumination decomposition, comprising the following steps:

[0057] Step 1: Construct an illumination decomposable weak light enhancement model based on coordinate attention, which includes an illumination decomposition and learning module and a reflection component self-calibration module;

[0058] The illumination decomposition learning module embeds a coordinate attention mechanism in the feature extraction layer, the feature extraction layer obtains a feature map of the original low-light image, and uses the coordinate attention mechanism to embed the position information of the original image into the feature map to obtain an illumination map of the original low-light image, and divides the original low-light image by the illumination map to obtain a reflectance map of the original low-light image;

[0059] The reflection component self-calibration module obtains the reflection component of the reflection image, combines the reflection component with the original low-light image, and obtains an enhanced low-light image;

[0060] The enhanced low-light image is input into the illumination decomposition learning module, and the reflectance component self-calibration module is repeatedly executed until the number of iterations is reached to obtain the reflectance map of the preliminary enhanced image;

[0061] The illumination decomposition learning module is used to learn the illumination residual in stages, which greatly reduces the computational difficulty. The reflectance component self-calibration module is used to learn the reflectance component, bridge the input of each stage with the weak light observation, indirectly explore the convergence behavior between each stage, and accelerate the convergence of the entire network. The entire network is carried out in stages. The network adds coordinate attention. By embedding position information in the channel attention, it not only improves the representation ability of the model, but also maintains a low computational cost, making its application in lightweight networks more feasible and effective. In the traditional attention mechanism, the channel dimension of the input feature is usually weighted to enhance the model's attention to different channels. This makes the model more flexible in processing features and can selectively focus on the channel information that is most important for the current task. Unlike traditional channel attention, coordinate attention embeds position information in the channel attention mechanism. Specifically, coordinate attention associates each position in the feature map with its corresponding coordinate information, and combines the position information with the channel attention weight so that the model can consider the importance of each position while paying attention to the channel. Coordinate attention achieves position attention weighting by considering position information while calculating the channel attention weight. This means that when calculating the attention weight of each channel, the model not only takes into account the relationship between channels, but also the relationship between different positions. This enables the model to more finely adjust the feature representation of each position, thereby improving the model's ability to process spatial information.

[0062] Step 2: Build an unsupervised denoising enhancement model;

[0063] The unsupervised denoising enhancement model uses upsampling and downsampling to denoise the final reflectance map output by the illumination decomposition learning module, and jump-connects the feature maps obtained from different sampling layers to fuse the feature maps from different sampling layers to obtain the final enhanced image.

[0064] The enhanced image and the original low-light image are input into the unsupervised denoising enhancement network model for re-enhancement and denoising. By utilizing the contrast information of the enhanced image, the contrast and brightness of the low-light image can be adjusted more effectively, thereby improving the clarity and visual quality of the image. At the same time, denoising can also be performed on the noise present in the image to further improve the quality of the image. It should be noted that the present invention chooses to perform re-enhancement and denoising on the original low-light image instead of only operating on the enhanced image. This is because information loss usually occurs during the image processing process, especially in the enhancement operation. Therefore, in order to retain the details and information of the image to the greatest extent, the present invention chooses to perform subsequent processing on the original low-light image. Through this re-enhancement strategy, the present invention can more effectively remove noise in the image and further improve the clarity and quality of the image.

[0065] Step 3: Construct a loss function and train the illumination decomposable weak-light enhancement model and the unsupervised denoising enhancement model in combination with the back-propagation algorithm until the model converges to obtain a trained lightweight model for weak-light image enhancement, and use the lightweight model for weak-light image enhancement to enhance the weak-light image.

[0066] The two models are trained using the data set. The network model is trained using a large amount of training data, and the model parameters are optimized by minimizing the loss function. Finally, the performance of the model is evaluated using the test set data. By comparing with the test set data, the effect of the model in denoising and enhancement can be verified, and the model can be adjusted and optimized as necessary.

[0067] Use the trained network model for low-light image enhancement tasks, and only use a single illumination learning network module and an unsupervised denoising enhancement network module for low-light image enhancement and denoising, which can not only ensure the enhancement effect but also reduce the amount of calculation.

[0068] In some embodiments, the illumination decomposition learning module and the reflection component self-calibration module are used to learn the illumination component and reflection component of the image, respectively. The illumination decomposition and learning module and the reflection component self-calibration module include at least 4 convolutional layers, which are composed of two-dimensional convolution, normalization, relu activation function and sigmoid activation function. First, the low-light image is input into the illumination decomposition and learning network, which learns the illumination component of the image and outputs the illumination map. Then, the present invention divides the original low-light image by the illumination map to obtain the reflection map of this stage, and then inputs the reflection map into the reflection component self-calibration network, which learns the reflection component of the image, and adds the output reflection component self-calibration map to the original low-light image as the input of the next stage.

[0069] The illumination decomposition learning module does not use a single decomposition method to calculate the illumination component of the image, but uses an iterative multi-stage illumination component learning module. This module uses a shared parameter method, that is, each stage uses the same network architecture and parameters to gradually calculate the accurate illumination component in a multi-stage progressive manner. In addition, the module can adjust the number of multi-stage illumination decomposition iterations to suppress the overexposure problem caused by image enhancement. The module consists of at least four convolutional layers, the first convolutional layer contains at least one two-dimensional convolution, relu activation function and coordinate attention. The second convolutional layer, like the third convolutional layer, contains at least one two-dimensional convolution, normalization and relu activation function. The fourth convolutional layer contains at least one two-dimensional convolution and sigmoid activation function.

[0070] The addition of coordinate attention to the feature extraction layer of the illumination decomposition learning module helps to accurately capture the spatial structure information in the image, so that the network module can perform efficient illumination decomposition and learning. When implementing coordinate attention, in order to introduce position information, each position in the input sequence needs to be positionally encoded. The position encoding is generated by a sine or cosine function, and its dimension is the same as the feature dimension of the input. For a given input sequence, the key (k) and query (q) are obtained by adding the position encoding to the input, so that each position has its own position information. The value (v) is usually directly represented by the feature in the input sequence. Next, the dot product between the key (k) and the query (q) is calculated, and then normalized by the softmax function to obtain the attention weight. Finally, the value (v) is weighted summed using the calculated attention weight to obtain the output representation of each position.

[0071] Reflection component self-calibration module, which consists of at least six convolutional layers. The first convolutional layer contains a two-dimensional convolution, normalization, relu activation function and coordinate attention. The second convolutional layer contains two sets of two-dimensional convolution, normalization and relu activation function. The next three convolutional layers have the same structure as the second convolutional layer. The sixth convolutional layer contains a two-dimensional convolution and a sigmoid activation function. During the training phase, this module is used to assist the illumination decomposition and learning module so that each stage converges to the same state, accelerates the convergence speed of the entire network, and reduces the number of iterations of the illumination decomposition and learning module, reducing the amount of calculation.

[0072] In some embodiments, the unsupervised enhancement denoising module uses the output of the illumination decomposable weak light enhancement network and the original weak light image as input, and uses the contrast information of the enhanced image to adjust and denoise the contrast and brightness information of the enhanced image. The unsupervised enhancement denoising module comprises at least 9 convolutional layers, each of which is composed of a two-dimensional convolution, a relu activation function and a sigmoid activation function. The present invention also uses upsampling and downsampling operations for the purpose of reducing noise. However, downsampling will make the image blurred, so the present invention uses the concat operation in the fifth and sixth layers to fuse the downsampling results with the original image features, thereby increasing the detail information of the enhanced image. The present invention sets a reflection loss function. Unlike the existing method that only uses the gradient-norm, the present invention proposes a new distance measurement method, which takes into account the information retention and consistency required for the image enhancement task, and uses the characteristics of histogram equalization to improve the information entropy of the image. Therefore, there is no need for paired data sets for training, and unsupervised weak light image enhancement and denoising is achieved.

[0073] Example 1

[0074] See also Figure 2 and 3 , a low-light image enhancement method based on illumination decomposition, comprising the following steps:

[0075] Step 1: Build a coordinate attention-based illumination decomposable weak light enhancement network model, which consists of an illumination decomposition and learning module and a reflection component self-calibration module.

[0076] S1.1 Illumination decomposition and learning module: This module can be mainly composed of multiple sets of two-dimensional convolutional layers, relu activation functions, normalization and sigmoid activation functions. Assume that the input of this module is a 3×600×400 image, and its output is a 3×600×400 feature map. The output shape of the illumination decomposition and learning module should be the same as the input shape of the reflectance component self-calibration module. The illumination decomposition and learning module learns the illumination residual in stages, which greatly reduces the computational difficulty.

[0077] 1.2 Reflection component self-calibration module: This module can be composed of multiple sets of two-dimensional convolutional layers, relu activation functions, normalization and sigmoid activation functions. Assuming that the input of this module is a 3×600×400 image, a 16×600×400 feature map will be obtained after passing through this module, and the final output is a 3×600×400 feature map. The reflection component self-calibration module bridges the input of each stage (except the first stage) with the low-light observation (i.e., the input of the first stage), and indirectly explores the convergence behavior between each stage.

[0078] The construction process of the illumination decomposition learning network is as follows:

[0079] The mapping of the reflectance component self-calibration module is added to the original low-light input to obtain a 3-channel image matrix as the input for the next stage of illumination estimation. The first convolution layer contains a 2D convolution, relu activation function, convolution kernel size of 3×3, output channels of 16, and convolution stride of 1, recorded as 3×3×16×1. The feature map output from the first convolution is input to the coordinate attention layer. First, in order to introduce position information, the input feature sequence is encoded along the horizontal coordinate and vertical coordinate for each channel using pooling kernels of size (H,1) and (1,W) respectively. Then the obtained position encoding is concat and the concat output is transformed using a convolution transformation function with a convolution kernel of 16, so that each position has its own position information. Next, the input feature map is normalized and relu activated, and then decomposed into two separate tensors along the spatial dimension. Next, two additional convolution transformations are used to transform the above two tensors into tensors with the same number of channels, keeping the dimension of the input tensor unchanged. In this step, an appropriate reduction ratio (e.g., 16) is used to reduce the number of channels of the tensor. This helps reduce the complexity and computational overhead of the model while retaining key feature information. Then, the two obtained tensors are operated with a sigmoid activation function to obtain two attention weight tensors. The two attention weights are used to weight the input tensors respectively to enhance the model's attention to different positions. Finally, the two weighted input tensors are multiplied by the original input to obtain the output of the coordinate attention, with an output channel of 16. The second convolutional layer, like the third convolutional layer, contains a 2D convolution, normalization and relu activation function, with a convolution kernel size of 3×3, an output channel of 16, and a convolution stride of 1, denoted as 3×3×16×1. The fourth convolutional layer contains a 2D convolution and a sigmoid activation function, with a convolution kernel size of 3×3, an output channel of 3, and a convolution stride of 1, denoted as 3×3×3×16×1.

[0080] The construction process of the reflection component self-calibration module is as follows:

[0081] The input of each stage of the reflectance component self-calibration network is derived from the previous stage. The input of the first stage is explicitly defined as low-light observations with 3 channels. The first convolution layer contains a 2D convolution, normalization, relu activation function, convolution kernel size of 3×3, output channels of 16, and convolution stride of 1, denoted as 3×3×16×1. The feature map output from the first convolution is input to the coordinate attention layer. First, in order to introduce position information, the input feature sequence is encoded along the horizontal coordinate and vertical coordinate for each channel using pooling kernels of size (H,1) and (1,W) respectively. Then the obtained position encoding is concat and the concat output is transformed using a convolution transformation function with a convolution kernel of 16, so that each position has its own position information. Next, the input feature map is normalized and relu activated, and then decomposed into two separate tensors along the spatial dimension. Next, two additional convolution transformations are used to transform the above two tensors into tensors with the same number of channels, keeping the dimension of the input tensor unchanged. In this step, an appropriate reduction ratio (e.g., 16) is used to reduce the number of channels of the tensor. This helps reduce the complexity and computational overhead of the model while retaining key feature information. Then, the two obtained tensors are subjected to a sigmoid activation function to obtain two attention weight tensors. These two attention weights are used to weight the input tensors respectively to enhance the model's attention to different positions. Finally, the two weighted input tensors are multiplied by the original input to obtain the output of the coordinate attention, with an output channel of 16. The second convolutional layer contains two sets of two-dimensional convolutions, normalization and relu activation functions, with a convolution kernel size of 3×3, an output channel of 16, and a convolution stride of 1, recorded as 3×3×16×1. The next three convolutional layers have the same structure as the second convolutional layer, with a convolution kernel size of 3×3, an output channel of 16, and a convolution stride of 1, recorded as 3×3×16×1. The sixth convolutional layer contains a two-dimensional convolution and a sigmoid activation function, with a convolution kernel size of 3×3, an output channel of 3, and a convolution step of 1, denoted as 3×3×3×1.

[0082] Step 2: Build an unsupervised denoising enhancement network model. This module can be mainly composed of multiple sets of two-dimensional convolutional layers, relu activation functions, normalization and sigmoid activation functions. Concat the 3×600×400 feature map obtained in step 1.1 above with the original low-light image to obtain a 6×600×400 image matrix, which is input into the unsupervised denoising enhancement network to obtain a 3×600×400 feature matrix.

[0083] The construction process of the unsupervised denoising enhancement network is as follows:

[0084] The network contains 9 convolutional layers. The first convolutional layer contains a 2D convolution with 6 input channels, 64 output channels, and a kernel size of 3 × 3. The second convolutional layer contains a 2D convolution and relu activation function with 64 input and output channels and a kernel size of 3. The third convolutional layer contains a 2D convolution and relu activation function with 64 input channels, 128 output channels, and a kernel size of 3. The fourth convolutional layer contains a 2D convolution and relu activation function with 128 input channels, 128 output channels, and a kernel size of 3. The fifth convolutional layer contains a 2D convolution and relu activation function with 128 input channels, 64 output channels, and a kernel size of 3. The sixth convolutional layer contains a 2D convolution and relu activation function with 128 input channels, 64 output channels, and a kernel size of 3. The seventh convolution layer contains a 2D convolution and relu activation function, with 96 input channels, 64 output channels, and a kernel size of 3. The eighth convolution layer contains a 2D convolution and relu activation function, with 64 input channels, 64 output channels, and a kernel size of 3. The ninth convolution layer contains a 2D convolution, with 64 input channels, 3 output channels, and a kernel size of 3.

[0085] Step 3, use the training data set to train the network model constructed in steps 1 and 2. In the present invention, fidelity loss is used to ensure the pixel-level consistency of the estimated illumination component and the input of each stage. At the same time, a smoothing term with a spatial variation norm L1 is introduced to constrain the smoothness of the output image. In addition, a calibrated color deviation loss is designed to correct the potential color deviation in the enhanced image, and the relationship between the three adjustment channels is established. Finally, the reflection loss is used to improve the contrast of the image and suppress noise. During the training process, the model parameters are updated by minimizing the total loss value. Specifically, the input data is passed to the model to calculate the loss, and the model parameters are updated using the back propagation algorithm. 1000 epochs of training were performed on the training set, and each epoch included passing the entire training set to the model once and updating the parameters. The training loss function is shown in formula (1).

[0086] The loss function is defined as shown in Formula 1:

[0087] L total =1.5·L f +L s +2·L color +L R (1)

[0088] Among them, L f is the fidelity loss, L s is the smoothing loss, Lcolor is the color constancy loss, L R is the reflection loss, as shown in formula (2), formula (3), formula (5) and formula (6) respectively.

[0089]

[0090] In formula (2), T is the total number of stages. In fact, this function uses the redefined input y+s t-1 To constrain the output lighting x t , rather than hand-crafted ground-truth or common low-light input. The fidelity loss is to ensure the consistency of the estimated illumination with the input pixel level at each stage.

[0091]

[0092] In formula (3), N is the total number of pixels. i is the i-th pixel. N(i) represents the neighboring pixels of i in its 5×5 window. ω i,j Represents the weight, and its formula is

[0093]

[0094] Where c is the image channel in the YUV color space and σ = 0.1 is the standard deviation of the Gaussian kernel.

[0095]

[0096] In formula (5), J p Represents the average intensity value of the p channel in the enhanced image, and (p,q) represents a pair of channels.

[0097]

[0098] Where F(x) is the histogram equalization operator of image x, λ is the weight parameter, is the gradient operator. The first term of the loss function is used to directly control the contrast of the enhanced image. The second term is used to suppress noise.

[0099] Step 4: Use the trained network model for low-light enhancement. Use the illumination decomposition and learning module and the unsupervised denoising enhancement network module trained in step 3 to perform low-light image enhancement. First, use the network parameters saved in step 3 to initialize the network built in steps 1 and 2; then, input the test data set into the trained network, and finally output the enhanced image result.

[0100] During the training phase, an auxiliary learning method is adopted, that is, the reflection component self-calibration module is used to assist the training of the illumination decomposition and learning module, ensuring that the outputs of different stages in the training process can converge to the same state and reducing the computational burden of the cascade mode of the illumination decomposition and learning module; in the testing phase, the reflection component self-calibration module can be discarded after training, and only the illumination decomposition and learning module and the unsupervised denoising enhancement module can be used to enhance the low-light image.

[0101] Example 3

[0102] Another method of the present invention also provides a system for a weak-light image enhancement method based on illumination decomposition, including an illumination decomposable weak-light enhancement model construction module and an unsupervised denoising enhancement model construction module;

[0103] An illumination decomposable weak light enhancement model construction module is used to construct an illumination decomposable weak light enhancement model based on coordinate attention, which includes an illumination decomposition and learning module and a reflection component self-calibration module;

[0104] The illumination decomposition learning module embeds a coordinate attention mechanism in the feature extraction layer, the feature extraction layer obtains a feature map of the original low-light image, and uses the coordinate attention mechanism to embed the position information of the original image into the feature map to obtain an illumination map of the original low-light image, and divides the original low-light image by the illumination map to obtain a reflectance map of the original low-light image;

[0105] The reflection component self-calibration module obtains the reflection component of the reflection image, combines the reflection component with the original low-light image, and obtains an enhanced low-light image;

[0106] The enhanced low-light image is input into the illumination decomposition learning module, and the reflectance component self-calibration module is repeatedly executed until the number of iterations is reached to obtain the reflectance map of the preliminary enhanced image;

[0107] Unsupervised denoising enhancement model building module, used to build an unsupervised denoising enhancement model;

[0108] The unsupervised denoising enhancement model uses upsampling and downsampling to denoise the final reflectance map output by the illumination decomposition learning module, and jump-connects the feature maps obtained from different sampling layers to fuse the feature maps from different sampling layers to obtain the final enhanced image.

[0109] Example 4

[0110] An image detection system is provided, wherein the execution program of the weak-light image enhancement method based on illumination decomposition is set in the image detection system. When the image detection system is running, the weak-light image enhancement method based on illumination decomposition is used to enhance the acquired weak-light image, and image detection is performed based on the enhanced image.

[0111] The image detection system may be a monitoring system in a factory or a monitoring system on a mobile terminal.

[0112] The present invention discloses a low-light image enhancement method based on illumination decomposition. The method first performs preliminary enhancement on the low-light image through an illumination decomposable low-light enhancement model based on coordinate attention, and then simultaneously inputs the enhanced image and the original low-light image into an unsupervised denoising enhancement model for re-enhancement and denoising. A new parameter sharing mechanism is introduced, and the same network architecture and weights are used in the feature extraction stage, thereby significantly reducing the amount of calculation for learning the illumination component. In addition, an unsupervised denoising enhancement module is designed to further improve the generalization and noise resistance of the network. Through this module, the network can better handle noise from various environments, improve the performance of the network under complex conditions, and further improve the performance of the present invention in the low-light image enhancement task. The present invention also adds coordinate attention, which links each position in the feature map with its corresponding coordinate information, and combines the position information with the channel attention weight, so that the model can take into account the importance of each position while paying attention to the channel, which not only improves the representation ability of the model, but also maintains a low computational cost.

[0113] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A low-light image enhancement method based on illumination decomposition, characterized in that: The following steps are involved: Step 1: Construct an illumination decomposable weak light enhancement model based on coordinate attention, which includes an illumination decomposition and learning module and a reflection component self-calibration module; The illumination decomposition learning module embeds a coordinate attention mechanism in the feature extraction layer, the feature extraction layer obtains a feature map of the original low-light image, and uses the coordinate attention mechanism to embed the position information of the original image into the feature map to obtain an illumination map of the original low-light image, and divides the original low-light image by the illumination map to obtain a reflectance map of the original low-light image; The reflection component self-calibration module obtains the reflection component of the reflection image, combines the reflection component with the original low-light image, and obtains an enhanced low-light image; The enhanced low-light image is input into the illumination decomposition learning module, and the reflectance component self-calibration module is repeatedly executed until the number of iterations is reached to obtain the reflectance map of the preliminary enhanced image; Step 2: Build an unsupervised denoising enhancement model; The unsupervised denoising enhancement model uses upsampling and downsampling to denoise the final reflectance map output by the illumination decomposition learning module, and jump-connects the feature maps obtained from different sampling layers to fuse the feature maps from different sampling layers to obtain the final enhanced image.

2. The low-light image enhancement method based on illumination decomposition according to claim 1, characterized in that: The method for extracting the illumination map of the original low-light image in step 1 is as follows: The illumination decomposition learning module includes at least four convolutional layers; The first convolutional layer contains at least one 2D convolution, relu activation function and coordinate attention; The second convolutional layer has the same structure as the third convolutional layer, including at least one 2D convolution, normalization and relu activation function; The fourth convolutional layer contains at least one 2D convolution and a sigmoid activation function.

3. The low-light image enhancement method based on illumination decomposition according to claim 1, characterized in that: The method for extracting the illumination map of the original low-light image in step 1 is as follows: Each channel of the feature sequence of the low-light image is encoded, and then the obtained position code is concat-operated and transformed to obtain a feature map. The feature map is normalized and relu activated, and then decomposed into two separate tensors along the spatial dimension. The two tensors are transformed into tensors with the same number of channels, keeping the dimension of the input tensor unchanged, and the sigmoid activation function is performed on the two obtained tensors to obtain two attention weight tensors and use them to weight the input tensor. The two weighted input tensors are multiplied with the initial input to obtain the illumination map.

4. The low-light image enhancement method based on illumination decomposition according to claim 3, characterized in that: The number of channels of the tensor transformation is reduced using the reduction ratio.

5. The method for low-light image enhancement based on illumination decomposition according to claim 1, characterized in that: The reflection component self-calibration module includes at least six convolutional layers; The first convolutional layer includes a 2D convolution, normalization, relu activation function and coordinate attention; The last convolutional layer includes a two-dimensional convolution and a sigmoid activation function; The structure of the first convolutional layer is the same as that of the multiple convolutional layers between the last convolutional layer, including two sets of 2D convolutions, normalization, and relu activation functions.

6. The method for low-light image enhancement based on illumination decomposition according to claim 1, characterized in that: Construct a loss function and combine the back-propagation algorithm to train the illumination decomposable low-light enhancement model and the unsupervised denoising enhancement model until the model converges to obtain a lightweight model for low-light image enhancement after training; The loss function includes fidelity loss, smoothness loss, color constancy loss and reflection loss.

7. The method for low-light image enhancement based on illumination decomposition according to claim 6, characterized in that: The expression of the loss function is as follows: L total 1.5 L f +L s +2·L color +L R Among them, L f is the fidelity loss, L s is the smoothing loss, L color is the color constancy loss, L R is the reflection loss.

8. The method for low-light image enhancement based on illumination decomposition according to claim 1, characterized in that: The expression of the fidelity loss is as follows: Where T is the total number of stages, y+s t-1 Input lighting, x t Output lighting; The expression of the smoothing loss is as follows: Where N is the total number of pixels, i is the i-th pixel, N(i) represents the neighboring pixels of i in the window, ω i,j represents weight; Where c is the image channel in the YUV color space, σ is the standard deviation of the Gaussian kernel; The expression of the color constancy loss is as follows: The expression of the reflection loss is as follows: Where F(x) is the histogram equalization operator of image x, λ is the weight parameter, is the gradient operator.

9. A system for executing the low-light image enhancement method based on illumination decomposition according to any one of claims 1 to 8, characterized in that: include: An illumination decomposable weak light enhancement model construction module is used to construct an illumination decomposable weak light enhancement model based on coordinate attention, which includes an illumination decomposition and learning module and a reflection component self-calibration module; The illumination decomposition learning module embeds a coordinate attention mechanism in the feature extraction layer, the feature extraction layer obtains a feature map of the original low-light image, and uses the coordinate attention mechanism to embed the position information of the original image into the feature map to obtain an illumination map of the original low-light image, and divides the original low-light image by the illumination map to obtain a reflectance map of the original low-light image; The reflection component self-calibration module obtains the reflection component of the reflection image, combines the reflection component with the original low-light image, and obtains an enhanced low-light image; The enhanced low-light image is input into the illumination decomposition learning module, and the reflectance component self-calibration module is repeatedly executed until the number of iterations is reached to obtain the reflectance map of the preliminary enhanced image; Unsupervised denoising enhancement model building module, used to build an unsupervised denoising enhancement model; The unsupervised denoising enhancement model uses upsampling and downsampling to denoise the final reflectance map output by the illumination decomposition learning module, and jump-connects the feature maps obtained from different sampling layers to fuse the feature maps from different sampling layers to obtain the final enhanced image.

Images