Multi-scale lightweight low-light image enhancement method

Through the lightweight method of multi-scale transformation and deep feature fusion, the problems of weak generalization ability and high model complexity in low-light image enhancement are solved, and the image quality is efficiently improved, details and colors are retained in multiple scenarios. Real-time application of computer vision systems.

CN112884668BActive Publication Date: 2024-11-15DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110195729.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-22
Publication Date
2024-11-15
Estimated Expiration
2041-02-22

AI Technical Summary

Technical Problem

The existing low-light image enhancement methods, including traditional methods and deep learning methods, have problems with weak generalization ability, high model complexity, slow processing speed, and possible loss of texture details and color distortion, making it difficult to effectively improve images in multiple scenarios. quality.

Method used

Using a lightweight low-light image enhancement method based on multi-scale, the initial light is extracted through multi-scale transformation, depth feature extraction and cross-scale fusion are performed to obtain fine light, and the enhanced image is calculated based on Retinex vision theory, and a multi-layer network structure is designed and loss function constrains lighting estimation, simplifying the model and improving generalization capabilities.

Benefits of technology

It realizes that while ensuring clear and realistic enhancement results, it can improve the generalization ability of images, reduce model complexity, and improve processing speed. It can effectively improve the overall brightness and visual quality of low-light images, retain local details and textures, and improve color performance. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112884668B_ABST
    Figure CN112884668B_ABST
Patent Text Reader

Abstract

The present invention provides a lightweight low-light image enhancement method based on multi-scale. It solves the problems that the existing methods cannot fully retain image details, have poor clarity, color distortion, and weak generalization ability. The present invention utilizes the multi-scale representation of the image and uses a deep neural network to extract the multi-scale illumination information of the low-light image, effectively exploring and retaining the detailed features of the image. A progressive fusion strategy is used to fuse and adjust the multi-scale illumination information, and a fine illumination image that can accurately represent the external illumination distribution is restored. Finally, the adverse effects of external illumination are eliminated according to visual theory to achieve image enhancement. The present invention makes full use of the complementary feature information of the image at different scales, enhances the image visual effect, and has a simple network and fast processing speed, which has practical value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image enhancement, and in particular relates to a multi-scale based lightweight low-light image enhancement method. Background Art

[0002] Low-light environments have always been a difficult problem in the field of computer vision. Any image taken in an environment with insufficient or uneven lighting will have problems such as low brightness, high noise, and color distortion, which will seriously reduce the image quality and affect the performance of the computer vision system. Therefore, in order to improve the image quality and the performance of the visual system, it is necessary to enhance the low-light images. Over the past few decades, researchers have been committed to the research and application of low-light image enhancement methods. At present, the existing low-light image enhancement methods are divided into traditional methods and deep learning methods.

[0003] Traditional methods can be divided into histogram equalization-based methods and Retinex theory-based methods. The former changes the overall distribution of image grayscale by expanding the grayscale dynamic range of the image, thereby improving the image brightness, but the enhancement effect on image details is poor. The latter is based on Retinex visual theory, which believes that low-light images are the product of the object's reflection image and the external illumination image. The illumination image is obtained by performing Gaussian blur and other operations on the low-light image, and then mathematical calculations are used to remove the influence of illumination to obtain the reflection image, i.e., the enhancement result. This type of method has limited effect on color image enhancement and is prone to color distortion.

[0004] In recent years, with the rapid development of deep learning and the advent of the big data era, enhancement methods based on deep learning have been widely used. This method uses neural networks to learn an image enhancement and adjustment capability from a large number of low-light data sets. It can be divided into end-to-end direct enhancement methods and indirect enhancement methods including illumination estimation. The former directly learns the enhanced image through the training data set, such as LLNet. The latter first uses the network to estimate the illumination image, and then obtains the enhancement result based on the Retinex theory, such as MSR-net. This type of method effectively improves the generalization ability of the method with the help of the excellent feature extraction ability of the neural network, but the network structure is complex and the processing time is long, which makes it difficult to meet the real-time requirements of the visual system.

[0005] In summary, both traditional methods and deep learning methods have many shortcomings. Traditional methods are simple to implement, fast in processing, and independent of training data sets, but they have weak generalization capabilities and cannot effectively enhance low-light images in multiple scenes. Deep learning methods use deep networks and large data sets to learn an enhancement model with strong generalization capabilities, but they are overly dependent on domain knowledge, have high model complexity, and the enhanced image may lose texture details and suffer from color distortion. Therefore, a new multi-scale lightweight low-light image enhancement method must be provided to improve the generalization ability of the method, reduce model complexity, and increase processing speed while ensuring that the enhancement results are clear and realistic. Summary of the invention

[0006] In view of the shortcomings and improvement needs of existing deep learning-based methods, the present invention provides a lightweight low-light image enhancement method based on multi-scale, and the specific scheme includes the following steps:

[0007] Step 1: Perform multi-scale transformation on the low-light image to obtain low-light images of different scales;

[0008] Step 2: Use the network to extract the initial illumination of low-light images of different scales;

[0009] Step 3: Perform deep feature extraction and cross-scale fusion on the initial illumination to obtain fused illumination;

[0010] Step 4: Upsample, connect and adjust the multi-scale fusion illumination to obtain fine illumination;

[0011] Step 5: According to Retinex visual theory, the enhanced image is obtained by fine lighting calculation.

[0012] Furthermore, the specific process of the above step 1 is: using the low-light image as the bottom image, a Gaussian image pyramid is established from bottom to top. The specific implementation process is shown in formulas (1) and (2):

[0013]

[0014]

[0015] In formula (1), G i Represents the i-th layer image of the pyramid, with a total of n layers. Down() represents the downsampling operation. represents the convolution operation, g k×k represents a Gaussian convolution kernel of size k×k; Formula (2) is a Gaussian kernel function, where σ represents a scale parameter; the above process is iterated multiple times to obtain image pyramids of different heights.

[0016] Furthermore, the above step 2 is specifically as follows: designing an extraction network with a multi-layer network structure to independently extract the initial illumination of low-light images of different scales, wherein each layer of the network structure is composed of units, and each unit is composed of a convolutional layer, a batch normalization layer, and an activation function in sequence; according to the different scales of the input image, the convolution kernel size of each layer of the network is set to the convolution kernel size of the corresponding layer in step 1 in sequence.

[0017] Furthermore, the above step three is specifically as follows: inputting the low-resolution initial illumination obtained in step two into the fusion network model, further extracting the illumination information, and then upsampling by using deconvolution, and adding it pixel by pixel with all the high-resolution initial illuminations to achieve feature fusion; iterating the above process to obtain multi-scale fused illumination; the fusion network includes residual blocks, each residual block is composed of a convolutional layer, a batch normalization layer and an activation function, and each residual block uses a jump connection to achieve progressive fusion of illumination information and back propagation of gradients.

[0018] Furthermore, the above step 4 is specifically as follows: first, the initial illumination obtained in step 2 and the fused illumination obtained in step 3 are connected at their respective scales; then, the convolutional layer is used to further integrate the illumination information, learn the correlation and dependency between the channels of the multi-scale illumination, and output the new illumination; then, from top to bottom, the illumination of adjacent scales is connected, and the low-scale illumination is first upsampled to the same size as the high-scale illumination and then connected to obtain a new illumination of a higher scale. The above operation is iterated for all adjacent scale illuminations until a multi-channel illumination of the same size as the original image is obtained. Finally, the convolution operation is used to adjust the channels to obtain a refined illumination, which can accurately reflect the impact of external illumination on the original image.

[0019] Compared with the prior art, the multi-scale lightweight low-light image enhancement method provided by the present invention has the following beneficial effects:

[0020] 1) The multi-scale representation of images is introduced into the field of image enhancement to fully explore the unique image detail features of different scales, remove redundant information and fuse complementary information in the fusion process.

[0021] 2) The illumination estimation problem in the image enhancement process is comprehensively considered, and a variety of loss functions are designed to adjust the constraints and improve the visual quality of the enhanced image.

[0022] 3) The residual learning mechanism is introduced to enable the network to adjust low-light images with different exposure levels. The method has strong generalization ability, simple model and fast processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a multi-scale lightweight low-light image enhancement method provided by the present invention;

[0024] Figure 2 Schematic diagram of the overall structure of the low-light image enhancement model of the present invention;

[0025] Figure 3 This is a structural diagram of the network extracted in the low-light image enhancement model of the present invention;

[0026] Figure 4 It is a structural diagram of the fusion network in the low-light image enhancement model of the present invention;

[0027] Figure 5 This is a comparison chart of low-light images processed by other methods and this method. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention are fully described below in conjunction with the accompanying drawings of the examples of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] The present invention provides a lightweight low-light image enhancement method based on multi-scale Figure 1 The method is specifically implemented by the following specific steps:

[0030] Step 1: construct a multi-scale representation of the low-light image to obtain a multi-scale image. Image pyramid is an important means of image preprocessing and is widely used. In this step, Gaussian pyramid is selected as a multi-scale transformation tool to perform multi-scale representation of the low-light image.

[0031] Specifically, the low-light image is used as the bottom image of the pyramid, and a Gaussian pyramid is established from bottom to top. The implementation process is shown in formulas (1) and (2):

[0032]

[0033]

[0034] In formula (1), G i Represents the i-th layer image of the pyramid, with a total of n layers. Down() represents the downsampling operation. represents the convolution operation, g k×kRepresents a Gaussian convolution kernel of size k×k. In this embodiment, the pyramid height n is set to 3, and the convolution kernel size k can be 7, 5, and 3 respectively. Formula (2) is a Gaussian kernel function, where σ represents a scale parameter. In specific implementation, in order to increase the difference between images of different scales, the scale parameters of the three Gaussian kernels can be set to 15, 80, and 250. The process shown in formula (1) is iterated multiple times to establish a Gaussian image pyramid, and three low-light images of different scales and resolutions are obtained.

[0035] Step 2: Use the network to extract the initial illumination of low-light images of different scales.

[0036] Specifically, the multi-scale low-light image obtained in step 1 is input into the network model. The network model involved here is a multi-layer network, called an extraction network. Each layer of the network independently extracts the initial illumination of low-light images of different scales. In specific implementation, Figure 2 , Figure 3 As shown in the figure, the structures of each network layer are basically the same. Each network layer contains multiple units, each of which is composed of a convolution layer, a batch normalization layer, and an activation function. The units share feature information using long and short connections. The convolution kernel size of each network layer can be set to 7, 5, and 3 from the bottom up according to the different scales of the input image to effectively extract the initial illumination of different scales.

[0037] Step 3: Perform cross-scale fusion on the initial illuminations of different scales to obtain fused illumination. There is a large amount of redundant and complementary feature information between the multi-scale initial illuminations obtained in step 2. This step will fuse the complementary information and remove the redundant information.

[0038] Specifically, the multi-scale initial illumination is input into the network model. The network involved here is still a multi-layer structure, called a fusion network, such as Figure 2 , Figure 4 As shown. Unlike the extraction network, there are associations between the layers of this network. In addition, the fusion network includes multiple residual blocks, each of which consists of a convolutional layer, a batch normalization layer, and an activation function. Each residual block uses jump connections to achieve progressive fusion of illumination information and back propagation of gradients. In specific implementation, the low-resolution initial illumination is first passed through multiple residual recursive units to deeply extract the illumination information, and then up-sampled by deconvolution, and added pixel by pixel to all high-resolution initial illumination to achieve cross-scale illumination fusion. The above operations are continuously iterated until all initial illuminations are fused. At this point, multi-scale fused illumination can be obtained, and each scale of illumination incorporates the complementary information of all other low-scale illuminations.

[0039] Step 4: Upsample, connect and adjust the multi-scale fused illumination to obtain refined illumination.

[0040] A multi-layer network is designed to further integrate and adjust the illumination information. First, the initial illumination obtained in step 2 and the fused illumination obtained in step 3 are concatenated at their respective scales. Then the concatenation result is passed through a convolutional layer to learn the correlation and dependency between the illumination channels and output the final refined illumination. Specifically, from top to bottom, the illumination of two adjacent scales is concatenated. First, the low-scale illumination is upsampled to the same size as the high-scale illumination and concatenated to obtain a new illumination of a higher scale. The above operation is iterated for all adjacent scale illuminations until a multi-channel illumination of the same size as the original image is obtained. Finally, the convolution operation is used to adjust the channels to obtain refined illumination, which can accurately reflect the impact of external illumination on the original image.

[0041] Step 5: According to the Retinex visual theory, the enhanced result is calculated. This method uses the fine illumination output by the neural network to indirectly generate an enhanced image according to the Retinex theory. The Retinex theory believes that the low-light image S is the product of the original image R and the illuminated image L, as shown in formula (3).

[0042] S(x,y)=R(x,y)×L(x,y) (3)

[0043] Therefore, in order to obtain an enhanced image, it is necessary to perform an inverse Retinex transform. The specific process is as follows:

[0044] Step 51: First, perform logarithmic transformation on the low-light image and the illuminated image, and convert the division operation into a subtraction operation to reduce the amount of calculation, as shown in formula (4).

[0045] log(S(x,y))=log(R(x,y))+log(L(x,y)) (4)

[0046] Step 52: Difference the two in the logarithmic domain, as shown in formulas (5) and (6).

[0047] r(x,y)=log(R(x,y))=log(S(x,y))-log(L(x,y)) (5)

[0048] r(x,y)=s(x,y)-l(x,y) (6)

[0049] Step 5: Perform an exponential transformation on the calculation result to return to the original number domain to obtain the enhanced image, as shown in formula (7).

[0050] R(x,y)=e r(x,y) (7)

[0051] The above contents are all embodiments of the present invention. In the network model training stage, low-light and normal-light image pairs in the low-light image dataset are respectively input into the network, the former is used as the network input, and the latter is used as the label, and the loss function is used to measure the loss between the enhanced image and the label. The network is gradient-transferred based on the loss, the network parameters are updated, and the network is iteratively trained multiple times.

[0052] In the network model training stage, the image enhancement method provided by the present invention inputs low-light and normal-light image pairs in the low-light image dataset into the network respectively, with the former serving as the network input and the latter serving as labels. A loss function is used to measure the loss between the enhanced image and the label, and the network is gradient-transferred based on the loss to update the network parameters, and the network is trained iteratively for multiple times.

[0053] The composite loss function is used in the network training stage, as shown in formula (8). Specifically, it includes three loss sub-items, namely: the reconstruction loss L that measures the similarity between the normal image and the low-light image under the influence of the estimated illumination recon , as shown in formula (9); the structural similarity loss L that measures the spatial structural similarity between the enhanced image and the normal illumination image ssim , as shown in formula (10); the smoothness loss L used to constrain the illumination smooth , as shown in formula (11). Where λ1, λ2, and λ3 represent the weights of each loss sub-item respectively.

[0054] L=λ1L recon +λ2L ssim +λ3L smooth (8)

[0055]

[0056]

[0057]

[0058] In formula (9), S i represents the i-th light map, represents the i-th normal light portrait, I i represents the i-th low-light image, N represents the amount of training data, and 2 represents the Euclidean distance. In formula (10), μ x , μ y Represent the average brightness of the enhanced result and the normal illumination image, σ x , σ y represents the contrast between the enhanced result and the normal illumination image, C1 and C2 are constants to prevent zero division anomalies. In formula (11), p and c represent the number of pixels and channels of the illumination image, respectively. and Represents the square of the horizontal and vertical gradients of the illumination image at pixel p of channel c, and Represents the weights of horizontal and vertical gradients. The final loss function L is composed of three weighted loss functions, which effectively supervises the training process of the network model.

[0059] Experimental results and analysis: The simulation software used in this paper is PyCharm. The simulation hardware platform used is CPU: Intel i7 7700, GPU: NVIDIA 1060 graphics card, memory: 16GB DDR4. The input and output of the simulation program are both PNG format images. In order to prove the superiority of the method, a control experiment was designed, and the superior performance KinD algorithm and ZeroDCE algorithm were selected as comparison methods. The experimental results are as follows Figure 5 As shown. It can be seen that compared with the low-light image, the overall brightness of the image processed by the algorithm is effectively improved and conforms to the visual characteristics of the human eye. Compared with other enhancement algorithms, the local details and textures of the enhanced image obtained by the algorithm of the present invention are completely preserved, the colors are realistic, and the visual effect is good. Therefore, the image enhancement algorithm provided by the present invention effectively enhances the overall brightness, color and contrast of the image while retaining the original detail information, and has superiority in enhancing the visual effect of the image.

[0060] The above content is only a detailed description of the calculation model and processing flow of the present invention, and is not a limitation on the implementation of the present invention. Any equivalent structure and equivalent process replacement made using the contents of the present invention specification and drawings is still within the protection scope of the present invention.

Claims

1. A lightweight low-light image enhancement method based on multi-scale, It is characterized in that The steps include: Step 1: Perform multi-scale transformation on the low-light image to obtain low-light images of different scales; Step 2: Use the network to extract the initial illumination of low-light images of different scales; Step 3: Perform cross-scale fusion on the initial illuminations of different scales to obtain fused illumination; Step 4: upsample, connect and adjust the multi-scale fused illumination to obtain fine illumination; specifically: first, connect the initial illumination obtained in step 2 and the fused illumination obtained in step 3 at their respective scales; then use the convolution layer to further integrate the illumination information, learn the correlation and dependency between the channels of the multi-scale illumination, and output the new illumination; then connect the illumination of adjacent scales from top to bottom, first upsample the low-scale illumination to the same size as the high-scale illumination and then connect it to obtain a new illumination of a higher scale; iterate the above operations on all adjacent-scale illuminations until a multi-channel illumination of the same size as the original image is obtained; finally, use the convolution operation to adjust the channels to obtain fine illumination, which can accurately reflect the impact of external illumination on the original image; Step 5: According to the Retinex visual theory, the enhanced image is obtained by fine illumination calculation; specifically: Step 51: The low-light image S is the product of the original image R and the light image L. Perform logarithmic transformation on the low-light image S and the light image L: log(S(x,y))=log(R(x,y))+log(L(x,y)) (4) Step 52: Difference between the two in the logarithmic domain: r(x,y)=log(R(x,y))=log(S(x,y))-log(L(x,y)) (5) r(x,y)=s(x,y)-l(x,y) (6) Step 53: Perform an exponential transformation on the calculation result, return it to the original number domain, and obtain the enhanced image: R(x,y)=e r(x,y) (7)。 2. The multi-scale lightweight low-light image enhancement method according to claim 1, It is characterized in that The specific process of step 1 is: using the low-light image as the bottom image, and building a Gaussian image pyramid from bottom to top. The specific implementation process is: In formula (1), G i Represents the i-th layer image of the pyramid, with a total of n layers. Down() represents the downsampling operation. represents the convolution operation, g k×k represents a Gaussian convolution kernel of size k×k; Formula (2) is a Gaussian kernel function, where σ represents a scale parameter; the above process is iterated multiple times to obtain image pyramids of different heights.

3. The multi-scale lightweight low-light image enhancement method according to claim 2, It is characterized in that The step 2 is specifically as follows: designing an extraction network with a multi-layer network structure to independently extract the initial illumination of low-light images of different scales, wherein each layer of the network structure is composed of units, and each unit is composed of a convolutional layer, a batch normalization layer, and an activation function in sequence; according to the different scales of the input image, the convolution kernel size of each layer of the network is set to the convolution kernel size of the corresponding layer in step 1 in sequence.

4. The multi-scale lightweight low-light image enhancement method according to claim 1, It is characterized in that The step three is specifically as follows: inputting the low-resolution initial illumination obtained in step two into the fusion network model, further extracting the illumination information, then upsampling by using deconvolution, and adding it pixel by pixel with all the high-resolution initial illuminations to achieve feature fusion; iterating the above process to obtain multi-scale fused illumination; the fusion network includes residual blocks, each residual block is composed of a convolution layer, a batch normalization layer and an activation function, and each residual block uses a jump connection to achieve progressive fusion of illumination information and back propagation of gradients.

5. The multi-scale lightweight low-light image enhancement method according to claim 1, It is characterized in that In the network model training stage, this method inputs the low-light and normal-light image pairs in the low-light image dataset into the network respectively, with the low-light image as the network input and the normal-light image as the label. The loss function is used to measure the loss between the enhanced image and the label. The gradient of the network is returned based on the loss, the network parameters are updated, and the network is trained iteratively for multiple times.

6. The multi-scale lightweight low-light image enhancement method according to claim 5, It is characterized in that The loss function includes three sub-items, namely: the reconstruction loss L that measures the similarity between the normal image and the low-light image under the influence of estimated illumination recon ; Structural similarity loss L that measures the spatial structural similarity between the enhanced image and the normal illumination image ssim ; Smoothness loss L for smoothness constraint on illumination smooth ; Among them, λ 1 ,λ 2 ,λ 3 Respectively represent the weight of each loss sub-item; L=λ 1 L recon +λ 2 L ssim +λ 3 L smooth (8) In formula (9), S i represents the i-th light map, represents the i-th normal light portrait, I i represents the i-th low-light image, N represents the amount of training data, || || 2 represents the Euclidean distance; in formula (10), μ x , μ y Represent the average brightness of the enhanced result and the normal illumination image, σ x , σ y represents the contrast between the enhanced result and the normal illumination image, C 1 , C 2 are constants to prevent division by zero anomalies; in formula (11), p and c represent the number of pixels and channels of the illumination image, respectively. and Represents the square of the horizontal and vertical gradients of the illumination image at pixel p of channel c, and Represents the weights of horizontal and vertical gradients; the final loss function L is composed of three weighted loss functions, which effectively supervises the training process of the network model.