Low-light image enhancement method based on multistage depth feature fusion

Abstract

The invention relates to a low-light image enhancement method based on multistage depth feature fusion, and belongs to the technical field of image enhancement processing. According to the invention, image features under M scales are extracted through the coding network, and feature fusion is carried out on a superposition result of the obtained low-light image, the output feature of each coding branch network and the image features under the previous scales through the multi-stage feature dynamic fusion network, so that M scale dynamic fusion features are obtained. And decoding the fused features of each scale by using a decoding network so as to obtain an enhanced low-light image. Therefore, the low-light image enhancement model can utilize the multi-scale input encoder to extract the multi-scale features of the image, and adopts the multi-level feature fusion network to dynamically fuse the multi-level features. Therefore, the problems of detail blurring, noise amplification and color distortion in the low-light image enhancement process can be effectively solved, and the low-light image enhancement effect is greatly improved.

Description

Technical Field

[0001] The invention relates to a low-light image enhancement method of multi-level depth feature fusion, belonging to the technical field of image enhancement processing. Background Art

[0002] In low-light conditions, images may suffer from problems such as decreased contrast, noise, and loss of details, making it difficult for observers to recognize or understand the information in the image. The goal of low-light image enhancement is to use various image processing methods to make images taken in low-light conditions clearer, rich in details, and to reduce the noise in the image as much as possible. Low-light image enhancement technology has a wide range of applications in various fields. For example, in military reconnaissance, low-light image enhancement technology can improve the quality of images captured by reconnaissance cameras. In unmanned driving, low-light image enhancement technology can improve the ability of on-board cameras to recognize targets such as road signs and obstacles, thereby ensuring the safety and stability of unmanned driving systems.

[0003] Traditional low-light image enhancement techniques include histogram equalization and Retinex. The histogram equalization method enhances the contrast of an image by redistributing the pixel values ​​of the image. However, it is a global processing method that may lead to over-enhancement of the image and amplify the noise in the image. The Retinex method decomposes the low-light image into two parts, illumination and reflection, according to the Retinex model, and optimizes these two parts separately to enhance the image, but this method will produce unnatural artifacts.

[0004] Compared with traditional methods, low-light image enhancement methods based on deep learning can learn complex image features and patterns from a large amount of data, have strong adaptability, and can be applied to low-light image enhancement tasks of different types and scenes. However, there are still some problems in the existing deep learning methods. First, due to low-light conditions, the details and structures in the image become blurred, and it is difficult to clearly distinguish features at different scales, resulting in the inability to recover clear details from low-light images. In addition, the information represented by deep features and shallow features is different. The correlation between different channels cannot be obtained only by element-by-element addition or channel splicing, and it will also lead to feature redundancy. In addition, the pixel value of the low-light image itself is low. During the model training process, the gradient of the image may disappear during the back propagation process, thereby affecting the convergence of the model. Summary of the invention

[0005] The purpose of the present invention is to provide a low-light image enhancement method with multi-level deep feature fusion to solve the problem of unclear details in the current low-light image enhancement process using deep learning.

[0006] In order to solve the above technical problems, the present invention provides a low-light image enhancement method by multi-level depth feature fusion, the method comprising:

[0007] 1) obtaining a low-light image, and continuously downsampling the low-light image M times to obtain low-light images of M scales, where M is greater than or equal to 3 and less than or equal to 5;

[0008] 2) Inputting low-light images of various scales into the trained low-light image enhancement model to enhance the low-light image;

[0009] The low-light image enhancement model includes an encoding network, a multi-level feature dynamic fusion network and a decoding network; the encoding network includes M encoding branch networks, which are used to extract features of low-light images of M scales respectively, so as to extract image features at M scales; the multi-level feature dynamic fusion network includes M fusion branch networks, each fusion branch network is used to perform feature fusion on the superposition result of acquiring the low-light image, the output features of each encoding branch network and the image features at the previous scale, so as to obtain M multi-scale dynamic fusion features; the decoding network includes M+1 decoding branch networks, which are used to decode the fused features of each scale to obtain an enhanced low-light image.

[0010] Furthermore, each fusion branch network includes a global average pooling layer, a 1×1 convolution layer I, a convolution activation layer, a 1×1 convolution layer II, a softmax activation layer and a superposition fusion layer; the global average pooling layer is used to extract global information from the element-by-element superposition fusion input features; the 1×1 convolution layer I is used to process the global information; the 1×1 convolution layer II and the softmax activation layer are used to divide the processed global information into M+1 paths for processing to obtain M+1 weights; the superposition fusion layer is used to perform weighted summation based on the obtained M+1 weights and the M+1 input features.

[0011] Furthermore, one of the M+1 input features of each fusion branch network is a feature of the low-light image after encoding processing, and the other M features are the superposition of the features of the low-light image of the corresponding scale extracted by the corresponding encoding branch network and the low-light image of the previous scale after encoding processing.

[0012] Furthermore, the encoding branch network includes a shallow feature extraction network and a channel attention network, the shallow feature extraction network is used to extract shallow features of the image; the channel attention network is used to assign different weights to each channel in the shallow feature extraction network to suppress irrelevant or redundant channels.

[0013] Furthermore, the decoding branch network consists of a contrast calculation module and two 1*1 convolutional layers, and the contrast calculation module is used to calculate the standard deviation of all channels in the input features.

[0014] Furthermore, the low-light image enhancement model is trained in a multi-level supervised manner, and a normal-light image of a corresponding level is added to the output of each decoding branch network as a supervisory signal.

[0015] Furthermore, the loss function used in the training of the low-light image enhancement model is composed of an L1 loss function, a structural similarity loss function and a perceptual loss function.

[0016] Furthermore, the calculation formula of the L1 loss function is:

[0017] L1=||I out -I gt ||1

[0018] Among them, ||·||1 represents the distance in the sense of a norm, I out and I gt Represent the output of the model and the corresponding normal lighting image respectively.

[0019] Furthermore, the calculation formula of the structural similarity loss function is:

[0020]

[0021] Among them, μ x and μ y are the pixel averages of the output image and the normal image, respectively. and is their variance, σ xy is the covariance, C1 and C2 are constants, and N is the total number of blocks.

[0022] Furthermore, the calculation formula of the perceptual loss function is:

[0023]

[0024] Among them, φ t (·) represents the feature map of the tth convolutional layer in the network used by the perceptual loss, I out and I gt Represent the output of the model and the corresponding normal lighting image respectively.

[0025] The beneficial effects of the present invention are as follows: as an improved invention, the low-light image enhancement model constructed by the present invention includes a coding network, a multi-level feature dynamic fusion network and a decoding network. The image features at M scales are extracted through the coding network. The multi-level feature dynamic fusion network is used to obtain the low-light image, and the superposition results of the output features of each coding branch network and the image features at the previous scale are fused to obtain M-scale dynamic fusion features. The decoding network is used to decode the fused features of each scale to obtain an enhanced low-light image. Therefore, the low-light image enhancement model of the present invention can use a multi-scale input encoder to extract the multi-scale features of the image, and use a multi-level feature fusion network to dynamically fuse the multi-level features. Therefore, the present invention can effectively solve the problems of blurred details, noise amplification, and color distortion in the process of low-light image enhancement, and greatly improve the low-light image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the principle of the low-light image enhancement method of the present invention with multi-level deep feature fusion;

[0027] Figure 2 It is a schematic diagram of the architecture of the multi-level feature dynamic fusion network adopted by the present invention;

[0028] Figure 3 It is a schematic diagram of the architecture of the decoding network adopted by the present invention;

[0029] Figure 4 3 is a schematic diagram of the comparison of low-light image before and after enhancement in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0031] The low-light image enhancement model includes a coding network, a multi-level feature dynamic fusion network and a decoding network; the coding network includes M coding branch networks, which are used to extract features of low-light images of M scales respectively, so as to extract image features at M scales; the multi-level feature dynamic fusion network includes M fusion branch networks, each fusion branch network is used to perform feature fusion on the acquired color low-light image, the output features of each coding branch network and the superposition results of the image features at the previous scale, so as to obtain M multi-scale dynamic fusion features; the decoding network includes M decoding branch networks, which are used to decode the fused features of each scale to obtain an enhanced low-light image.

[0032] 1. Get a low-light image and its corresponding normal-light image.

[0033] In this embodiment, a color low-light image is obtained by shooting and is denoted as And obtain the corresponding normal illumination image by shooting, denoted as I normal In order to facilitate subsequent processing, the present invention also converts the low-light image and the corresponding normal illumination image I normal The image values ​​are normalized. and I normal All images are cropped to 256×256 pixels and normalized to the [0,1] space.

[0034] 2. The low-light image and the corresponding normal-light image are continuously downsampled M times to obtain low-light images and corresponding normal-light images of M scales, which are used as training data.

[0035] The present invention is effective for low light images After M consecutive downsampling, M is greater than or equal to 3 and less than or equal to 5. In this embodiment, M is 3. Then the obtained and Three scales of low-light images, is the low-light image before downsampling, This is the low-light image obtained after downsampling by 2 times. This is the low-light image obtained after downsampling 3 times. is the low-light image obtained after downsampling by 4 times. Similarly, the normal light image I normal The same three scales of downsampling are performed.

[0036] 3. Build a low-light image enhancement model, and use the multi-scale low-light images obtained in step 2 and the corresponding multi-scale normal-light images as training data to train the low-light image enhancement model.

[0037] The low-light image enhancement model constructed by the present invention includes an encoding network, a multi-level feature dynamic fusion network and a decoding network; wherein the encoding network includes M encoding branch networks, which are used to perform feature extraction on low-light images of M scales respectively, so as to extract image features at M scales; the multi-level feature dynamic fusion network includes M fusion branch networks, each fusion branch network is used to perform feature fusion on the acquired color low-light image, the output features of each encoding branch network and the superposition results of the image features at the previous scale, so as to obtain M multi-scale dynamic fusion features; the decoding network includes M decoding branch networks, which are used to decode the fused features of each scale, so as to obtain an enhanced low-light image.

[0038] Specifically, if Figure 1As shown, the coding network in this embodiment includes three coding branch networks (referred to as the first coding branch network, the second coding branch network and the third coding branch network from top to bottom), and each coding score network adopts a multi-scale input encoding (MIE) device for respectively and The multi-scale input encoder includes a shallow feature extraction network FEM for feature extraction of low-light images of three scales. The shallow feature extraction network FEM and a channel attention network are used to extract shallow features of the image. The channel attention network structure is a global average pooling layer-fully connected layer I-fully connected layer II, which is used to assign different weights to each channel in the shallow feature extraction network FEM to suppress irrelevant or redundant channels and improve the performance of the shallow feature extraction network. Figure 1 EB1, EB2, EB3 and EB4 represent encoder blocks (EncoderBlock) of different scales, and DB1, DB2, DB3 and DB4 represent encoder blocks (DecoderBlock) of different scales.

[0039] The multi-level feature dynamic fusion network includes three fusion branch networks, each of which has a structure as follows: Figure 2 As shown in the figure, it is mainly composed of global average pooling layer GAP—1×1 convolution layer I—convolution activation layer—1×1 convolution layer II—softmax activation layer. Figure 1 As shown, from top to bottom, the three fusion branch networks are called the first fusion branch network, the second fusion branch network and the third fusion branch network. The first fusion branch network has four inputs, namely, the low-light image The features after EB1 treatment (denoted as U Ⅰ ), The features extracted by the first encoding branch network and U Ⅰ The features after EB2 treatment (denoted as U Ⅱ ), The features extracted by the second encoding branch network are Ⅱ The characteristics after EB3 treatment (denoted as U Ⅲ )as well as The features extracted by the third encoding branch network are Ⅲ The characteristics after EB4 treatment (denoted as U Ⅳ ), that is U Ⅰ , U Ⅱ , U Ⅲ and U ⅣThe inputs of the second and third fusion branch networks are the same as the inputs of the first fusion branch network, which are U Ⅰ , U Ⅱ , U Ⅲ and U Ⅳ ,and Although the input is the same, the multi-level feature dynamic fusion network of the present invention will adaptively fuse the input features, so that the final output V is different in different branches.

[0040] Each fusion branch network is processed in the same way, such as Figure 2 As shown, Figure 2 1×1×C1 is the feature dimension of the convolution output, indicating that the feature map S is 1×1 in size and has C1 channels; 1×1×C is the feature dimension of the convolution output, indicating that the feature map S is 1×1 in size and has C channels. The specific processing process is as follows: The input feature U is added element by element. Ⅰ , U Ⅱ , U Ⅲ and U Ⅳ After fusion, a global average pooling layer is used to extract the global information of the fused features to obtain the feature map A 1×1 convolutional layer (convolutional layer I) is used to reduce the number of feature channels to reduce the amount of computation, and finally the feature map is obtained. Then four parallel 1×1 convolutional layers (convolutional layer II) and softmax activation function are used to obtain the weight map Finally, the weight map is multiplied by the corresponding input feature to obtain the result of dynamic fusion: V = α × U Ⅰ +β×U Ⅱ +λ×U Ⅲ +μ×U Ⅳ , that is to say, the outputs of the three fused branch networks are all V.

[0041] The decoding network in this embodiment includes four decoding branch networks, each of which is used to decode the output result of the corresponding encoding branch network. Figure 3 As shown in the figure, each decoding branch network consists of a contrast calculation module and two 1*1 convolutional layers. The contrast calculation module is used to calculate the standard deviation of all channels in the input features. The calculation formula is:

[0042]

[0043] in, and β c They represent the elements of each point in channel c and the average value of channel c respectively. H and W represent the length and width of the feature map respectively.

[0044] like Figure 1 As shown, the working principle of the decoding network is: first, the fourth decoding branch network decodes U Ⅳ Perform DB4 processing and divide the processing result M4 into two paths. The result obtained after conversion is The corresponding enhanced image The other path is superimposed with the output result V of the third encoding branch network and then processed by the third decoding branch network to obtain M3. The result obtained after M3 is converted is The corresponding enhanced image M3 is then superimposed with the output result V of the second encoding branch network and processed by the second decoding branch network to obtain M2. The result obtained after M2 is converted is The corresponding enhanced image M2 is then superimposed with the output result V of the first encoding branch network and processed by the first decoding branch network to obtain M1. The result obtained after M1 is The corresponding enhanced image Enhanced images That is the ultimate desired result.

[0045] During training, the present invention adopts a multi-level supervision method to add a corresponding normal illumination image to the output of each decoding branch as a supervision signal. The calculation formula of the multi-level supervision is as follows:

[0046]

[0047] Among them, L (k) represents the loss function used by the kth branch, α k is the weight of each branch. Multi-level supervision means supervising the output of each branch. (k) The calculation formula is as follows:

[0048]

[0049] in They represent L1 loss function, structural similarity loss function and perceptual loss function respectively, and ω1, ω2 and ω3 represent the weights of different loss functions. The calculation formula of L1 loss function is:

[0050] L1=||I out -I gt ||1

[0051] Among them, ||·||1 represents the distance in the sense of a norm, I out and I gt Represent the output of the model and the corresponding normal lighting image respectively. This function can constrain the distance between the output and the label to be as close as possible.

[0052] The calculation formula of the structural similarity loss function is:

[0053]

[0054] Among them, μ x and μ y are the pixel averages of the output image and the normal image, respectively. and is their variance, σ xy is the covariance. To prevent the denominator from becoming 0, C1 and C2 are added to the denominator. This loss function can retain a more complete image structure. C1 and C2 are constants to avoid instability caused by the denominator being close to 0. In practical applications, it is necessary to use a sliding window to divide the image into blocks, and N is the total number of blocks.

[0055] The calculation formula of the perceptual loss function is:

[0056]

[0057] Among them, φ t (·) represents the feature map of the tth convolutional layer in the network used by the perceptual loss. In this embodiment, the network used by the perceptual loss is the VGG-16 network. This loss function can restore finer image details and improve visual quality.

[0058] Through the above training, a trained low-light image enhancement model can be obtained.

[0059] 4. Use the trained low-light image enhancement model to enhance the low-light image to be enhanced.

[0060] Get the low-light image to be enhanced, crop it to an image of 256×256 pixels, and standardize it to the [0,1] space. Then, perform three consecutive downsampling on the cropped image according to step 2, and input the image before downsampling and the three images after downsampling into the trained low-light image enhancement model. The trained low-light image enhancement model can then output the enhanced image.

[0061] like Figure 4 As shown, the input image is the acquired low-light image, and the enhanced image is obtained after being processed by the trained low-light image enhancement model of the present invention. It can be seen that the details of the enhanced image are clearer.

[0062] Therefore, the low-light image enhancement model constructed by the present invention can use a multi-scale input encoder to extract multi-scale features of the image, use a multi-level feature fusion network to dynamically fuse multi-level features, and take into account the correlation between channels and reduce redundant information. In the image reconstruction process, a contrast perception network is used to extract image contrast information; at the same time, the present invention also uses L1 loss function, structural similarity loss function and perception loss function to train the low-light image enhancement model in a multi-level supervised manner. The present invention uses a trained low-light image enhancement model to effectively solve the problems of blurred details, noise amplification, and color distortion in the low-light image enhancement process, greatly improving the low-light image enhancement effect, and the convergence is also improved.

Claims

1. A low-light image enhancement method based on multi-level deep feature fusion, characterized in that: The method includes: 1) obtaining a low-light image, and continuously downsampling the low-light image M times to obtain low-light images of M scales, where M is greater than or equal to 3 and less than or equal to 5; 2) Inputting low-light images of various scales into the trained low-light image enhancement model to enhance the low-light image; The low-light image enhancement model includes an encoding network, a multi-level feature dynamic fusion network and a decoding network; the encoding network includes M encoding branch networks, which are used to extract features of low-light images of M scales respectively, so as to extract image features at M scales; the multi-level feature dynamic fusion network includes M fusion branch networks, each fusion branch network is used to perform feature fusion on the superposition result of acquiring the low-light image, the output features of each encoding branch network and the image features at the previous scale, so as to obtain M multi-scale dynamic fusion features; the decoding network includes M+1 decoding branch networks, which are used to decode the fused features of each scale to obtain an enhanced low-light image.

2. The low-light image enhancement method based on multi-level deep feature fusion according to claim 1, characterized in that: Each fusion branch network includes a global average pooling layer, a 1×1 convolution layer I, a convolution activation layer, a 1×1 convolution layer II, a softmax activation layer and a superposition fusion layer; the global average pooling layer is used to extract global information from the element-by-element superposition fusion input features; the 1×1 convolution layer I is used to process the global information; the 1×1 convolution layer II and the softmax activation layer are used to divide the processed global information into M+1 paths for processing to obtain M+1 weights; The stacking fusion layer is used to perform weighted summation based on the obtained M+1 weights and M+1 input features.

3. The low-light image enhancement method of multi-level deep feature fusion according to claim 1, characterized in that: One of the M+1 input features of each fusion branch network is the feature of the low-light image after encoding processing, and the other M features are the superposition of the features of the low-light image of the corresponding scale extracted by the corresponding encoding branch network and the low-light image of the previous scale after encoding processing.

4. The low-light image enhancement method of multi-level deep feature fusion according to claim 1, characterized in that: The encoding branch network includes a shallow feature extraction network and a channel attention network. The shallow feature extraction network is used to extract shallow features of the image; the channel attention network is used to assign different weights to each channel in the shallow feature extraction network to suppress irrelevant or redundant channels.

5. The low-light image enhancement method of multi-level deep feature fusion according to claim 1, characterized in that: The decoding branch network consists of a contrast calculation module and two 1*1 convolutional layers, and the contrast calculation module is used to calculate the standard deviation of all channels in the input features.

6. The low-light image enhancement method of multi-level deep feature fusion according to claim 1, characterized in that: The low-light image enhancement model is trained in a multi-level supervised manner, and a normal-light image of a corresponding level is added to the output of each decoding branch network as a supervisory signal.

7. The low-light image enhancement method of multi-level deep feature fusion according to claim 1 or 6, characterized in that: The loss function used in the training of the low-light image enhancement model is composed of an L1 loss function, a structural similarity loss function and a perceptual loss function.

8. The low-light image enhancement method of multi-level deep feature fusion according to claim 7, characterized in that: The calculation formula of the L1 loss function is: L1=||I out -I gt ||1 Among them, ||·||1 represents the distance in the sense of a norm, I out and I gt Represent the output of the model and the corresponding normal lighting image respectively.

9. The low-light image enhancement method of multi-level deep feature fusion according to claim 7, characterized in that: The calculation formula of the structural similarity loss function is: Among them, μ x and μ y are the pixel averages of the output image and the normal image, respectively. and is their variance, σ xy is the covariance, C1 and C2 are constants, and N is the total number of blocks.

10. The low-light image enhancement method of multi-level deep feature fusion according to claim 7, characterized in that: The calculation formula of the perceptual loss function is: Among them, φ t (·) represents the feature map of the tth convolutional layer in the network used by the perceptual loss, I out and I gt Represent the output of the model and the corresponding normal lighting image respectively.

Images