A low-light image enhancement method based on multi-level feature extraction and fusion

Abstract

The present invention discloses a low-light image enhancement method based on multi-level feature extraction and fusion, comprising: preprocessing a low-light image to be enhanced; inputting the preprocessed low-light image into a trained low-light enhancement network model to obtain a low-light enhancement map; the low-light enhancement network model is based on a convolutional neural network; the present invention extracts features at different levels by repeatedly utilizing a feature extraction module in the low-light enhancement network model, extracts features at different scales by using a hole convolution branch in the feature extraction module, adjusts the weights of the extracted features by using a channel attention module and a spatial attention module, and then enhances the obtained features at different levels and scales by using a feature enhancement network module, and finally performs feature fusion, so that the details, structure, contrast, color and other features of the enhanced image are more obvious.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, in particular to an end-to-end RGB color image restoration technology, and relates to a low-light image enhancement method based on multi-level feature extraction and fusion. Background Art

[0002] Images and videos carry rich real-world content and information. Intelligent systems capture and process image and video data to perform various tasks, such as object detection, classification, segmentation, recognition, scene understanding, and 3D reconstruction. These tasks are then applied to many practical applications, including autonomous driving, video surveillance, and virtual / augmented reality. However, intelligent systems rely heavily on the quality of the input images and videos. They perform well when the input data is high-quality, but perform poorly when it is low-quality. In real life, not all captured images and videos are high-quality, reflecting the information and content of real scenes. For example, in low-light conditions, insufficient light intensity can lead to insufficient light entering the camera, affecting the visual quality of the captured image. Images can exhibit significant distortion, such as low brightness, low contrast, color distortion, and loss of detail. Therefore, improving the quality of low-light images without requiring additional and expensive hardware is urgently needed in practical applications. Low-light image enhancement technology not only improves image quality and restores important image information, but also reduces noise. Images and videos processed through low-light enhancement are more consistent with human visual perception.

[0003] In recent years, deep learning has achieved tremendous success in low-level image processing. End-to-end networks and generative adversarial networks have been adopted in various applications, including image super-resolution and image denoising. Methods for low-light image enhancement have also been proposed. The MBLLEN algorithm (F. Lv, et al. MBLLEN: Low-light Image / Video Enhancement Using CNNs [C], BMVC, 2018) is a multi-branch low-light image enhancement network. The core idea of ​​this algorithm is to extract rich image features at different levels, thereby enhancing the image through multiple sub-networks, and finally generating the output image through multi-branch fusion. Image quality is significantly improved in both objective metrics such as PSNR and SSIM, as well as subjective human visual evaluation. This algorithm is not only suitable for image enhancement, but also for low-light video enhancement. However, this low-light video enhancement process does not take into account the information between video frames. The EnlightenGAN algorithm (Y. Jiang, et al. EnlightenGAN: Deep Light Enhancement without Paired Supervision [J]. IEEE Transactions on Image Processing, 30: 2340-2349, 2021) is an efficient unsupervised generative adversarial network that does not require low-light / normal-light image pairs for training. The algorithm uses information extracted from the input image itself to regularize unpaired training. Compared with existing supervised training methods, it is more flexible and has the advantage of being more generalized for real low-light images. This also opens the precedent for the application of generative adversarial networks in the field of low-light image enhancement, but the output image has excessive enhancement in local areas. Summary of the Invention

[0004] The purpose of the present invention is to provide a low-light image enhancement method based on multi-level feature extraction and fusion, which solves the problems of low visibility, low contrast, color distortion and invisible details of low-light images in the prior art.

[0005] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0006] In a first aspect, a low-light image enhancement method based on multi-level feature extraction and fusion is provided, comprising:

[0007] Preprocessing the low-light image to be enhanced; inputting the preprocessed low-light image into a trained low-light enhancement network model, and obtaining a low-light enhancement map according to the output of the low-light enhancement network model;

[0008] The training process of the low-light enhancement network model includes:

[0009] S1: Obtain low-light images and corresponding normal-light images as a dataset, perform pixel normalization on the images in the dataset, and obtain a training dataset;

[0010] S2: Input the low-light images in the training dataset into the low-light enhancement network model and perform feature extraction through at least three feature extraction modules in sequence;

[0011] S3: The feature map output by each feature extraction module serves as the input of the corresponding feature enhancement module and the next feature extraction module;

[0012] S4: The feature enhancement images output by the multiple feature enhancement modules are input into the feature fusion module for feature fusion to obtain the final low-light enhancement image;

[0013] S5: Use the loss function to constrain the difference between the low-light enhancement image and the corresponding normal-light image, and continuously adjust the model parameters until the model converges and the model training is completed.

[0014] In some embodiments, pixel normalization is performed on the images in the dataset, including normalizing the pixel values ​​of the images in the dataset to a range of [0, 1] to obtain a training dataset.

[0015] In some embodiments, the feature extraction module processing includes:

[0016] The input image is input to the first convolution Conv layer, and the output of the first convolution Conv layer is input to the two Conv branches respectively. The feature maps output by the two Conv branches are then channel-connected through the first Concatenate layer. The output of the first Concatenate layer and the output of the first convolution Conv layer are channel-connected through the second Concatenate layer. The output image of the second Concatenate layer is then passed through the Conv layer, the channel attention module, and the spatial attention module in sequence, and finally the extracted feature map is output;

[0017] Among them, the first convolution Conv layer performs shallow feature extraction on the input image, the convolution kernel size is 3×3, the step size is 1, the activation function uses the ReLU activation function, and the total number of feature maps is 8; both Conv branches contain two Conv layers, and the two Conv layers of the first Conv branch both use ordinary convolution, the convolution kernel size is 3×3, the step size is 1, the activation function is ReLU, and the total number of feature maps is 8; the first Conv layer in the second Conv branch uses void convolution, the convolution kernel size is 3×3, the void rates are 1, 2, and 3 respectively, the step size is 1, the activation function is ReLU, and the total number of feature maps is 8, the second Conv layer in the second Conv branch uses ordinary convolution, the convolution kernel size is 3×3, the step size is 1, the activation function is ReLU, and the total number of feature maps is 8; the second Conv branch has the same number of parameters as the first Conv branch, but due to the use of void convolution, the receptive field is larger, and the two branches extract features in parallel to jointly consider the context information of the feature map.

[0018] In some embodiments, the channel attention module processing process includes:

[0019] Input the input feature map into two branches respectively;

[0020] In the first branch, the input feature map first passes through the GlobalAvgPooling layer to calculate the global average of each channel, and the output size is 1×8. It then passes through a Reshape layer with an output size of 1×1×8 and two Dense fully connected layers with an output size of 1×1×1. The Reshape layer assigns a new shape to the image output by the global pooling layer according to the channel.

[0021] In the second branch, the input feature map first passes through the GlobalMaxPooling layer to calculate the global maximum value of each channel channel by channel, and the output size is 1×8. It then passes through a Reshape layer with an output size of 1×1×8 and two Dense fully connected layers with an output size of 1×1×1.

[0022] The outputs of the two branches are added pixel by pixel through an Add layer and activated with a sigmoid activation function. The channel weight feature map output by the Add layer and the feature map input at the beginning of the module are multiplied pixel by pixel through a Multiply layer. The channel attention mechanism is used to obtain the correlation on the channel, and the output is a channel attention adjustment map of size H×W×8, where H and W represent the length and width of the output image, respectively.

[0023] In some embodiments, the spatial attention module processing includes:

[0024] The input feature maps are input to the maximum pooling MaxPool layer and the average pooling AvgPool layer respectively, and the output feature map size is H×W×1;

[0025] The outputs of the maximum pooling layer and the average pooling layer are passed through a Concatenate layer, and channel-wise concatenation is performed. The output feature map size is H×W×2. The concatenated feature map is further extracted through a Conv layer to obtain a pixel weight map. The convolution kernel size is 3×3, the stride is 1, the number of mappings is 1, the activation function is sigmoid, and the output feature map size is H×W×1.

[0026] Compare the feature map input to the spatial attention module with the pixel weights output by the Conv layer Figure 1 It then passes through a Multiply layer for pixel multiplication, uses the spatial attention mechanism to obtain spatial correlation, and outputs a spatial attention adjustment map of size H×W×8 as a supplement to the channel attention.

[0027] In some embodiments, the feature enhancement module processing includes:

[0028] The input feature map is sequentially sent to 4 Conv layers and 3 Conv2DTranspose layers to extract feature information. Among them, the convolution kernel size of the 4 Conv layers is 3×3 and the step size is 1. The activation function is ReLU, and the number of output channels is 8, 8, 16, and 32 respectively; the convolution kernel size of the 3 Conv2DTranspose layers is 3×3 and the step size is 1. The activation function is ReLU, and the number of output channels is 16, 8, and 3 respectively. The extracted feature information is then sequentially input into the channel attention module and the spatial attention module, and finally a feature map of size H×W×3 is output.

[0029] In some embodiments, the feature fusion module processing process includes: first connecting the feature enhancement maps output by all feature enhancement modules through a Concatenate layer, and then passing the obtained output through a Conv layer. The convolution kernel size of the Conv layer is 1×1, the step size is 1, and the activation function is ReLU. The output image with a size of H×W×3 is the final low-light enhancement map.

[0030] In some embodiments, the loss function Loss of the low-light enhancement network model total The expression is:

[0031] Loss total =L MAE +L SSIM +L perc +L region +L color

[0032] in, is the mean absolute error loss at the pixel level, N is the number of images, P is the total number of pixels in an image, the subscript p represents a pixel in the image, and y represents the original normal illumination image. A low-light enhancement map representing the output;

[0033] is the structural similarity loss, μ y 、 Represents image y, The average pixel value, Represents image y, The variance of Represents image y and The covariance between them, C1 and C2 are two constants to avoid the denominator being 0 in the formula;

[0034] is the regional loss function, where the subscripts l and h represent the low-light area and other areas in the image respectively, and w l and w h represents the proportional coefficient between the two regions, i and j are the horizontal and vertical coordinates of the pixel points on the image, m and n are the number of horizontal and vertical pixels in the region, |·| is the absolute value operation, y l (i, j) represents the pixel intensity of the i-th row and j-th column in the low-light area of ​​the normal-light image, represents the pixel intensity of the i-th row and j-th column in the low-light area of ​​the low-light enhancement image, y h (i, j) represents the pixel intensity of the i-th row and j-th column in other areas of the normal illumination image, Represents the pixel intensity of the i-th row and j-th column in other areas of the low-light enhancement image;

[0035] is the perceptual loss, using the VGG-19 network as the feature map extractor, h i,j 、w i,j 、c i,j Represents the length, width, and channel dimensions of the feature map, φ i,j Indicates that the jth convolutional layer of the i-th block of the VGG-19 network is used to extract the feature map;

[0036] is the color loss, where G(·) represents Gaussian filtering.

[0037] In some embodiments, when calculating the region loss function, the method for determining the low-light area and other areas in the image is:

[0038] First, the R, G, and B channels of the color image are combined in proportion to form a single-channel image Img = r·R+g·G+b·B, where r, g, and b are the proportional coefficients of each channel; then all pixel intensity values ​​in the single-channel image Img are sorted from small to large, and the pixel areas with the first a% of the smallest intensity values ​​are defined as low-light areas, and the remaining pixel areas are other areas.

[0039] In a second aspect, the present invention provides a low-light image enhancement device based on multi-level feature extraction and fusion, comprising a processor and a storage medium;

[0040] The storage medium is used to store instructions;

[0041] The processor is configured to operate according to the instructions to execute the steps of the method according to the first aspect.

[0042] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processor.

[0043] Advantages of the present invention:

[0044] 1. This paper combines the channel attention mechanism and the spatial attention mechanism to make the network model pay more attention to some important channel features and pixel features, thereby fully extracting and utilizing local and global features in the image;

[0045] 2. The network in this invention uses different dilated convolutions each time it extracts features, and obtains feature maps of different scales through receptive fields of different sizes, allowing the network to focus on more contextual information in the image, making the feature information extraction more comprehensive;

[0046] 3. The present invention uses a multi-level feature extraction mechanism to extract as many features as possible, and at the same time uses a feature fusion mechanism to ensure that the extracted features can be organically combined and fully utilized. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the method of the present invention;

[0048] Figure 2 It is a flow chart of the feature extraction module of the method of the present invention;

[0049] Figure 3 This is a flow chart of the channel attention module of the method of the present invention;

[0050] Figure 4 This is a flowchart of the spatial attention module of the method of the present invention;

[0051] Figure 5 It is a flow chart of the feature enhancement module of the method of the present invention;

[0052] Figure 6 The figures show the comparison of the enhancement results of the embodiment and other methods on the MIT dataset, where (a) is the input low-light image, (b) is the enhancement result of the MBLLEN algorithm, (c) is the enhancement result of the ZeroDCE algorithm, (d) is the enhancement result of the EnlightenGAN algorithm, (e) is the normal-light image, and (f) is the enhancement result of the embodiment.

[0053] Figure 7 This is a comparison chart of the enhancement results of the embodiment and the enhancement results of other methods on the LOL dataset, (a) is the input low-light image, (b) is the enhancement result of the MBLLEN algorithm, (c) is the enhancement result of the ZeroDCE algorithm, (d) is the enhancement result of the EnlightenGAN algorithm, (e) is the normal light image, and (f) is the enhancement result of the embodiment. DETAILED DESCRIPTION

[0054] The following is a clear and complete description of the technical solutions in the examples of the invention, with reference to the accompanying drawings. Obviously, the embodiments described are only some examples of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by persons of ordinary skill in the art without inventive effort are also within the scope of protection of the present invention.

[0055] In the description of the present invention, "several" means more than one, "plurality" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0056] In the description of the present invention, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the exemplary expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0057] Example 1

[0058] As shown in Question 1, a low-light image enhancement method based on multi-level feature extraction and fusion includes: preprocessing the low-light image to be enhanced; inputting the preprocessed low-light image into a trained low-light enhancement network model, and obtaining a low-light enhancement map based on the output of the low-light enhancement network model;

[0059] The training process of the low-light enhancement network model includes:

[0060] S1: Obtain low-light images and corresponding normal-light images as a dataset, perform pixel normalization on the images in the dataset, and obtain a training dataset;

[0061] S2: Input the low-light images in the training dataset into the low-light enhancement network model and perform feature extraction through at least three feature extraction modules in sequence;

[0062] S3: The feature map output by each feature extraction module serves as the input of the corresponding feature enhancement module and the next feature extraction module;

[0063] S4: The feature enhancement images output by the multiple feature enhancement modules are input into the feature fusion module for feature fusion to obtain the final low-light enhancement image;

[0064] S5: Use the loss function to constrain the difference between the low-light enhancement image and the corresponding normal-light image, and continuously adjust the model parameters until the model converges and the model training is completed.

[0065] The present invention mainly extracts features at different levels by repeatedly utilizing the feature extraction module, extracts features of different scales through the dilated convolution branch in the feature extraction module, adjusts the weights of the extracted features through the channel attention module and the spatial attention module, and then enhances the features of different levels and scales obtained through the feature enhancement network module. Finally, feature fusion is performed to make the details, structure, contrast, color and other features of the enhanced image more obvious.

[0066] In this embodiment, the LOL dataset and the MIT dataset are used; because the LOL dataset collects images in non-synthetic real scenes, the MIT dataset contains original unprocessed images and corresponding high-quality images modified and processed by professionals, which can better verify the effectiveness of the method of the present invention; the LOL dataset contains 500 groups of low-light images, normal-light images and their corresponding low-light images. This embodiment uses 485 groups of images, and normalizes the image pixel values ​​to the range of [0,1] as a training dataset, and the remaining 15 groups of images are used as a test set; this embodiment also uses 3200 groups of original images and high-quality images processed by Expert C in the MIT dataset, and normalizes the image pixel values ​​to the range of [0,1] as a training dataset, and takes another 22 groups of images as a test set.

[0067] like Figure 2 As shown in Figure 2, the process of feature extraction module processing image data is as follows:

[0068] The input image is input to the first convolutional (Conv) layer, and its output is input to two Conv branches respectively. The feature maps output by the two Conv branches are then channel-connected through a Concatenate layer. The output of the Concatenate layer and the output of the first convolutional Conv layer are channel-connected through another Concatenate layer. The output image is then passed through the Conv layer, the channel attention module, and the spatial attention module in sequence, and finally the extracted feature map is output;

[0069] Among them, the first convolution Conv layer performs shallow feature extraction on the input image, the convolution kernel size is 3×3, the step size is 1, the activation function uses the ReLU activation function, and the total number of feature maps is 8; both Conv branches contain two Conv layers, and the two Conv layers of the first Conv branch both use ordinary convolution, the convolution kernel size is 3×3, the step size is 1, the activation function is ReLU, and the total number of feature maps is 8; the first Conv layer in the second Conv branch uses void convolution, the convolution kernel size is 3×3, the void rates are 1, 2, and 3 respectively, the step size is 1, the activation function is ReLU, and the total number of feature maps is 8, and the second Conv layer uses ordinary convolution, the convolution kernel size is 3×3, the step size is 1, the activation function is ReLU, and the total number of feature maps is 8; the second Conv branch has the same number of parameters as the first Conv branch, but because it uses void convolution, the receptive field is larger, and the two branches extract features in parallel, which is conducive to jointly considering the context information of the feature map.

[0070] The main function of the feature extraction module is to extract features from the input image. It uses dilated convolution to expand the receptive field to extract different contextual information without changing the convolution kernel parameters, thereby achieving the purpose of extracting and enhancing feature maps of different levels and scales. The channel attention layer adjusts the channels of the input image in the form of weights. The spatial attention layer supplements the channel attention layer and also adjusts the pixels of the input image in the form of weights, making the extracted features richer and more detailed.

[0071] like Figure 3 As shown in Figure 2, the process of the channel attention module processing image data is as follows: first, the input feature map is input into two branches respectively; in the first branch, the input feature map first passes through the global average pooling layer to calculate the global average value of each channel channel by channel, and the output result size is 1×8, and then passes through a Reshape layer with an output size of 1×1×8 and two Dense fully connected layers with an output size of 1×1×1. The Reshape layer gives the image output by the global pooling layer a new shape according to the channel; in the second branch, the input feature map first passes through the global maximum pooling layer (GlobalMaxPooling The ng) layer calculates the global maximum of each channel channel by channel, and the output result is of size 1×8. It then passes through a Reshape layer with an output size of 1×1×8 and two Dense fully connected layers with an output size of 1×1×1. The outputs of the two branches are added pixel by pixel through an Add layer and activated with a sigmoid activation function. Finally, the channel weight feature map output by the Add layer is multiplied pixel by pixel with the feature map input at the beginning of the module through a Multiply layer. The channel attention mechanism is used to obtain the correlation on the channel, and the output is a channel attention adjustment map of size H×W×8, where H and W represent the length and width of the output image, respectively.

[0072] like Figure 4 As shown in Figure 2, the process of spatial attention module processing image data is as follows: first, the input feature map is input into the maximum pooling (MaxPool) layer and the average pooling (AvgPool) layer respectively, and the output feature map size is H×W×1; then the output of the two pooling layers passes through a Concatenate layer, and the channel is connected according to the channel, and the output feature map size is H×W×2. The connected feature map passes through a Conv layer to further extract features to obtain a pixel weight map, the convolution kernel size is 3×3, the step size is 1, the mapping number is 1, the activation function is sigmoid, and the output feature map size is H×W×1; finally, the feature map at the beginning of the module is combined with the pixel weight map Figure 1It then passes through a Multiply layer for pixel multiplication, uses the spatial attention mechanism to obtain spatial correlation, and outputs a spatial attention adjustment map of size H×W×8 as a supplement to the channel attention.

[0073] like Figure 5 As shown in the figure, the process of feature enhancement module processing image data is as follows: first, the input feature map is sequentially sent to 4 Conv layers and 3 Conv2DTranspose layers to extract feature information. Among them, the convolution kernel size of the 4 Conv layers is 3×3 and the step size is 1. The activation function is ReLU, and the number of output channels is 8, 8, 16, and 32 respectively; the convolution kernel size of the 3 Conv2DTranspose layers is 3×3 and the step size is 1. The activation function is ReLU, and the number of output channels is 16, 8, and 3 respectively; then the extracted feature information is sequentially input into the channel attention module and the spatial attention module, and finally the feature map of size H×W×3 is output.

[0074] When training the low-light enhancement network model, the MAE loss function performs better than the MSE in terms of the contrast of the target contour and the smoothing effect of the uniform area; at the same time, the SSIM loss function introduces structural constraints to well restore the structure and details of the image; the regional loss function can balance the degree of enhancement between the low-light area and other areas in the image; the perceptual loss function can constrain the content difference between the real image and the predicted image, maintaining the fidelity of image perception and details; the color loss function aims to measure the color difference between the two images, so that the network-enhanced image has similar colors to the reference image. In the present invention, the loss function expression of the low-light enhancement network model is:

[0075] Loss total =L MAE +L SSIM +L perc +L region +L color

[0076] in, is the mean absolute error loss at the pixel level, N is the number of images, P is the total number of pixels in an image, the subscript p represents a pixel in the image, and y represents the original normal illumination image. A low-light enhancement map representing the output;

[0077] is the structural similarity loss, μ y 、 Represents image y, The average pixel value, Represents image y, The variance of Represents image y and The covariance between them, C1 and C2 are two constants to avoid the denominator being 0 in the formula;

[0078] is the regional loss function, where the subscripts l and h represent the low-light area and other areas in the image respectively, and w l and w h represents the proportional coefficient between the two regions, i and j are the horizontal and vertical coordinates of the pixel points on the image, m and n are the number of horizontal and vertical pixels in the region, |·| is the absolute value operation, y l (i, j) represents the pixel intensity of the i-th row and j-th column in the low-light area of ​​the normal-light image, represents the pixel intensity of the i-th row and j-th column in the low-light area of ​​the low-light enhancement image, y h (i, j) represents the pixel intensity of the i-th row and j-th column in other areas of the normal illumination image, Represents the pixel intensity of the i-th row and j-th column in other areas of the low-light enhancement image;

[0079] is the perceptual loss, using the VGG-19 network as the feature map extractor, h i,j 、w i,j 、c i,j Represents the length, width, and channel dimensions of the feature map, φ i,j Indicates that the jth convolutional layer of the i-th block of the VGG-19 network is used to extract the feature map;

[0080] is the color loss, where G(·) represents Gaussian filtering.

[0081] When calculating the regional loss function, the three channels R, G, and B of the color image are first combined in proportion to form a single-channel image Img = r·R+g·G+b·B, where the proportional coefficients of each channel are set to r = 0.39, g = 0.5, and b = 0.11; then all pixel intensity values ​​in the single-channel image Img are sorted from small to large, and the areas with the first 40% of pixels with the smallest intensity values ​​are defined as low-light areas, and the remaining pixel areas are defined as other areas.

[0082] In this embodiment, the number of iterations of network model training is set to 200, and the learning rate is updated once per iteration. During the iteration process, the models and parameters that achieve good results on the validation set are saved.

[0083] In this example, the MBLLEN, ZeroDCE, and EnlightenGAN algorithms were compared. PSNR, SSIM, LOE, NIQE, and LPIPS were used as evaluation metrics. PSNR stands for Peak Signal-to-Noise Ratio, SSIM stands for Structural Similarity, LOE stands for Luminance Sequential Difference, NIQE stands for Natural Image Quality Evaluation, and LPIPS stands for Perceived Image Block Similarity. The average PSNR, SSIM, LOE, NIQE, and LPIPS of the test image were calculated.

[0084] The following combination Figure 6 、 Figure 7 The present invention is further illustrated by the effect evaluation of the examples shown in Tables 1 and 2.

[0085] Table 1 Comparison of quantitative tests on the MIT dataset

[0086]

[0087] Table 2 Comparison of quantitative tests on the LOL dataset

[0088]

[0089] Table 1 shows the results of evaluating the enhancement results of the embodiment and other methods on the MIT dataset using different indicators. It can be seen that the results of the method of the present invention in terms of PSNR, SSIM, LOE, and LPIPS are significantly better than those of other methods, and the results in terms of NIQE are not much different from those of the MBLLEN algorithm.

[0090] Table 2 shows the results of evaluating the enhancement results of the embodiment and other methods on the LOL dataset using different indicators. It can be seen that the results of the method of the present invention in terms of PSNR, SSIM, and LOE are significantly better than those of other methods, and the results in terms of NIQE and LPIPS are not much different from those of the MBLLEN algorithm.

[0091] Figure 6 and Figure 7 The visual effects of the enhancement results of the embodiment are compared with those of the enhancement results of other methods on the MIT dataset and the LOL dataset respectively. It can be seen that the results of the method of the present invention are superior to those of other methods in terms of details, structure, color, etc., and the visual effect is more natural.

[0092] Example 2

[0093] In a second aspect, this embodiment provides a low-light image enhancement device based on multi-level feature extraction and fusion, including a processor and a storage medium;

[0094] The storage medium is used to store instructions;

[0095] The processor is configured to operate according to the instructions to execute the steps of the method according to embodiment 1.

[0096] Example 3

[0097] In a third aspect, this embodiment provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Example 1 are implemented.

[0098] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

Claims

1. A low-light image enhancement method based on multi-level feature extraction and fusion, characterized in that: include: Preprocess the low-light image to be enhanced; Inputting the preprocessed low-light image into a trained low-light enhancement network model, and obtaining a low-light enhancement map according to the output of the low-light enhancement network model; The training process of the low-light enhancement network model includes: S1: Obtain low-light images and corresponding normal-light images as a dataset, perform pixel normalization on the images in the dataset, and obtain a training dataset; S2: Input the low-light images in the training dataset into the low-light enhancement network model and perform feature extraction through at least three feature extraction modules in sequence; S3: The feature map output by each feature extraction module serves as the input of the corresponding feature enhancement module and the next feature extraction module; S4: The feature enhancement images output by the multiple feature enhancement modules are input into the feature fusion module for feature fusion to obtain the final low-light enhancement image; S5: Use the loss function to constrain the difference between the low-light enhancement image and the corresponding normal-light image, and continuously adjust the model parameters until the model converges, completing the model training; Among them, the feature extraction module processing process includes: inputting the input image into the first convolution Conv layer, inputting the output of the first convolution Conv layer into the two Conv branches respectively, and then channel-connecting the feature maps output by the two Conv branches through the first Concatenate layer, and channel-connecting the output of the first Concatenate layer and the output of the first convolution Conv layer through the second Concatenate layer, and then passing the output image of the second Concatenate layer through the Conv layer, the channel attention module, and the spatial attention module in sequence, and finally outputting the extracted feature map; wherein, the first convolution Conv layer performs shallow feature extraction on the input image; the two Conv branches both contain two Conv layers, the two Conv layers of the first Conv branch both use ordinary convolution, the first Conv layer in the second Conv branch uses void convolution, and the second Conv layer in the second Conv branch uses ordinary convolution, and the two branches extract features in parallel to jointly consider the context information of the feature map.

2. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 1, characterized in that: Perform pixel normalization on the images in the dataset, including: Normalize the pixel values ​​of the images in the dataset to the range of [0, 1] to obtain the training dataset.

3. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 1, characterized in that: The convolution kernel size of the first convolution Conv layer is 3×3, the stride is 1, the activation function uses the ReLU activation function, and the total number of feature maps is 8; The convolution kernel size of the two Conv layers of the first Conv branch is 3×3, the stride is 1, the activation function is ReLU, and the total number of feature maps is 8; The convolution kernel size of the first Conv layer in the second Conv branch is 3×3, the stride is 1, the activation function is ReLU, and the total number of feature maps is 8; the convolution kernel size of the second Conv layer in the second Conv branch is 3×3, the stride is 1, the activation function is ReLU, and the total number of feature maps is 8; The second Conv branch has the same number of parameters as the first Conv branch, but uses dilated convolution and has a larger receptive field.

4. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 3, characterized in that: The channel attention module processing process includes: Input the input feature map into two branches respectively; In the first branch, the input feature map first passes through the GlobalAvgPooling layer to calculate the global average of each channel channel by channel, and the output result size is 1×8. It then passes through a Reshape layer with an output size of 1×1×8 and two Dense fully connected layers with an output size of 1×1×1. The Reshape layer assigns a new shape to the image output by the global pooling layer according to the channel; In the second branch, the input feature map first passes through the GlobalMaxPooling layer to calculate the global maximum value of each channel channel by channel, and the output size is 1×8. It then passes through a Reshape layer with an output size of 1×1×8 and two Dense fully connected layers with an output size of 1×1×1. The outputs of the two branches are added pixel by pixel through an Add layer and activated with a sigmoid activation function. The channel weight feature map output by the Add layer and the feature map input at the beginning of the module are multiplied pixel by pixel through a Multiply layer. The channel attention mechanism is used to obtain the correlation on the channel, and the output is a channel attention adjustment map of size H×W×8, where H and W represent the length and width of the output image, respectively.

5. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 3, characterized in that: The spatial attention module processing process includes: The input feature maps are input to the maximum pooling MaxPool layer and the average pooling AvgPool layer respectively, and the output feature map size is H×W×1; The outputs of the maximum pooling layer and the average pooling layer are passed through a Concatenate layer, and channel-wise concatenation is performed. The output feature map size is H×W×2. The concatenated feature map is further extracted through a Conv layer to obtain a pixel weight map. The convolution kernel size is 3×3, the stride is 1, the number of mappings is 1, the activation function is sigmoid, and the output feature map size is H×W×1. The feature map input to the spatial attention module and the pixel weight map output by the Conv layer are passed through a Multiply layer for pixel multiplication. The spatial attention mechanism is used to obtain spatial correlation, and the output is a spatial attention adjustment map of size H×W×8 as a supplement to the channel attention.

6. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 1, characterized in that: The feature enhancement module processing process includes: The input feature map is sequentially sent to 4 Conv layers and 3 Conv2DTranspose layers to extract feature information. Among them, the convolution kernel size of the 4 Conv layers is 3×3 and the step size is 1. The activation function is ReLU, and the number of output channels is 8, 8, 16, and 32 respectively; the convolution kernel size of the 3 Conv2DTranspose layers is 3×3 and the step size is 1. The activation function is ReLU, and the number of output channels is 16, 8, and 3 respectively. The extracted feature information is then sequentially input into the channel attention module and the spatial attention module, and finally a feature map of size H×W×3 is output.

7. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 1, characterized in that: The feature fusion module processing process includes: first, the feature enhancement maps output by all feature enhancement modules are connected through the Concatenate layer, and the output is then passed through a Conv layer. The convolution kernel size of the Conv layer is 1×1, the step size is 1, and the activation function is ReLU. The output image of size H×W×3 is the final low-light enhancement map.

8. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 1, characterized in that: Loss function of low light enhancement network model total The expression is: Loss total =L MAE +L SSIM +L perc +L region +L color in, is the mean absolute error loss at the pixel level, N is the number of images, P is the total number of pixels in an image, the subscript p represents a pixel in the image, and y represents the original normal illumination image. A low-light enhancement map representing the output; is the structural similarity loss, μ y 、 Represents image y, The average pixel value, Represents image y, The variance of Represents image y and The covariance between them, C1 and C2 are two constants to avoid the denominator being 0 in the formula; is the regional loss function, where the subscripts l and h represent the low-light area and other areas in the image respectively, and w l and w h Represents the coefficients of low-light areas and other areas, i and j are the horizontal and vertical coordinates of the pixel points on the image, m and n are the horizontal and vertical pixel values ​​of the area, |·| is the absolute value operation, y l (i, j) represents the pixel intensity of the i-th row and j-th column in the low-light area of ​​the normal-light image, represents the pixel intensity of the i-th row and j-th column in the low-light area of ​​the low-light enhancement image, y h (i, j) represents the pixel intensity of the i-th row and j-th column in other areas of the normal illumination image, Represents the pixel intensity of the i-th row and j-th column in other areas of the low-light enhancement image; is the perceptual loss, using the VGG-19 network as the feature map extractor, h i,j 、w i,j 、c i,j Represents the length, width, and channel dimensions of the feature map, φ i,j Indicates that the jth convolutional layer of the i-th block of the VGG-19 network is used to extract the feature map; is the color loss, where G() represents Gaussian filtering.

9. The low-light image enhancement method based on multi-level feature extraction and fusion according to claim 8, characterized in that: When calculating the regional loss function, the method for determining the low-light areas and other areas in the image is: First, the R, G, and B channels of the color image are combined in proportion to form a single-channel image Img = r·R+g·G+b·B, where r, g, and b are the proportional coefficients of each channel; then all pixel intensity values ​​in the single-channel image Img are sorted from small to large, and the pixel areas with the first a% of the smallest intensity values ​​are defined as low-light areas, and the remaining pixel areas are other areas.

10. A low-light image enhancement device based on multi-level feature extraction and fusion, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 9.

Images