Context-aware lightweight low-light image enhancement method based on feature fusion

Abstract

The application discloses a context-aware lightweight low-light image enhancement method based on feature fusion, and specifically comprises the following steps: step 1, collecting low-light images, constructing a low-light image dataset, pre-processing the low-light image dataset to obtain an input image dataset; step 2, constructing a context-aware lightweight low-light image enhancement model based on contrast feature fusion; step 3, training the model constructed in step 2 by using the input image dataset I obtained in step 1, and outputting the trained context-aware lightweight low-light image enhancement model based on contrast feature fusion; and step 4, inputting a low-light image to be enhanced into the model trained in step 3, and finally obtaining an enhanced normal-light image. The application overcomes the problems of overexposure and color distortion in the low-light image enhancement task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to a context-aware lightweight low-light image enhancement method based on feature fusion. BACKGROUND

[0002] Low-light images refer to images collected by imaging equipment in environments with poor light conditions such as night, overcast days, etc. There are usually serious image degradation problems such as low visibility, low signal-to-noise ratio, color distortion, etc., which seriously affect subsequent visual analysis and understanding. Therefore, it is of great significance to use image processing technology to enhance images collected in low-light environments and improve low-light image quality, and it has become a research hotspot in the field of image processing and computer vision in recent years.

[0003] Traditional methods often enhance image contrast by stretching image gray dynamic range, although they can enhance the visibility of low-light images to a certain extent, but due to the lack of consideration of the imaging mechanism of the image or the limitation of the image priori, the enhancement result has problems such as over-enhancement or under-enhancement. Low-light image enhancement methods based on deep learning have better effects in improving image quality, but due to the limitation of convolution scale, the receptive field is relatively small, and the dependence relationship between pixels in the image scene is weak, which leads to poor enhancement effect for low-light images with complex light conditions. It has been proved that multi-scale image processing methods have a certain effect on processing low-light images with complex light conditions, but most of the existing multi-scale image processing methods ignore the feature interaction and influence between different scales, and do not fully utilize the information extracted by each branch in the processing process. Many existing methods also increase the depth of the network to improve the performance of the convolutional neural network, but the method model is complex, the parameter quantity is large, and it poses a great challenge to computing resources, making it difficult to deploy and apply in actual visual devices. SUMMARY

[0004] The purpose of the present application is to provide a context-aware lightweight low-light image enhancement method based on feature fusion, which overcomes the problems of overexposure and color distortion in low-light image enhancement tasks.

[0005] The technical solution adopted by the present application is a context-aware lightweight low-light image enhancement method based on feature fusion, which specifically includes the following steps:

[0006] Step 1, collect low-light images, construct a low-light image dataset, and preprocess the low-light image dataset to obtain an input image dataset

[0007] Step 2, construct a context-aware lightweight low-light image enhancement model based on contrast feature fusion

[0008] Step 3, training the model constructed in step 2 by using the input image data set I obtained in step 1, and outputting a trained contrast feature fusion-based context-aware lightweight low-light image enhancement model;

[0009] Step 4, inputting a low-light image to be enhanced into the model trained in step 3, and finally obtaining an enhanced normal-light image.

[0010] The application has the characteristics that:

[0011] In step 1, the pre-processing manner of the low-light image data set is: randomly rotating and randomly cropping the low-light image.

[0012] The specific process of step 2 is:

[0013] Step 2.1, inputting the input image data set I obtained in step 1 into the model constructed in step 2, and outputting a feature map x1. Expanding the number of channels to obtain a feature map As shown in the following formula (1); then, the size of the feature map x1 is halved and the number of channels is expanded to twice the original number, to obtain a feature map x2. As shown in the following formula (2):

[0014] x1=Conv1(I) (1);

[0015] x2=Conv2(x1) (2);

[0016] Wherein, conv1 represents a convolution with a kernel size of 3x3 and a step of 1, and conv2 represents a convolution with a kernel size of 3x3 and a step of 2.

[0017] Step 2.2, sending the feature map x1 into a context-aware control module CAM, the context-aware control module CAM including a context-aware block and a gating unit, the feature map x1 passing through the context-aware block, and the size of the feature map x1 remaining unchanged after passing through the context-aware block, still being The process of the feature map x1 passing through the context-aware block is shown in the following formulas (3)-(5):

[0018] CAB(x)=x+CA(LN(x)) (3);

[0019] CA(x)=W3(σ(W2(CM(x)))) (4);

[0020] CM(x)=G(soft(W1(x)*xr1)) (5);

[0021] Wherein, The input feature information representing the context awareness module, LN(·) is layerNorm, + represents residual connection, CA(·) represents the processing process of global context awareness, W2 and W3 represent two 1x1 step convolution respectively, σ is the PRule activation function, and CM(·) is the process of global transformation;

[0022] Step 2.3, the feature map after the context awareness block is sent into the gating unit again, and the feature map The process of the gating unit is shown in the following formula (6):

[0023]

[0024] wherein LN(·) is layerNorm, respectively represent two 1x1 step convolution kernels, respectively represent two 3x3 depth separable convolutions, and σ2 is a GELU activation function.

[0025] Step 2.4, the feature map after the processing in step 2.3 is continuously sent into the next context awareness control module CAM, and after passing through four context awareness control modules CAM, it is sent into the Mix module. Through the Mix module, a learnable parameter matrix is defined to multiply the feature map after passing through the four context awareness control modules CAM, and then added with x1 to obtain a feature map with global context information

[0026] Step 2.5, the output of step 2.4 is sent into the semi-instance normalization module for down-sampling. The down-sampling process is shown in the following formula (7), and the output feature map is obtained. d Then, the semi-instance normalization is performed, and the process is shown in the following formula (8):

[0027] x d = DOWN(W1(x)) (7);

[0028] f2 = δ(cat(IN(W2(x d )), W2(x d ))) (8); Step 2.6, the feature map output from step 2.5 is multiplied with the feature map point by point, and the point multiplication result is taken as the input feature and then sent into four consecutive context awareness control modules CAM in sequence, i.e., steps 2.2-2.4 are repeatedly executed, and the obtained feature map is denoted as will be The feature map is obtained by upsampling through PixelUnshuffle

[0029] Step 2.7, will be and into the contrast feature fusion module CFFM to obtain a fused feature map

[0030] Step 2.8, the feature map will be obtained by convolution through a convolution kernel The input image will be added through residual connection and to obtain the final network output result, that is, a context-aware lightweight low-light image enhancement model based on contrast feature fusion.

[0031] In the training process of step 3, the total loss function used is:

[0032] L total = L char + L ssim + aL fft + bL color (9);

[0033] Where a and b represent the weights of the loss terms, L char , L ssim , L fft , L color represent different loss functions.

[0034] In step 3, the definition of the loss function L char is shown in the following formula (10):

[0035]

[0036] Where I * and I represent the GT and the output image of the network respectively, and e represents a constant;

[0037] The definition of the loss function L ssim is shown in the following formula (11):

[0038] L ssim = 1-SSIM(I * -I) (11);

[0039] The definition of the loss function L fft is shown in the following formula (12):

[0040] L fft =||F(I *) - F(I) || (12) ;

[0041] Loss function L color is defined as shown in equation (13) as follows:

[0042]

[0043] The beneficial effects of the present application are that the present application introduces contextual information into the modeling of non-local dependence of image pixels, constructs a context-aware control module to make up for the deficiency of the limited receptive field of convolution, introduces a Gaussian function to re-establish the inter-channel illumination distribution mapping to avoid overexposure of the enhanced image, constructs a down-sampling module to aggregate the feature maps extracted on one branch and then performs semi-instance normalization processing, aiming to adaptively retain the contextual information while improving the expression ability of the pixels within the channel to weaken the poor performance problem of the lightweight network due to the shallow network depth, and finally constructs a contrast feature fusion module to adaptively adjust the color and brightness of the image. The verification on the public dataset shows that the method provided by the present application only needs a small number of parameters, has high indicators and excellent performance. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 Fig. 1 is a flowchart of the context-aware lightweight low-light image enhancement method based on feature fusion of the present application;

[0045] Figure 2 Fig. 2 is a network model framework diagram of the context-aware lightweight low-light image enhancement method based on feature fusion of the present application;

[0046] Fig. 3(a) and Fig. 3(b) are the context-aware control module of the context-aware lightweight low-light image enhancement method based on feature fusion of the present application;

[0047] Figure 4 Fig. 4 is a semi-instance normalization down-sampling module of the context-aware lightweight low-light image enhancement method based on feature fusion of the present application;

[0048] Figure 5 Fig. 5 is a contrast feature fusion module of the context-aware lightweight low-light image enhancement method based on feature fusion of the present application;

[0049] Fig. 6(a) and Fig. 6(b) are comparison diagrams of the enhancement results of the context-aware lightweight low-light image enhancement method based on feature fusion of the present application. DETAILED DESCRIPTION

[0050] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0051] The context-aware lightweight low-light image enhancement method based on feature fusion of the present application is shown in the flowchartFigure 1 As shown, specifically comprising the following steps:

[0052] Step 1, the weak light image (low light image) data set (LOL data set) is preprocessed by data enhancement methods such as random rotation and random cropping, and then the size of the image is cropped to 256x256 to obtain the preprocessed input image data set

[0053] Step 2, construct a network model, that is, a context-aware lightweight weak light image enhancement model based on contrast feature fusion, first construct a backbone network, such as Figure 2 As shown, the backbone network includes a basic convolutional layer, a context-aware control module CAM composed of a context-aware fast and gating unit, a half-instance normalization down-sampling module HID, and a contrast feature fusion module CFFM.

[0054] Step 2 is specifically implemented as follows:

[0055] Step 2.1, input image Expand the number of channels to get the feature map Then the feature map x1 is obtained by a convolution kernel size of 3x3 and a step of 2 to get a feature map with a size of half and a channel number expanded to twice the original. As shown in equations (1) and (2):

[0056] x1=Conv1(I) (1);

[0057] x2=Conv2(x1) (2);

[0058] Where, Conv1 represents a convolution kernel size of 3x3 and a step of 1, Conv2 represents a convolution kernel size of 3x3 and a step of 2, I is the initial input image, x1, x2 are the feature maps after two channel expansions respectively.

[0059] Step 2.2: First, send x1 into the context-aware control module CAM. As shown in Figures 3(a) and (b), the context-aware control module CAM is composed of a context-aware block and a gating unit. Figure 3(a) is a context-aware block, which can be expressed as formula (3) through the context-aware block process:

[0060] CAB(x)=x+CA(LN(x)) (3)

[0061] CA(x)=W3(σ(W2(CM(x)))) (4)

[0062] CM(x) = G(soft(W1(x) * xr1)) (5)

[0063] wherein, represents the input feature information of the context-aware module, LN(·) is layerNorm, + represents residual connection. CA(·) represents the processing procedure of global context awareness, which can be expressed as formula (4), W2 and W3 represent two 1x1 convolutional kernels respectively, σ is a PRule activation function, CM(·) is a process of global transformation, which is defined as formula (5), wherein, is the dimension transformation result of the input feature W1 represents a 1x1 convolutional kernel with 1 number, and the feature map soft(·) is a softmax operation. * is matrix multiplication, and the feature maps of the two branches are multiplied to obtain a global feature map so that all information in the spatial domain of each channel is aggregated in one pixel unit. G represents a Gaussian function used to fit the normal distribution of light information, and the weight of the bright channel is reduced to avoid exposure. The size of the feature map x1 does not change after the context-aware block, and is still

[0064] Step 2.3: The feature map after the context-aware block is sent to the gating unit again, as shown in Figure 3(b).

[0065]

[0066] wherein, LN(·) is layerNorm, respectively represent two 1x1 convolutional kernels with a step size of 1, respectively represent two 3x3 depth separable convolutions, which can compress the model parameter amount without reducing the convolution kernel receptive field, so as to improve the calculation efficiency. · is dot multiplication, and σ2 is a GELU activation function. After the gating unit, the size and channel number of x1 remain unchanged.

[0067] Step 2.4: The feature maps after step 2.2 and step 2.3 are continuously sent to the next context control module, that is, steps 2.2 and 2.3 are repeated four times to obtain the feature map The x1 in step 2.1 and f0 are sent to the Mix module, and the Mix module represents a skip connection with a parameter matrix, that is, a learnable parameter matrix is defined to multiply the feature map after the four context control modules, and then added to x1 to obtain a feature map with global context information The process is shown in equation (7),

[0068] f1 = Mix(x1, f0) = θ * x1 + (1 - θ) f0 (7);

[0069] where θ is a random initialized and learnable parameter matrix during the training process.

[0070] Step 2.5, put into the semi-instance normalization module, as shown in Figure 4 , first do a down-sampling, the process is shown in equation (8),

[0071] x d = DOWN(W1(x)) (8);

[0072] where W1 represents a convolution kernel with a size of 1x1 and a step of 1, DOWN(·) represents PixelUnshuffle, which is used to reduce the feature map size to half of the original size and expand the channel number to twice the original size, and the output feature map

[0073] Then x d is subjected to semi-instance normalization, the process is shown in equation (9), W2 is a 3x3 convolution with a step of 1, IN(·) represents instance normalization, cat represents concat, and δ represents the Sigmoid activation function. This module only normalizes on half of the channels, and is spliced back to the original size at the end. The output feature from the semi-instance normalization module is denoted as

[0074] f2 = δ(cat(IN(W2(x d )), W2(x d ))) (9);

[0075] Step 2.6, multiply the feature map output by equation (9) with the feature map obtained by convolution down-sampling, and take the result as the input feature to send into four consecutive context feature control modules, that is, repeat steps 2.2, 2.3 and 2.4, and the obtained feature map is denoted as

[0076] Step 2.7, put and into the contrast feature fusion module CFFM, as shown in Figure 5 ​​As shown, first, the two feature maps f1 and f4 are element-level added to obtain a new feature map, which is respectively subjected to two operations, one is the global pooling operation GAP, and the other is the calculation of standard deviation Contarst. The two results obtained after the two operations are element-level added to synthesize a feature map, so that it is compressed through a convolution kernel with a size of 1x1 and a step of 1, and then a PRelu activation function is used to obtain the channel aggregation contrast weight ω. The weight is sent into two parallel branches, and the convolution kernel size of the two branches is 1x1 and the step is 1. Finally, the two weights α and β obtained on the two branches are respectively multiplied with the original input features f1 and f4, and then added to obtain the self-calibration contrast-enhanced fusion feature map

[0077] Step 2.8, finally Through a convolution kernel with a size of 3x3 and a step of 1, the The input image is added through a residual connection to obtain the final network output result.

[0078] Step 3: The weak light image data set processed by step 1 is used to train the network model to obtain a trained feature fusion context-aware lightweight weak light image enhancement model. Specifically, the loss function is used to constrain the result when training the network, and then the parameters are updated through back propagation. Through 200 rounds of training network convergence, 1 round here refers to that the training data set passes through the convolutional neural network once and returns the training result. The total loss function is formula (10):

[0079] L total =L char +L ssim +αL fft +βL color (10);

[0080] In formula (10), α and β represent the weights of the loss terms, L char , L ssim , L fft , L color are Charbonnier loss, SSIM loss, Fast Fourier Transformation loss and Color loss respectively.

[0081] L char is Charbonnier loss, which can also be derived when Y is close to 0. The definition of the loss function is formula (12)

[0082]

[0083] In formula (11), I * and I represent the output image of GT and network respectively, and ε represents a constant, which is set to 10 -3 .

[0084] L ssim is the SSIM loss, which is defined as formula (12):

[0085] L ssim = 1-SSIM(I * -I) (12);

[0086] wherein SSIM(·) refers to structural similarity calculation, which is defined as formula (13):

[0087]

[0088] wherein μ x and μ y represent the mean of input image x and input image y respectively, σ x and σ y represent the variance of input image x and input image y respectively, σ xy represents the mean square error between image x and image y, and c1 and c2 are two hyperparameters used to avoid division by zero.

[0089] L fft is the Fast Fourier Transformation loss, which is based on the constraint of frequency domain reconstruction by Fast Fourier Transformation and has fine local detail analysis ability. It is defined as formula (14):

[0090] L fft = ||F(I * )-F(I)|| (14);

[0091] Herein F(·) refers to Fast Fourier Transformation, which can convert the image in spatial domain into signal in frequency domain, and then constrain the L1 distance between them to help realize the recovery of edge texture.

[0092] L color is the Color loss, which constrains the learning of image color by the cosine similarity between the output image and the reference standard image, so as to realize the correction of image color. The definition of the loss function is formula (15):

[0093]

[0094] In formula (15), H and W represent the height and width of the input image of the network respectively, i is the image pixel position, · represents the inner product operation of the two preceding terms, and k represents the calculation on the R, G, and B channels respectively.

[0095] Step 4: input the weak light image to be enhanced into the trained model, and finally obtain the enhanced normal light image. The comparison results are shown in FIG. 6(a) and FIG. 6(b). FIG. 6(a) represents the weak light image input into the network, and FIG. 6(b) represents the enhancement result of the network.

Claims

1. A context-aware light enhancement method based on feature fusion, characterized in that: Specifically comprising the following steps: Step 1, collect weak light images, construct a weak light image dataset, and pre-process the weak light image dataset to obtain an input image dataset ; in step 1, the pre-processing manner of the weak light image dataset is random rotation and random cropping of the weak light images; Step 2, constructing a context-aware lightweight low-light image enhancement model based on contrast feature fusion; The specific process of step 2 is: Step 2.1, the input image data set obtained in Step 1 is input into the first convolutional layer of the first convolutional neural network Expanding the number of channels to obtain a feature map As shown in the following formula (1); then the feature map x1 size is halved and the number of channels is expanded to twice the original, to obtain a feature map As shown in the following formula (2): （1） （2） Wherein, conv1 represents a convolution kernel size of 3x3 and a step of 1, and conv2 represents a convolution kernel size of 3x3 and a step of 2; Step 2.2, the feature map x1 is sent into a context-aware control module CAM, the context-aware control module CAM includes a context-aware block and a gating unit, the feature map x1 passes through the context-aware block, and the size of the feature map x1 is unchanged after passing through the context-aware block, and the feature map x1 is still The process of the feature map x1 passing through the context-aware block is shown in the following formulas (3)-(5): （3） （4） （5） wherein, represents input feature information of the context awareness module, is layerNorm, represents a residual connection, represents a processing procedure of global context awareness, respectively represent two steps of 1x1 convolution, is an activation function, is a procedure of global transformation; Step 2.

3. Feature maps after the context-aware block The feature maps are sent to the re-entrant gate unit The process of the gate unit is shown in the following equation (6): （6） wherein, is layerNorm, represent two convolutions with kernel size 1 x 1 and stride 1, represent two 3 x 3 depthwise separable convolutions, is a GELU activation function; Step 2.

4. The feature map after step 2.3 is sent to the next CAM After passing through four CAMs, the feature map is sent to a Mix module, which defines a learnable parameter matrix to multiply with the feature map after passing through the four CAMs, and then adds the result to the feature map with global context information ; Step 2.5, first process the output of step 2.

4. The sample is fed into the semi-instance normalization module for downsampling. The downsampling process is shown in formula (7), and the feature map is output. Then Then perform semi-instance normalization, as shown in formula (8): （7） （8） Step 2.

6. The feature map output from step 2.5 with the feature map Point multiplication is performed, and the point multiplication result is sequentially input as an input feature into four consecutive context-aware control modules CAM, i.e., steps 2.2-2.4 are repeatedly executed, and the obtained feature map is denoted as , The feature map is obtained by PixelUnshuffle up-sampling ; Step 2.7, send into the contrast feature fusion module CFFM, obtain the fusion feature map and into the contrast feature fusion module CFFM, obtain the fusion feature map ; Step 2.8, the feature map is obtained by convolution of a convolution kernel , the input image is added through a residual connection , and the final network output result is obtained, that is, a context-aware lightweight low-light image enhancement model based on contrast feature fusion is obtained; Step 3, using the input image data set I obtained in step 1 to train the model constructed in step 2, and outputting a trained context-aware lightweight low-light image enhancement model based on contrast feature fusion; In the training process of step 3, the total loss function used is: （9） where a and b represent the weights of the loss terms, respectively, , , , denote different loss functions, respectively. Step 4, input the weak light image to be enhanced into the model trained in step 3, and finally obtain the enhanced normal light image; in step 3, the loss function is defined as shown in the following formula (10): （10） wherein, and respectively represent and the output image of the network, represents a constant; Loss function is defined as shown in Equation (11) below: （11） Loss function is defined as shown in Equation (12) below: （12） Loss function is defined as shown in Equation (13) below. （13）。

Images