Low-illumination image enhancement method based on central difference convolution

Abstract

The invention relates to a low-illumination image enhancement method based on central difference convolution, and belongs to the field of image processing. The method comprises the following steps: firstly, dividing a brightness component of an image in an HSV color space into a bright region and a dark region through a binary mask, and calculating the average brightness of each region; performing exposure of different degrees on the brightness component of the image according to the average brightness of different areas through an exposure method based on a binary mask; fusing the brightness components with different exposure degrees by the central difference convolutional neural network under the guidance of the fusion loss function; and finally, the amplified noise is suppressed and texture details in the image are enhanced under the guidance of the content loss function through the other central difference convolutional neural network. According to the method, noise can be suppressed, texture details can be enhanced, and the image definition can be improved; moreover, the method has a better enhancement effect on images under a non-uniform illumination condition and an extremely low illumination condition.

Description

Technical Field

[0001] The invention belongs to the field of image processing technology, in particular to the field of image enhancement methods, and relates to a low-illumination image enhancement method based on center difference convolution. Background Art

[0002] During imaging, the acquired images may show problems such as blurred details, low brightness, and unclear colors due to insufficient ambient light or inappropriate camera parameter settings. Although it is possible to obtain brighter images by adjusting imaging device parameters, such as extending exposure time and increasing aperture, this requires manual adjustment according to environmental changes and will also introduce other shooting problems, such as motion blur. Compared with hardware adjustments, low-light image enhancement technology can easily perform post-enhancement on images to make the image clearer. In addition to improving the look and feel of images, low-light image enhancement technology can also improve the performance of advanced computer vision tasks. Summary of the invention

[0003] The purpose of the present invention is to provide a low-illumination image enhancement method based on center difference convolution, which can suppress noise, enhance texture details, and improve image clarity; and has a good enhancement effect on images under non-uniform illumination conditions and extremely low illumination conditions.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A low-light image enhancement method based on center difference convolution, firstly, the brightness component of the low-light image is extracted using the HSV color space and the brightness component is divided into two bright and dark regions through a binary mask and the average brightness of each region is obtained; then, the brightness component of the image is exposed to different degrees according to the average brightness of different regions through an exposure method based on a binary mask; then, the brightness components after different degrees of exposure are fused under the guidance of a fusion loss function by a center difference convolutional neural network; finally, another center difference convolutional neural network is used to suppress the amplified noise and enhance the texture details in the image under the guidance of a content loss function;

[0006] The method specifically comprises the following steps:

[0007] S1: Extract the brightness component V of the low-light image using the HSV color space low ;

[0008] S2: Create a binary mask that divides the brightness component into bright and dark areas;

[0009] S3: Obtain brightness components of different exposure levels through a binary mask-based exposure method and

[0010] S4: Construct a central difference convolutional neural network;

[0011] S5: Luminance component and The brightness component V is obtained by fusion through the central difference convolutional neural network under the guidance of the fusion loss function. fused ;

[0012] S6: The brightness component V fused Then convert it back to RGB color space through HSV inverse transformation to get the exposed image I exp ;

[0013] S7: Image after exposure I exp The final enhancement result is obtained by suppressing noise through the central difference convolutional neural network under the guidance of the content loss function.

[0014] Preferably, the step S3 specifically includes:

[0015] S31: Segment the image into a bright area and a dark area through a binary mask;

[0016] S32: Calculate the parameters of the gamma transformation function according to the average brightness of the bright area, and transform the brightness component V low Expose to get the brightness component of the bright area exposure

[0017] S33: As with S2, the gamma transformation function exposes the brightness component according to the average brightness of the dark area to obtain the brightness component of the dark area exposure

[0018] S34: Construct a two-parameter piecewise linear transformation function f(r; α, β), which is expressed as follows:

[0019]

[0020] Among them, r is the input brightness value, s is the output brightness value, both range is [0,1], α determines the slope of the transformation function in the low brightness value interval, and β determines the slope of the transformation function in the high brightness value interval;

[0021] S35: Calculate the α parameter in f(r; α, β) by the average brightness of the dark area;

[0022] S36: Calculate the β parameter by solving the following optimization objective:

[0023]

[0024] Among them, h and w are the height and width of the image respectively, i represents the image position index, and b is the average global brightness of the target;

[0025] S37: Expose the brightness component according to the determined parameters f(r; α, β) to obtain the overall exposure brightness component

[0026] Preferably, the step S4 specifically includes:

[0027] S41: First, use 3×3 size convolution to extract coding features;

[0028] S42: The features are then further encoded through a residual block with a center difference convolution module and a multi-scale feature attention module;

[0029] S43: Then use the downsampling operation to obtain a smaller scale encoding feature;

[0030] S44: Accordingly, the scale of the encoded feature is restored by an upsampling operation to obtain a decoded feature;

[0031] S45: Combine the encoding features and decoding features of the same scale through the concatenation operation, decode them again using the residual block, and fuse the two features using a 1×1 convolution;

[0032] S46: Finally, a 3×3 convolution is used to convert the decoded features into an image.

[0033] Furthermore, the central difference convolution module in step S42 in step S4 specifically includes:

[0034] S421: First, input the input feature into the channel attention mechanism to obtain a weight of 2×1×1;

[0035] S422: Then the weights are split to obtain the fusion weights of the center difference convolution kernel and the common convolution kernel respectively;

[0036] S423: Finally, the center difference convolution kernel and the common convolution kernel are weighted and summed according to the fusion weight to obtain the convolution kernel K that is dynamically adjusted with the input features;

[0037] S424: Use the convolution kernel K to perform a convolution operation on the input features to further extract valid information from the input features.

[0038] Furthermore, the multi-scale feature attention module in step S42 in step S4 specifically includes:

[0039] S42a: First, input the feature input channel attention mechanism to obtain the channel weight W of size C×1×1 c ;

[0040] S42b: Set the channel weight W c The rough weight W is obtained by adding it to the input features through the broadcast mechanism coa ;

[0041] S42c: Use group convolution with 3×3 small receptive field and 7×7 large receptive field for the coarse weight W coa Convolution is performed to obtain weights W3 and W7 of two different scales;

[0042] S42d: weight W according to the corresponding channel coa , W3 and W7 are recombined in an alternating manner through channel shuffle operations and the three weights are fused with 1×1 group convolution to obtain the refined weight W;

[0043] S42e: Attention weights the input feature map by refining the weights W.

[0044] The beneficial effects of the present invention are as follows: in terms of the network framework, a local area in the brightness component of an image is fully exposed through an exposure method based on a binary mask and brightness components of different exposure levels are fused through a two-stage network, thereby suppressing noise amplified by brightness enhancement and enhancing texture details in the image. The framework can mine detail information in the image and is robust to low-illumination images under different lighting conditions; in terms of the network module, a central differential convolution module is designed to extract local gradient-level detailed information and intensity-level semantic information, and the feature attention module is improved so that the neural network can pay attention to the differences between each feature channel. The two modules enhance the neural network's ability to extract image features, which is conducive to improving the performance of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0046] Figure 1 It is a flow chart of the low illumination image enhancement method based on center difference convolution described in the present invention;

[0047] Figure 2 Constructing an example graph for the center difference convolutional neural network described in the present invention;

[0048] Figure 3 (a) is an example diagram of residual block construction according to the present invention;

[0049] Figure 3 (b) An example diagram for constructing the multi-scale feature attention module of the present invention;

[0050] Figure 4 An example diagram is constructed for the central difference convolution module described in the present invention. DETAILED DESCRIPTION

[0051] The specific contents of the present invention are further explained in detail below in conjunction with embodiments.

[0052] This embodiment provides a low-light image enhancement method based on center difference convolution, which uses center difference convolution to extract gradient-level detailed features for low-light image detail enhancement. First, the V channel in the HSV color space is used as the brightness component of the low-light image and the brightness component is divided into two bright and dark areas through a binary mask and the average brightness of each area is obtained; then, the brightness component of the image is exposed to different degrees according to the average brightness of different areas through an exposure method based on a binary mask; then, the center difference convolutional neural network fuses the brightness components after different degrees of exposure under the guidance of a fusion loss function; finally, after recombining the H, S and V channels, another center difference convolutional neural network is used to suppress the amplified noise and enhance the texture details in the image under the guidance of a content loss function to obtain the final enhancement result;

[0053] like Figure 1 As shown, the low-illumination image enhancement method based on center difference convolution described in the present invention specifically includes the following steps:

[0054] S1: Use the V channel in the HSV color space as the brightness component V of the image low , and preserve the hue and saturation components;

[0055] S2: Make the brightness component V low The binary mask is segmented into bright and dark areas. The specific steps include:

[0056] S21: V low Multiply by 255 and round down;

[0057] S22: Statistics V low In the gray value range of 0-255, the number of pixels belonging to each gray value obtains a histogram sequence;

[0058] S23: using a mean filter with a window size of 3 and a moving step size of 1 on the histogram sequence to avoid the influence of the non-smooth probability density function curve on the estimation of the segmentation threshold;

[0059] S24: Use the following convolution kernel W1 with a moving step size of 1 to extract the first-order gradient sequence of the probability density function on the histogram sequence:

[0060] W1=[-1,1]

[0061] S25: Extract the indexes whose element value is 0 and whose previous index value is positive and whose previous index value is negative, and whose element value is negative and whose previous index value is positive, in the first-order gradient sequence, to form a maximum index sequence. If there is at most one element in the maximum index sequence, select the index value corresponding to the maximum element value in the histogram sequence as the brightness component V low The segmentation threshold is set, and the process jumps to step S26, otherwise the process goes to step S25;

[0062] S26: Select the two indexes with the largest corresponding element values ​​in the maximum value index sequence and search for the index of the minimum element value in the range between the two indexes in the histogram sequence. If there is only one index that meets the minimum element value condition, then select the index as the segmentation threshold. Otherwise, select the minimum index that meets the minimum element value condition as the segmentation threshold.

[0063] S27: traverse the brightness component V low For each pixel in , if the pixel value is greater than the segmentation threshold, the point is set to 0, otherwise it is set to 1, and finally a binary mask Mask is obtained;

[0064] S3: Obtain brightness components V of different exposure levels through a binary mask-based exposure method light , V dark and V global , the specific steps include:

[0065] S31: Divide the dark area V by binary mask Mask dark , which is expressed as follows:

[0066] V dark =V low ⊙Mask,

[0067] Where ⊙ is the pixel-by-pixel multiplication;

[0068] S32: Then divide the bright area V light , which is expressed as follows:

[0069] V light =V low ⊙(1-Mask);

[0070] S33: Calculate the dark area V dark and bright area V light The average brightness X dark and X light ;

[0071] S34: By X dark and X light The parameters of the gamma transform function are calculated by the following equation:

[0072] γ=log 10 a / log 10 X,

[0073] Where a is the target local average brightness value and is set to 0.5. If X is X dark Then we get γ dark , is X light Then we get γ light ;

[0074] S35: By using different parameters γ dark and γ light The gamma transform function is for the brightness component V low Exposure to obtain and Due to γ dark Compared to γ light Small, use γ dark The gamma transform function of the parameters is for the brightness component V low The dark areas of the Compared to There is more shadow detail, but bright areas appear overexposed. Compared to On the contrary, it has more highlight details;

[0075] S36: Although the single-parameter gamma transform can well enhance the local area, it is still not good in terms of overall exposure. Therefore, a piecewise linear transformation function f(r; α, β) with two parameters is constructed, which is expressed as follows:

[0076]

[0077] Among them, r is the input brightness value, s is the output brightness value, both range is [0,1], α determines the slope of the transformation function in the low brightness value interval, and β determines the slope of the transformation function in the high brightness value interval;

[0078] S37: Calculate the α parameter in f(r; α, β) by the average brightness of the dark area, which is expressed as follows:

[0079] α=a / X dark ,

[0080] S38: Calculate the β parameters by solving the following optimization objective:

[0081]

[0082] Where h and w are the height and width of the image respectively, i represents the image position index, and b is the average global brightness of the target, which is set to 0.5;

[0083] S39: Calculate the brightness component V according to the determined parameters f(r; α, β) low Expose to get a good overall exposure of the brightness components

[0084] Furthermore, the specific steps of solving the optimization target in step S38 are:

[0085] S381: First, sample n points at equal intervals on the following transformation function:

[0086] s = α·r,

[0087] In order to reduce the computational burden, the number of sampling points n is set to 10;

[0088] S382: Then the straight line connecting each of the n points with the point (1,1) and the straight line connecting each of the n points with the point (0,0) form a piecewise linear transformation function f(r; α, β), and there are n different piecewise linear transformation functions in total;

[0089] S383: Finally, the global average brightness value Y of the brightness component after exposure through the n transformation functions f(r; α, β) is calculated. If the difference between one of the Y and the target global brightness average value b is the smallest, the β of the corresponding piecewise linear transformation function is the final solution result;

[0090] S4: Figure 2 As shown, a central difference convolutional neural network is constructed, and its specific steps include:

[0091] S41: First, use 3×3 size convolution to extract coding features;

[0092] S42: The features are then further encoded through 4 residual blocks with center difference convolution modules and multi-scale feature attention modules, such as Figure 3 (a)

[0093] S43: Then a 1×1 convolution is used to extract more image features and a nearest neighbor downsampling operation is used to reduce the scale of the image;

[0094] S44: Accordingly, a 1×1 convolution is used to reduce the number of feature channels layer by layer and a nearest neighbor upsampling operation is used to restore the scale of the decoded features;

[0095] S45: Combine the encoding features and decoding features of the same scale through the concatenation operation, use 4 residual blocks for decoding again, and use 1×1 convolution to fuse the two features;

[0096] S46: Finally, a 3×3 convolution is used to convert the decoded features into an image;

[0097] Further, such as Figure 4 As shown, the central difference convolution module in step S42 in step S4 specifically includes:

[0098] S421: Adopt adaptive global average pooling (AvgPool) across spatial dimensions to reduce the dimension of input features to obtain a tensor of size C×1×1;

[0099] S422: The tensor is passed through a fully connected layer consisting of two 1×1 convolutions and a ReLU activation function to reduce the number of channels and normalized using a Softmax function to obtain a 2×1×1 weight.

[0100] S423: Then the weights are split to obtain the fusion weights of the central difference convolution kernel and the common convolution kernel respectively. The fusion weights can determine the contribution of the central difference convolution for extracting gradient-level detailed information and the common convolution for extracting intensity-level semantic information in the feature extraction task.

[0101] S424: Finally, the center difference convolution kernel and the common convolution kernel are weighted and summed according to the fusion weight to obtain the convolution kernel K that is dynamically adjusted with the input features;

[0102] S425: performing a convolution operation on the input features using the convolution kernel K to further extract valid information from the input features;

[0103] Further, such as Figure 3 As shown in (b), the multi-scale feature attention module in step S42 in step S4 specifically includes:

[0104] S42a: First, input the feature input channel attention mechanism to obtain the channel weight W of size C×1×1 c ;

[0105] S42b: Set the channel weight W c The rough weight W is obtained by adding it to the input features through the broadcast mechanism coa ;

[0106] S42c: Use group convolution with 3×3 small receptive field and 7×7 large receptive field for the coarse weight W coa Convolution is performed to obtain weights W3 and W7 of two different scales;

[0107] S42d: weight W according to the corresponding channel coa , W3 and W7 are recombined in an alternating manner through channel shuffle operations and the three weights are fused with 1×1 group convolution to obtain the refined weight W;

[0108] S42e: Attention weighting of input feature map by refining weight W. S5: Brightness component and The brightness component V is obtained by fusion through the central difference convolutional neural network under the guidance of the fusion loss function. fused , the fusion loss function is expressed as follows:

[0109]

[0110] in, Represents the brightness component after exposure, and q represents the brightness components of three different exposure levels. Fusion weight in is the Laplacian edge detection operator, which means that the fusion weight is determined according to the richness of the gradient information contained in each exposure image;

[0111] S6: The brightness component V fused Then convert it back to RGB color space through HSV inverse transformation to get the exposed image I exp ;

[0112] S7: Image after exposure I exp The final enhancement result is obtained by suppressing noise through the central difference convolutional neural network under the guidance of the content loss function. The content loss function specifically includes: MAE loss function Perceptual Loss Function and generate adversarial loss function

[0113] Furthermore, each loss function in step S7 is expressed as follows:

[0114] S71: MAE loss function The pixel value difference between the enhanced image and the reference image can be expressed as:

[0115]

[0116] Among them, I high is the reference image;

[0117] S72: Generative Adversarial Loss The guided enhancement result is visually similar to the normal illumination image and is expressed as:

[0118]

[0119] Among them, Dnet is the trained discriminant network;

[0120] S73: Perceptual Loss Function The enhanced image is close to the normal illumination image from the feature level and conforms to human visual perception, which can be expressed as:

[0121]

[0122] Among them, φ l These are the 3rd, 8th, 15th, and 22nd feature layer outputs in the VGG16 network.

[0123] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A low illumination image enhancement method based on center difference convolution, characterized in that: First, the image is divided into two areas, bright and dark, through a binary mask and the average brightness of each area is calculated. Then, the brightness component of the image is exposed to different degrees according to the average brightness of different areas through an exposure method based on a binary mask. Then, the brightness components of different exposure levels are fused under the guidance of a fusion loss function through a central difference convolutional neural network. Finally, another central difference convolutional neural network is used to suppress the amplified noise and enhance the texture details in the image under the guidance of a content loss function. The method specifically comprises the following steps: S1: Extract the brightness component V of the low-light image using the HSV color space low ; S2: Create a binary mask that divides the brightness component into bright and dark areas; S3: Obtain brightness components of different exposure levels through a binary mask-based exposure method and S4: Construct a central difference convolutional neural network; S5: Luminance component and The brightness component V is obtained by fusion through the central difference convolutional neural network under the guidance of the fusion loss function. fused ; S6: The brightness component V fused The exposed image I is obtained by converting back to RGB color space through HSV inverse transformation exp ; S7: Image after exposure I exp The final result is obtained by suppressing noise and enhancing details under the guidance of the content loss function through the center difference convolutional neural network.

2. The low illumination image enhancement method based on center difference convolution according to claim 1, characterized in that: The step S3 specifically includes: S31: Segment the image into a bright area and a dark area through a binary mask; S32: Calculate the parameters of the gamma transformation function according to the average brightness of the bright area, and transform the brightness component V low Expose to get the brightness component of the bright area exposure S33: As with S32, the gamma transformation function exposes the brightness component according to the average brightness of the dark area to obtain the brightness component of the dark area exposure S34: Construct a two-parameter piecewise linear transformation function f(r; α, β), which is expressed as follows: Among them, r is the input brightness value, s is the output brightness value, both range is [0,1], α determines the slope of the transformation function in the low brightness value interval, and β determines the slope of the transformation function in the high brightness value interval; S35: Calculate the α parameter in f(r; α, β) by the average brightness of the dark area; S36: Calculate the β parameter by solving the following optimization objective: Among them, h and w are the height and width of the image respectively, i represents the image position index, and b is the average global brightness of the target; S37: Expose the brightness component according to the determined parameters f(r; α, β) to obtain the overall exposure brightness component 3. The low illumination image enhancement method based on center difference convolution according to claim 1, characterized in that: The step S4 specifically includes: S41: First, use 3×3 size convolution to extract coding features; S42: The features are then further encoded through a residual block with a center difference convolution module and a multi-scale feature attention module; S43: Then use the downsampling operation to obtain a smaller scale encoding feature; S44: Accordingly, the scale of the encoded feature is restored by an upsampling operation to obtain a decoded feature; S45: Combine the encoding features and decoding features of the same scale through the concatenation operation, decode them again using the residual block, and fuse the two features using a 1×1 convolution; S46: Finally, a 3×3 convolution is used to convert the decoded features into an image.

4. The low illumination image enhancement method based on center difference convolution according to claim 4, characterized in that: The central difference convolution module in step S42 specifically includes: S421: First, input the input feature into the channel attention mechanism to obtain a weight of 2×1×1; S422: Then the weights are split to obtain the fusion weights of the center difference convolution kernel and the common convolution kernel respectively; S423: Finally, the center difference convolution kernel and the common convolution kernel are weighted and summed according to the fusion weight to obtain the convolution kernel K that is dynamically adjusted with the input features; S424: Use the convolution kernel K to perform a convolution operation on the input features to further extract valid information from the input features.

5. The low illumination image enhancement method based on center difference convolution according to claim 4, characterized in that: The multi-scale feature attention module in step S42 specifically includes: S42a: First, input the feature input channel attention mechanism to obtain the channel weight W of size C×1×1 c ; S42b: Set the channel weight W c The rough weight W is obtained by adding it to the input features through the broadcast mechanism coa ; S42c: Use group convolution with 3×3 small receptive field and 7×7 large receptive field for the coarse weight W coa Convolution is performed to obtain weights W3 and W7 of two different scales; S42d: weight W according to the corresponding channel coa , W3 and W7 are recombined in an alternating manner through channel shuffle operations and the three weights are fused with 1×1 group convolution to obtain the refined weight W; S42e: Attention weights the input feature map by refining the weights W.

Images