A de-black patch self-convolution method for dark image enhancement

Abstract

The application provides a de-black-block self-convolution method for dark image enhancement, which comprises the following steps: S1: inputting a feature map obtained from a previous layer into a convolution with a convolution kernel of 1x1 and a channel number of 1 / 2 of an original channel number, so that two part feature maps are obtained; S2: further processing the two part feature maps obtained from step S1, taking one feature map as Out1, performing an average pooling operation, that is, focusing on important information of a potential surrounding area, then performing a sigmoid operation and multiplying with an input image element by element, so that more useful information is globally extracted and redundant information is weakened, and a feature map is taken as Out1_5; S3: via step S1, taking another part feature map obtained to directly pass through a convolution to obtain a feature map taken as out2_1; and S4: finally, cascading the features processed through steps S2 and S3.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of low-illumination image enhancement, in particular to a black block removal self-convolution method for dark image enhancement. BACKGROUND

[0002] Due to a large number of camera devices under insufficient light or shielding conditions, the images captured are often dark, especially the images captured at night, and the target objects are basically invisible. Such images are not conducive to face recognition or detection tasks. In the low-illumination enhancement task, it is difficult to obtain a certain amount of real label images and low-illumination images, especially real labels with good brightness, saturation, and contrast. Most studies obtain the corresponding darkness map by performing r transformation on images with good light, and then train the model. Current low-illumination enhancement methods are mainly based on multi-scale fusion and deep learning methods of image decomposition into reflectance and illumination maps, that is, different scale feature maps are extracted by multi-layer convolution and down-sampling and up-sampling operations, and then added or cascaded to combine, such as the typical unet network and the KIND network based on reflectance and illumination decomposition.

[0003] At present, in the multi-scale fusion network, ordinary convolution fuses information from space and channels and stacks by using a fixed size convolution kernel, which easily causes different features to be learned to be similar, and it is not easy to distinguish the difference between the features. In the deep network, the details are easily lost, which is not conducive to task learning.

[0004] Common terms in the prior art include:

[0005] Multi-scale: indicates that the width and height of the output of the image through different layers of the network are different.

[0006] Convolution: a mathematical operation that represents the multiplication of two variables within a certain range and then sums the results.

[0007] Average pooling operation: represents the sum of the addition of adjacent numbers in a matrix.

[0008] Darkness map: represents that the pixel value on the image is very low, and some objects cannot be seen.

[0009] Reflectance map: represents the image of the object reflecting light.

[0010] Illumination map: the component of the ambient light shining on the object, which determines the range of the dynamic image pixel value. SUMMARY

[0011] In order to solve the problems in the prior art, the present application aims at: the conventional convolution operation often causes black blocks in different regions of an image in a low-illumination image enhancement task, resulting in that some regions of the enhanced image are brightened and some regions are still dark, and the image brightening effect is poor, which causes the dark image enhancement effect to be unable to meet the actual application.

[0012] Specifically, the present application provides a dark image enhancement de-blackening self-convolution method, which comprises the following steps:

[0013] S1: input the feature maps obtained by the last layer of convolution into a convolution with a convolution kernel of 1x1 and a channel number of 1 / 2 of the original channel number, respectively, to obtain two parts of feature maps;

[0014] S2: further process the two parts of feature maps obtained in step S1, and mark one of the obtained feature maps as Out1, first perform an average pooling operation, that is, pay attention to the important information of the surrounding area, the important information includes high-level semantic information of the image and detail information of the target in the image, then perform a sigmoid operation of an activation function and multiply the input image element by element, so that more useful information is globally extracted and redundant information is weakened, and finally obtain a feature map marked as Out1_5;

[0015] S3: via step S1, directly pass the other part of the obtained feature map through a convolution to obtain a feature map marked as out2_1;

[0016] S4: finally, cascade the features processed by steps S2 and S3.

[0017] The step S2 further comprises:

[0018] perform 2x2 average pooling on the feature map of one branch of step S1, the pooling operation slides on the feature map according to a step of 2 through a window of 2x2, and takes the average value of 2x2 pixel values contained in the window as the output, to obtain a feature map marked as Out1_2;

[0019] perform convolution upsampling on the obtained feature map to obtain a feature map marked as Out1_3, to obtain a multi-channel attention map, the channel attention map is an attention map in which an activation function is used to assign a large weight to useful information in the feature map and a small weight to unimportant information, therefore, the attention map is also called a weight map, as shown in formula (1):

[0020] out1_5 = sigmoid(out1_3 + out1_1) ⊗ out1_4 (1)

[0021] In the formula, sigmoid the () function is an activation function, and the output value range is [0, 1] after the input value passes through the function,out 1_1、 out 1_3、 out 1 and 4 represent the outputs of different convolutional layers; where ⊗ indicates that corresponding pairs of feature map elements are multiplied.

[0022] Then, from equation (1) sigmoid The attention map obtained by the () function and out 1_4 Multiply the corresponding elements of the feature maps to obtain the output. out 1_5; in out In each channel feature map of 1_5, because the weight values ​​on the attention map of the corresponding channel are different, the useful information on the channel is highlighted and focused on, while the smaller weight will reduce the value on the same feature map, that is, reduce useless information such as noise. Returning to the original image, we get feature maps with different information on each channel.

[0023] Finally, the results of the two branches are concatenated.

[0024] The calculation of the cascade in step S4 is shown in equation (2):

[0025] final_out = concat [out2_2, out2_1] (2)

[0026] In the formula concat () indicates a cascading operation. out 2_2、 out 2_1 represents the outputs of the two convolutional branches; after cascading, the output feature map is obtained. final_out .

[0027] Therefore, the advantage of this application is:

[0028] (1) Since most of the pixel values ​​in low-light images are small, ordinary convolution will cause some details to be lost in each channel. Therefore, upsampling after pooling is helpful to capture the surrounding environment information. The feature map after this upsampling will focus on more useful information in each channel, preventing the problem of local black blocks that cannot be brightened due to loss of details.

[0029] (This multi-scale information fusion method across multiple channels not only has a good effect on removing local black blocks, but also can further improve image brightness.) Attached Figure Description

[0030] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0031] Figure 1 This is a flowchart of the method of the present invention.

[0032] Figure 2 is a block diagram schematic view of the method of the present application. DETAILED DESCRIPTION

[0033] In order to enable a clearer understanding of the technical content and advantages of the present application, the present application will be further described in detail in conjunction with the accompanying drawings.

[0034] The present application proposes a self-convolution method for removing dark image black blocks, as shown in Figure 1 The main implementation step flow of the method is as follows:

[0035] Step S1: input the feature maps obtained from the previous layer into convolution with the convolution kernel being 1x1 and the channel number being 1 / 2 of the original channel number, then two parts of feature maps are obtained;

[0036] Step S2: further process the two parts of feature maps obtained from step 1, as shown in Figure 1 The obtained feature map out1 is first subjected to an average pooling operation, i.e. focusing on the important information of the surrounding area, which contains high-level semantic information of the image and detailed information of the target in the image, then subjected to a sigmoid operation of an activation function and multiplied element by element with the input image, so that more useful information is globally extracted and redundant information is weakened, and finally the feature map out1_5 is obtained;

[0037] The step S2 further comprises:

[0038] The feature map of one branch of step S1 is subjected to a 2x2 average pooling operation, the pooling operation is performed by a 2x2 window on the feature map in turn with a step of 2, and the mean value of the 2x2 pixel values contained in the window is taken as the output, and the obtained feature map is denoted as Out1_2;

[0039] The feature map obtained by further convolution and up-sampling is denoted as Out1_3, and a multi-channel attention map is obtained, the channel attention map is an attention map obtained by assigning a large weight to useful information in the feature map through an activation function, the useful information includes image contour information and abstract information between different pixels; and unimportant information is assigned a small weight, therefore, the attention map is also called a weight map, the weight value learned by the neural network ranges from 0 to 1, as shown in formula (1):

[0040] out1_5 = sigmoid(out1_3 + out1_1) ⊗ out1_4 (1)

[0041] In the formula, sigmoid The function is an activation function, the output value ranges from 0 to 1 after the input value is input through the function, out 1_1、 out 1_3、out 1 and 4 represent the outputs of different convolutional layers; where ⊗ indicates that corresponding pairs of feature map elements are multiplied.

[0042] Then, from equation (1) sigmoid The attention map obtained by the () function and out 1_4 Multiply the corresponding elements of the feature maps to obtain the output. out 1_5; in out In each channel feature map of 1_5, the weight values ​​on the attention map of the corresponding channel are different, which makes the useful information on the channel stand out. The smaller weight will reduce the value on the same feature map, that is, reduce useless information such as noise and return to the original image. Returning to the original image means multiplying the obtained attention map with the original image to obtain feature maps with different information on each channel.

[0043] Finally, the results of the two branches are concatenated.

[0044] Step S3: From step S1, the other part obtained (such as...) Figure 1 The lower half of the branch shown is directly convolved to obtain out2_1.

[0045] Step S4: Finally, the features processed by the two paths are concatenated, as shown in equation (2):

[0046] final_out = [out1_5, out2_1] (2)

[0047] In the formula concat () indicates a cascading operation. out 2_2、 out 2_1 represents the outputs of the two convolutional branches; after cascading, the output feature map is obtained. final_out out1_5 is processed by a convolution layer to obtain out2_2.

[0048] In summary, the flowchart of this method is shown below. Figure 2 As shown,

[0049] The feature map is divided into two branches by two 1x1 convolution operations and processed separately.

[0050] The feature map of one of the branches is subjected to 2x2 average pooling and then convolutional upsampling to obtain multi-channel attention maps, as shown in formula (1). These attention maps are then returned to the original image to obtain feature maps with different information in each channel. Finally, the results of the two branches are concatenated.

[0051] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A de-black patch self-convolution method for dark image enhancement, characterized in that, The method comprises the following steps: S1: input the feature maps obtained by the previous layer convolution into a convolution with a convolution kernel of 1x1 and a channel number of 1 / 2 of the original channel number, respectively, to obtain two parts of feature maps; S2: further process the two parts of feature maps obtained in step S1, and record one of the obtained feature maps as Out1 to perform an average pooling operation, that is, focus on important information of a potential surrounding area, the important information including high-level semantic information of an image and detailed information of a target in the image, then perform a sigmoid operation of an activation function and multiply the input image element by element, so that more useful information is globally extracted and redundant information is weakened, and finally a feature map is obtained and recorded as Out1_5; The step S2 further comprises: perform 2x2 average pooling on the feature map of one branch of step S1, the pooling operation sliding on the feature map in a step of 2 through a window of 2x2, and taking the average of 2x2 pixel values contained in the window as the output, to obtain a feature map recorded as Out1_2; perform convolution upsampling on the obtained feature map to obtain a feature map recorded as Out1_3, to obtain a multi-channel attention map, the channel attention map being a weight map in which useful information in the feature map is assigned a large weight through an activation function, the useful information including image contour information and abstract information between different pixels; and unimportant information is assigned a small weight, so the attention map is also called a weight map, the weight value learned by the neural network ranging from 0 to 1, as shown in formula (1): out1_5=sigmoid(out1_3+out1_1) ⊗ out1_4 （1） In the formula, sigmoid The () function is an activation function, and the output value range is [0, 1] after the input value passes through the function, out 1_1, out 1_3, out 1_4 are respectively the outputs of different convolution layers; wherein ⊗ represents that the corresponding elements of the feature maps are multiplied. Then the attention map obtained by the function of (1) is multiplied with the original image sigmoid () to obtain the output out 1_4 feature map elements, to obtain the output out 1_5; on each channel feature map on out 1_5, because the weight value on the attention map of the corresponding channel is different, the useful information on the channel is highlighted, and the weight is small. The value on the same feature map is reduced, that is, the useless information is weakened. Return to the original image, return to the original image means that the obtained attention map is multiplied with the original image to obtain a feature map with different information on each channel. Finally, the results of the two branches are concatenated; S3: via step S1, directly perform convolution on the obtained another part of feature maps to obtain a feature map recorded as out2_1; S4: finally, concatenate the features processed by steps S2 and S3.

2. A method for dark image enhancement of de-blocking by self-convolution as claimed in claim 1, wherein, The unimportant information includes noise.

3. The method for dark image enhancement of de-blocking self-convolution as claimed in claim 1 wherein, The concatenation in step S4 is calculated as shown in formula (2): final_out=concat[out2_2,out2_1] （2） In the formula, concat () is a cascade operation, out 2_2, out 2_1 is the output of two convolution branches respectively; after the cascade operation, the output feature map final_out out2_2 is obtained by passing out1_5 through a convolution layer.

Images