A low-illumination image enhancement method based on Residual-Unet network

Abstract

The present invention discloses a low-light image enhancement method based on a Residual-Unet network, which belongs to the technical field of low-light image enhancement. The method comprises the following steps: data preprocessing; obtaining an image to be enhanced and performing histogram equalization preprocessing on the image; constructing a low-light image enhancement network based on Residual-Unet and completing network training; merging the image to be enhanced with the image preprocessed by histogram equalization, inputting the result into the low-light image enhancement network based on Residual-Unet, and outputting the enhanced image from the low-light image enhancement network based on Residual-Unet. Based on the traditional residual network structure, the present invention eliminates all batch normalization operations in the residual network, replaces the second weight layer from a simple convolutional layer and activation layer with the entire Unet convolutional neural network, reduces the influence of illumination changes, enriches the visual information of the image to be enhanced, avoids the influence of irrelevant information on the enhancement result, accelerates the convergence of the model, and simultaneously achieves an enhanced visual effect and significantly improves the peak signal-to-noise ratio.

Description

Technical Field

[0001] The present invention belongs to the technical field of low-illumination image enhancement, and in particular to a low-illumination image enhancement method based on a Residual-Unet network. Background Art

[0002] Images taken under poor lighting conditions are considered low-light images, while images taken under good lighting conditions are considered normally exposed images. Compared with normally exposed images, low-light images mainly exhibit problems such as low contrast, poor readability, color distortion, unclear texture features, and a large amount of noise. Due to the lack of light, the grayscale values of such images are generally small, the pixel values are concentrated, and the image is dark overall. The difference between the grayscale values of edge pixels such as contours and the grayscale values of non-edge neighboring pixels is small, resulting in weakened detail information. These problems not only profoundly affect people's demand for high-quality images in daily production and life, but are also not conducive to practical applications in computer vision fields such as night monitoring, target detection, and face recognition, which have higher requirements for image quality.

[0003] Among traditional low-light image enhancement methods, the most studied are histogram equalization and Retinex-based methods. The former, based on mathematical principles, fails to consider the physical model of the low-light imaging process and denoising issues, resulting in often unsatisfactory results. The latter, requiring elaborate illumination estimation models, has limited generalization capabilities for low-light images across diverse scenarios, resulting in limited application. Deep learning-based methods are primarily divided into supervised pairwise training and unsupervised single-image training. The former utilizes limited datasets but can achieve good results using simple loss functions. The latter, while relatively easy to obtain datasets, places higher demands on the network structure and loss function design. Furthermore, a common problem with deep learning-based methods is that as the network becomes deeper, the image size must be reduced to conserve computational resources. This means that the receptive field decreases as the number of neural network layers increases. Low-light image enhancement methods based on the Residual-Unet network are supervised pairwise training methods that balance the processing depth and receptive field of image illumination enhancement networks, achieving excellent enhancement results. Summary of the Invention

[0004] The purpose of the present invention is to provide a low-light image enhancement method based on the Residual-Unet network to solve the problem that deep neural networks lack the ability to capture global information as they deepen.

[0005] The object of the present invention is achieved as follows: a low-light image enhancement method based on a Residual-Unet network comprises the following steps:

[0006] Step S1: data preprocessing;

[0007] Obtain the image to be enhanced and perform histogram equalization preprocessing on it;

[0008] Step S2: Build a low-light image enhancement network based on Residual-Unet and complete network training;

[0009] The low-light image enhancement network based on Residual-Unet includes: a shallow feature encoding block, a Residual-Unet deep feature processing block, a dual-channel attention mechanism block and an upsampling block;

[0010] Step S3: Merge the image to be enhanced in step S1 with the image preprocessed by histogram equalization, input the image into the low-light image enhancement network based on Residual-Unet in step S2, and output the enhanced image from the low-light image enhancement network based on Residual-Unet.

[0011] Preferably, the shallow feature encoding block includes a convolution layer C1, an activation layer R1, a convolution layer C2 and a maximum pooling layer P1, and the convolution layer C1, the activation layer R1, the convolution layer C2 and the maximum pooling layer P1 are connected in sequence; the Residual-Unet deep feature processing block is composed of four Rs-Unet sub-blocks connected in sequence, and a convolution layer C3 with the same structure is embedded between every two Rs-Unet sub-blocks;

[0012] The dual-channel attention mechanism block is composed of a spatial attention mechanism sub-block and a channel attention mechanism sub-block in parallel, and the upsampling block is composed of an interpolation sampling layer I and a convolutional layer C connected in sequence. 10 Formed with activation layer R4.

[0013] Preferably, the Rs-Unet sub-block includes a first weight layer, a Unet convolutional neural network and a convolutional layer C4, the first weight layer, the Unet convolutional neural network and the convolutional layer C4 are connected in sequence, a residual structure is introduced and the input of the first weight layer and the output of the Unet convolutional neural network are merged as the input of the convolutional layer C4;

[0014] The first weight layer is composed of a convolutional layer C5 and an activation layer R2 connected in sequence, and the Unet convolutional neural network includes a downsampling feature extraction sub-block, a channel expansion sub-block E and an upsampling feature extraction sub-block.

[0015] Preferably, the downsampling feature extraction sub-block includes a downsampling feature extraction sub-block D1, a downsampling feature extraction sub-block D2, a downsampling feature extraction sub-block D3 and a downsampling feature extraction sub-block D4;

[0016] The upsampling feature extraction sub-block includes an upsampling feature extraction sub-block U1, an upsampling feature extraction sub-block U2, an upsampling feature extraction sub-block U3 and an upsampling feature extraction sub-block U4;

[0017] The Unet convolutional neural network consists of four downsampling feature extraction sub-blocks, one channel expansion sub-block E and four upsampling feature extraction sub-blocks connected in sequence.

[0018] Preferably, the downsampling feature extraction sub-block D1 is composed of sequentially connected convolutional layers C D1-1 , activation layer R Unet , convolutional layer C D1-2 , activation layer R Unet With the pooling layer P Unet The downsampling feature extraction sub-block D2 is composed of sequentially connected convolutional layers C D2-1 , activation layer R Unet , convolutional layer C D2-2 , activation layer R Unet With the pooling layer P Unet The downsampling feature extraction sub-block D3 is composed of sequentially connected convolutional layers C D3-1 , activation layer R Unet , convolutional layer C D3-2 , activation layer R Unet With the pooling layer P Unet The downsampling feature extraction sub-block D4 is composed of sequentially connected convolutional layers C D4-1 , activation layer R Unet , convolutional layer C D4-2 , activation layer R Unet With the pooling layer P Unet constitute;

[0019] The channel expansion sub-block E is composed of sequentially connected convolutional layers C E1-1 , activation layer R Unet , convolutional layer C E1-2 With activation layer R Unet constitute;

[0020] The upsampling feature extraction sub-block U1 is composed of the transposed convolution layer T1, the convolution layer C U1-1 , activation layer R Unet , convolutional layer C U1-2 With activation layer R Unet The upsampling feature extraction sub-block U2 is composed of the transposed convolution layer T2, the convolution layer C U2-1 , activation layer R Unet , convolutional layer C U2-2 With activation layer R Unet The upsampling feature extraction sub-block U3 is composed of the transposed convolution layer T3, the convolution layer CU3-1 , activation layer R Unet , convolutional layer C U3-2 With activation layer R Unet The upsampling feature extraction sub-block U4 is composed of the transposed convolution layer T4, the convolution layer C U4-1 , activation layer R Unet , convolutional layer C U4-2 With activation layer R Unet The outputs of the transposed convolutional layers T1, T2, T3, and T4 are respectively connected to the convolutional layers C D4-2 、C D3-2 、C D2-2 、C D1-2 The outputs of the convolutional layer C are combined U1-1 、C U2-1 、C U3-1 and C U4-1 input.

[0021] Preferably, the spatial attention mechanism sub-block is composed of a dual-channel pooling module and a basic convolution module connected in sequence. The dual-channel pooling module is composed of a global average pooling layer P3 and a global maximum pooling layer P4 in parallel, and the outputs of P3 and P4 will be merged as the output of the dual-channel pooling module; the basic convolution module is composed of a convolution layer C6 and an activation layer R M The channel attention mechanism sub-block is composed of the adaptive average pooling layer P2 and the feature dimension transformation module connected in sequence; the feature dimension transformation module is composed of the convolution layer C7, the activation layer R3, the convolution layer C8 and the activation layer R M constitute;

[0022] The spatial attention mechanism and channel attention mechanism will generate a spatial attention map and a channel attention map respectively and act on the original input features respectively before merging them. They are then processed by the convolution layer C9 and pixel-wise summed with the original features, and the pixel-wise sum result is used as the output of the dual-channel attention mechanism block.

[0023] Preferably, the convolution kernel size of the convolution layer C1 is 3×3, the input channel is 6, the output channel is 16, and the padding is 1; the activation layer R1 is LeakyReLU and the negative_slope is 0.2; the convolution kernel size of the convolution layer C2 is 3×3, the input channel is 16, the output channel is 32, and the padding is 1; the convolution kernel size of the maximum pooling layer P1 is 2×2, and the stride is 2; the convolution kernel size of the convolution layer C3 is 3×3, the input channel is 64, the output channel is 32, and the padding is 1; the convolution layer C4 has the same parameters as the convolution layer C3; the convolution kernel size of the convolution layer C5 is 3×3, the input channel is 32, the output channel is 32, and the padding is 1; the activation layer R2 is the ReLU function.

[0024] Preferably, the convolutional layer C D1-1 With the convolutional layer C D1-2 The parameters of the convolution layer are the same: the convolution kernel size is 3×3, the input channel is 32, the output channel is 32, and the padding is 1; the convolution layer C D2-1 The input channel is 32 and the output channel is 64; the convolution layer C D2-2 The input channel is 64 and the output channel is 64; the convolution layer C D3-1 The input channel is 64 and the output channel is 128; the convolution layer C D3-2 The input channel is 128, the output channel is 128; the convolution layer C D4-1 The input channel is 128 and the output channel is 256; the convolution layer C D4-2 The input channel is 256 and the output channel is 256; the convolution layer C E1-1 The input channel is 256 and the output channel is 512; the convolution layer C E1-2 The input channel is 512, the output channel is 512; the convolution layer C U1-1 、C U2-1 、C U3-1 、C U4-1 The input channels and output channels of the convolution layer C E1-1 、C D4-1 、C D3-1 and C D2-1 On the contrary; the convolution layer C U1-2 、C U2-2 、C U3-2 with C U4-2 The input channels and output channels of the convolution layer C D4-2 、C D3-2 、C D2-2 and C D1-2 Same; except for the input channel and output channel, the convolution layer C D1-1 、 1-2 ~C D4-1 、 4-2 , convolutional layer C E1-1 、 1-2 With the convolutional layer C U1-1 、 1-2 ~C U4-1 、 4-2 The parameters are the same: the convolution kernel size is 3×3, the padding is 1; the activation layer R Unet Use LeakyReLU block; pooling layer P UnetThe maximum pooling is used, the convolution kernel size is 2×2, and the stride is 2; the transposed convolution layer T uses the ConvTranspose2d block, the convolution kernel size is 2×2, and the stride is 2. The input channel of T1 is 512 and the output channel is 256, the input channel of T2 is 256 and the output channel is 128, the input channel of T3 is 128 and the output channel is 64, and the input channel of T4 is 64 and the output channel is 32; the global average pooling layer P3 and the global maximum pooling layer P4 are both channel pooling operations, and the average value and the maximum value are calculated along the channel dimension of the feature map respectively; the convolution kernel size of the convolution layer C6 is 5×5, the input channel is 2, the output channel is 1, and the padding is 2; the activation layer R M The Sigmoid block is used; the adaptive average pooling layer P2 uses the AdaptiveAvgPool2d block, and the output_size is 1; the convolution kernel size of the convolution layer C7 is 1×1, the input channel is 32, the output channel is 4, and the padding is 0; the activation layer R3 uses the ReLU block; the convolution kernel size of the convolution layer C8 is 1×1, the input channel is 4, the output channel is 32, and the padding is 0; the convolution kernel size of the convolution layer C9 is 1×1, the input channel is 64, the output channel is 32, and the padding is 0; the interpolation sampling layer I uses the Interpolate block, and the scale_factor is 2; the convolution layer C 10 The convolution kernel size is 1×1, the input channel is 32, the output channel is 3, and the padding is 0; the activation layer R4 uses the Sigmoid block.

[0025] Preferably, the training process of the Residual-Unet low-light image enhancement network is:

[0026] An existing image dataset is used as the training set. During the data loading stage, the low-light images to be enhanced in the training set and the normally exposed images used as labels are randomly cropped, flipped and rotated at the same position to obtain N×N images for network training. The original low-light images are then preprocessed with histogram equalization, and the low-light images and the images obtained by histogram equalization preprocessing are used as input to constitute the training samples. The loss function and training parameters are set, and the supervised batch learning method is used to train the low-light image enhancement network based on Residual-Unet.

[0027] Preferably, the loss function is:

[0028]

[0029] Where L(x,y) represents the loss function, Ref(x,y) represents the label, G(x,y) represents the output image generated by the low-light image enhancement network based on Residual-Unet, (x,y) represents the pixel coordinates, and N is the size of the cropped image.

[0030] Compared with the existing technology, the present invention has the following improvements and advantages: 1. By canceling all batch normalization operations in the Residual network on the basis of the traditional Residual structure, the second weight layer is replaced by a simple convolution and activation layer as a whole with a Unet deep neural network, thereby avoiding the problem of insufficient global information processing during the network deepening process; and performing histogram equalization preprocessing on low-light images to reduce the impact of illumination changes and enrich the visual information of the image to be enhanced, which can help the network to learn better.

[0031] 2. By introducing a dual-channel attention mechanism module to process the fused features, the influence of irrelevant information on the enhancement results is avoided, and the convergence of the model is accelerated; at the same time, the visual effect is enhanced and the peak signal-to-noise ratio (PSNR) is significantly improved, making it widely applicable to low-light image enhancement tasks in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the process of the low-light image enhancement method based on the Residual-Unet network in the present invention.

[0033] Figure 2 Schematic diagram of the structure of the Residual-Unet image enhancement network in the present invention.

[0034] Figure 3 for Figure 2 Schematic diagram of the shallow feature encoding block structure in the Residual-Unet image enhancement network.

[0035] Figure 4 for Figure 2 Schematic diagram of the Residual-Unet deep feature processing block structure in the Residual-Unet image enhancement network shown.

[0036] Figure 5 for Figure 4 Schematic diagram of the Rs-Unet structure in the Residual-Unet deep feature processing block.

[0037] Figure 6 for Figure 5 Schematic diagram of the Unet convolutional neural network structure in the Rs-Unet structure shown.

[0038] Figure 7 for Figure 2 Schematic diagram of the dual-channel attention mechanism block structure in the Residual-Unet image enhancement network.

[0039] Figure 8 for Figure 2 Schematic diagram of the upsampling block structure in the Residual-Unet image enhancement network. DETAILED DESCRIPTION

[0040] The present invention is further summarized below with reference to the accompanying drawings.

[0041] like Figure 1 As shown, a low-light image enhancement method based on the Residual-Unet network specifically includes the following steps:

[0042] Step S1: data preprocessing;

[0043] Acquire the image to be enhanced and perform histogram equalization preprocessing on it. In this embodiment, the acquired RGB image to be enhanced is channel-separated, and histogram equalization processing is performed on the three single-channel grayscale images of R, G, and B respectively, and then the processed channels are re-merged. This step can significantly improve the detail information of the underexposed image;

[0044] Step S2: Build a low-light image enhancement network based on Residual-Unet and complete network training;

[0045] Low-light image enhancement network based on Residual-Unet Figure 2 As shown, it includes: shallow feature encoding block, Residual-Unet deep feature processing block, dual-channel attention mechanism block and upsampling block;

[0046] Shallow feature encoding blocks such as Figure 3 As shown in the figure, it consists of a convolutional layer C1, an activation layer R1, a convolutional layer C2, and a maximum pooling layer P1 connected in sequence. The convolution kernel size of the convolution layer C1 is 3×3, with 6 input channels, 16 output channels, and a padding of 1; the activation layer R1 is LeakyReLU with a negative_slope of 0.2; the convolution kernel size of the convolution layer C2 is 3×3, with 16 input channels, 32 output channels, and a padding of 1; the convolution kernel size of the maximum pooling layer P1 is 2×2, with a stride of 2.

[0047] Residual-Unet deep feature processing block such as Figure 4As shown in the figure, it is composed of 4 Rs-Unet sub-blocks connected in sequence, and a convolution layer C3 with the same structure is embedded between every two Rs-Unet sub-blocks. Among them, the convolution kernel size of the convolution layer C3 is 3×3, the input channel is 64, the output channel is 32, and the padding is 1. The second shallow feature layer S2 After being processed by the Rs-Unet sub-block and then convolved with a 3×3 convolution kernel, it can be transformed into the first deep feature layer Layer d1 In order to enhance the network's ability to perceive features of different scales and levels, the output and input of Rs-Unet are merged into a channel, namely Layer d1 Can be remembered as: Layer d1 =Conv 3×3 (cat(RSU1(Layer S1 ), Layer S1 )). RSU1 is Figure 4 In the Rs-Unet1 sub-block shown, the cat operation is used to merge two feature maps of size C×H×W into a size of 2C×H×W; the same method can be used to obtain the second deep feature layer Layer d2 , the third deep feature layer Layer d3 And the fourth deep feature layer Layer d4 :

[0048] Layer d2 =Conv 3×3 (cat(RSU2(Layer d1 ), Layer d1 ))+Layer S1 ;

[0049] Layer d3 =Conv 3×3 (cat(RSU3(Layer d2 ), Layer d2 ))+Layer d1 ;

[0050] Layer d4 =Conv 3×3 (cat(RSU4(Layer d3 ), Layer d3 ))+Layer d2 ;

[0051] The purpose of using jumper connection to add layers is to better preserve the details of the original image. d1 、Layer d2 、Layerd3 and Layer d4 The sizes are the same, all 32×128×128.

[0052] Rs-Unet sub-blocks such as Figure 5 As shown, it is composed of the first weight layer, Unet convolutional neural network and convolutional layer C4 connected in sequence. The residual jumper is introduced to act on the Unet convolutional neural network, and the input of the first weight layer and the output of the Unet convolutional neural network are merged as the input of the convolutional layer C4. Among them, the convolutional layer C4 adopts the same parameter settings as C3; the first weight layer is composed of the convolutional layer C5 and the activation layer R2 connected in sequence. The convolution kernel size of the convolutional layer C5 is 3×3, the input channel is 32, the output channel is 32, and the padding is 1; the activation layer R2 uses the ReLU function. Therefore, the output of the Rs-Unet sub-block can be recorded as: RSU = Conv 3×3 (cat(Res(Unet(ReLU(Conv 3×3 (F in )))),F in )). Where Unet(·) represents the Unet convolutional neural network operation mechanism, Res represents the residual connection, and F in In this embodiment, the Rs-Unet sub-block does not change the size of the input features.

[0053] Unet convolutional neural network Figure 6 As shown, it is composed of 4 sequentially connected downsampling feature extraction sub-blocks (D1 to D4), 1 channel expansion sub-block E and 4 upsampling feature extraction sub-blocks (U1 to U4); the downsampling feature extraction sub-block D1 is composed of sequentially connected convolutional layers C D1-1 , activation layer R Unet , convolutional layer C D1-2 , activation layer R Unet With the pooling layer P Unet Among them, the convolution layer C D1-1 With the convolutional layer C D1-2 The parameters are the same: convolution kernel size is 3×3, input channels are 32, output channels are 32, padding is 1; activation layer R Unet Use LeakyReLU function; pooling layer P Unet Using maximum pooling, the convolution kernel size is 2×2, and the stride is 2. Thus, the first downsampling feature extraction layer is obtained. D1 for:

[0054] Layer D1 =Maxpool(LeReL U(Conv 3×3 (LeReL U(Conv3×3 (F in )))))) Wherein, the LeReLU(·) function is: LeReLU(x)=max(0.2×x,x). Figure 6 As shown in the figure, the structures of the downsampling feature extraction layer sub-blocks D2, D3 and D4 are the same as D1, the only difference is the convolution layer C D2-1 、C D2-2 、C D3-1 、C D3-2 、C D4-1 and C D4-2 Specifically, the convolution layer C D2-1 The input channel is 32 and the output channel is 64; the convolution layer C D2-2 The input channel is 64 and the output channel is 64; the convolution layer C D3-1 The input channel is 64 and the output channel is 128; the convolution layer C D3-2 The input channel is 128, the output channel is 128; the convolution layer C D4-1 The input channel is 128 and the output channel is 256; the convolution layer C D4-2 The input channel is 256 and the output channel is 256. Therefore, the same method can be used to obtain the second downsampling feature extraction layer Layer D2 , the third downsampling feature extraction layer D3 And the fourth downsampling feature extraction layer Layer D4 for:

[0055] Layer D2 =Maxpool(LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (Layer D1 )))))

[0056] Layer D3 =Maxpool(LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (Layer D2 )))))

[0057] Layer D4 =Maxpool(LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (Layer D3 )))))

[0058] After Maxpool pooling, the feature map size is halved. D1、Layer D2 、Layer D3 With Layer D4 The sizes are 32×64×64, 64×32×32, 128×16×16, and 256×8×8 respectively. Figure 6 As shown, the channel expansion sub-block E is composed of sequentially connected convolutional layers C E1-1 , activation layer R Unet , convolutional layer C E1-2 With activation layer R Unet Among them, the convolution layer C E1-1 The convolution kernel size is 3×3, the input channel is 256, the output channel is 512, and the padding is 1; the convolution layer C E1-2 The convolution kernel size is 3×3, the input channel is 512, the output channel is 512, and the padding is 1. In terms of working principle, the channel expansion sub-block E no longer performs downsampling on the input, but only expands its number of channels to enhance the network's ability to express different patterns and features. In this embodiment, the channel expansion layer E Can be written as:

[0059] Layer E =LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (Layer D4 )))),

[0060] Its size is 512×8×8.

[0061] like Figure 6 As shown, the upsampling feature extraction sub-block U1 consists of the transposed convolution layer T1, the convolution layer C U1-1 , activation layer R Unet , convolutional layer C U1-2 With activation layer R Unet Among them, the convolution kernel size of the transposed convolution layer T1 is 2×2, the input channel is 512, the output channel is 256, and the stride is 2; the convolution layer C U1-1 The convolution kernel size is 3×3, the input channel is 512, the output channel is 256, and the padding is 1; the convolution layer C U1-2 The convolution kernel size is 3×3, the input channel is 256, the output channel is 256, and the padding is 1. In terms of working principle, the upsampling feature extraction sub-block U1 is used to complete the opposite function of the downsampling feature extraction sub-block D4, that is, to increase the size of the feature map while reducing the dimension. In this process, in order to obtain better semantic information and improve the feature expression ability of the network, the Unet network also jumps the Layer D4Merge with U1, the first upsampling feature extraction layer U1 Can be written as:

[0062] Layer U1 =LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (cat(Conv 2×2 (Layer E ), Layer D4 )))))

[0063] Among them, Conv 2×2 (·) is the transposed convolution function.

[0064] Similarly, the structures of upsampling feature extraction sub-blocks U2, U3 and U4 are the same as U1, the only difference is the transposed convolution layers T2, T3, T4 and the convolution layer C U2-1 、C U2-2 、C U3-1 、C U3-2 、C U4-1 、C U4-2 Specifically, the input channel of the transposed convolution layer T2 is 256 and the output channel is 128; the convolution layer C U2-1 The input channel is 256 and the output channel is 128; the convolution layer C U2-2 The input channel of the convolution layer is 128, and the output channel is 128; the input channel of the transposed convolution layer T3 is 128, and the output channel is 64; the convolution layer C U3-1 The input channel is 128 and the output channel is 64; the convolution layer C U3-2 The input channel of the convolution layer is 64, and the output channel is 64; the input channel of the transposed convolution layer T4 is 64, and the output channel is 32; the convolution layer C U4-1 The input channel is 64 and the output channel is 32; the convolution layer C U4-2 The input channel is 32 and the output channel is 32. Therefore, the same method can be used to obtain the second upsampling feature extraction layer Layer U2 , the third upsampling feature extraction layer U3 And the fourth upsampling feature extraction layer Layer U4 for:

[0065] Layer U2 =LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (cat(Conv 2×2 (Layer U1 ), Layer D3 )))))

[0066] Layer U3 =LeReLU(Conv 3×3 (LeReLU(Conv 3×3 (cat(Conv 2×2 (Layer U2 ), Layer D2 )))))

[0067] Layer U4 =LeReLU(Conv 3x3 (LeReLU(Conv 3×3 (cat(Conv 2×2 (Layer U3 ), Layer D1 )))))

[0068] In this embodiment, Layer U1 、Layer U2 、Layer U3 and Layer U4 The sizes are 256×16×16, 128×32×32, 64×64×64, and 32×128×128 respectively.

[0069] The dual-channel attention mechanism block is as follows Figure 7 As shown in the figure, it is composed of a spatial attention mechanism sub-block and a channel attention mechanism sub-block in parallel; the spatial attention mechanism sub-block is composed of a dual-channel pooling module and a basic convolution module connected in sequence; the dual-channel pooling module is composed of a global average pooling layer P3 and a global maximum pooling layer P4 in parallel, and the outputs of P3 and P4 will be merged as the output of the dual-channel pooling module, and the basic convolution module is composed of a convolution layer C6 and an activation layer R M Among them, the global average pooling layer P3 and the global maximum pooling layer P4 are both channel pooling operations. For the feature tensor input of size [B, 32, 128, 128] (where B is the Batch Size during network training), P3 and P4 will perform average pooling and maximum pooling operations along the channel dimension respectively, and convert the above tensor into the size of [B, 1, 128, 128]. The convolution kernel size of the convolution layer C6 is 5×5, the input channel is 2, the output channel is 1, and the padding is 2. Using a large convolution kernel can reduce the number of parameters to a certain extent and reduce the risk of overfitting. The activation layer R M Using the Sigmoid function, its function expression is:

[0070]

[0071] The features after the Sigmoid activation layer will be mapped to the [O, 1] interval, and then act on the input features as a spatial attention map, that is, the spatial attention mechanism layer spa Can be written as:

[0072]

[0073] like Figure 7 As shown in the figure, the channel attention mechanism sub-block is composed of the adaptive average pooling layer P2 and the feature dimension transformation module connected in sequence; the feature dimension transformation module is composed of the convolution layer C7, the activation layer R3, the convolution layer C8 and the activation layer R M Specifically, the adaptive average pooling layer P2 uses the AdaptiveAvgPool2d block, and the output_size is 1. Unlike P3 and P4, P2 converts the input tensor of size [B, 32, 128, 128] to the size [B, 32, 1, 1], that is, it calculates the weight of each channel; the convolution kernel size of the convolution layer C7 is 1×1, the input channel is 32, the output channel is 4, and the padding is 0; the activation layer R3 uses the ReLU function; the convolution kernel size of the convolution layer C8 is 1×1, the input channel is 4, the output channel is 32, and the padding is 0. Similarly, the value after Sigmoid activation acts on the input feature as a channel attention map, that is, the channel attention mechanism layer Layer cha Can be written as:

[0074] Layer cha =Sig(Conv 3×3 (ReL U(Conv 3×3 (AdpPool(F in ))))).

[0075] In summary, the output layer of the dual-channel attention mechanism M By Layer spa With Layer cha After merging, the convolution kernel is 3×3 and the result is:

[0076]

[0077] In this embodiment, the dual-channel attention mechanism module does not change the size of the input features, that is, the input and output sizes are both 32×128×128.

[0078] The upsampling block is as follows Figure 8 As shown, the interpolation sampling layer I and the convolution layer C are connected in sequence. 10The interpolation sampling layer I uses the Interpolate block, and the scale_factor is 2, which converts the input feature map size from 32×128×128 to the original image size: 32×256×256; the convolution layer C 10 The convolution kernel size is 1×1, the number of input channels is 32, the number of output channels is 3, and the padding is 0. The activation layer R4 uses a Sigmoid block. In principle, the upsampling block is used to restore the feature map to a three-channel RGB image, the same size as the input image to be enhanced. In this embodiment, the final upsampled output size is 3×256×256.

[0079] Furthermore, the training process of the Residual-Unet low-light image enhancement network is as follows:

[0080] An existing image dataset is used as a training set. During the data loading phase, the low-light images to be enhanced in the training set and the normally exposed images used as labels are randomly cropped, flipped, and rotated at the same position to obtain images of size N×N for network training. The original low-light images are then preprocessed with histogram equalization, and the low-light images and the images obtained by histogram equalization preprocessing are used as input to constitute training samples. In this embodiment, the cropped image size N is 256.

[0081] Set the loss function and training parameters, and use the supervised batch learning method to train the low-light image enhancement network based on Residual-Unet. The loss function is:

[0082]

[0083] Where L(x,y) represents the loss function, Ref(x,y) represents the label, G(x,y) represents the output image generated by the Residual-Unet-based low-light image enhancement network, (x,y) represents the pixel coordinates, and N is the cropped image size. This loss function represents the "gap" between the illumination-enhanced image output by the Residual-Unet-based low-light image enhancement network and the properly exposed reference image.

[0084] In this example, 689 pairs of images in the LOL-v2-real dataset are used as the training set, and the remaining 100 pairs of images are used as the test set; Figure 1The example images in the flowchart are from the test set in LOL-v2-real. The original low-light image size is 400×600. To reduce computational overhead and facilitate network training, the image is cropped to 256×256, and the batch size during training is set to 24. The illumination enhancement model obtained by the present invention has a PSNR index of 25.942 when tested on the LOL-v2-real dataset, which significantly exceeds most existing illumination enhancement methods.

[0085] Step S3: Merge the image to be enhanced in step S1 with the image preprocessed by histogram equalization, input the image into the low-light image enhancement network based on Residual-Unet in step S2, and output the enhanced image from the low-light image enhancement network based on Residual-Unet.

[0086] Simulation experiment:

[0087] The experimental dataset used in the present invention is LOL-v2-real, which is divided into a training set and a test set. The number of images in the training set and the test set are 689 and 100 respectively. The evaluation indicator used in the comparative experiment is the peak signal-to-noise ratio (PSNR).

[0088] Table 1 is the comparative experimental results of each comparison method

[0089] method LIME GRI GRIR CRM RUAS PSNR 17.78 16.39 18.34 19.66 15.33

[0090] Table 2 is the comparative experimental results of each comparison method

[0091]

[0092] Table 3 is the comparative experimental results of each comparison method

[0093]

[0094] As shown in Tables 1 to 3, the peak signal-to-noise ratio (PSNR) value obtained using the method of the present invention is the highest, achieving an enhanced visual effect; the method of the present invention balances the relationship between the processing depth and receptive field of the image illumination enhancement network, and on this basis achieves a good enhancement effect, significantly improving the peak signal-to-noise ratio (PSNR), making it widely applicable to low-light image enhancement tasks in different scenarios.

[0095] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A low-light image enhancement method based on Residual-Unet network, characterized by: The method comprises the following steps: Step S1: data preprocessing; Obtain the image to be enhanced and perform histogram equalization preprocessing on it; Step S2: Build a low-light image enhancement network based on Residual-Unet and complete network training; The low-light image enhancement network based on Residual-Unet includes: a shallow feature encoding block, a Residual-Unet deep feature processing block, a dual-channel attention mechanism block and an upsampling block; The shallow feature encoding block includes a convolutional layer , Activate Layer , convolutional layer With the maximum pooling layer , convolutional layer , Activate Layer , convolutional layer With the maximum pooling layer The Residual-Unet deep feature processing block is composed of four Rs-Unet sub-blocks connected in sequence, and a convolutional layer with the same structure is embedded between every two Rs-Unet sub-blocks. ; The dual-channel attention mechanism block is composed of a spatial attention mechanism sub-block and a channel attention mechanism sub-block in parallel, and the upsampling block is composed of an interpolation sampling layer I, a convolutional layer connected in sequence. With activation layer constitute; The Rs-Unet sub-block includes the first weight layer, the Unet convolutional neural network and the convolution layer , the first weight layer, Unet convolutional neural network and convolutional layer Connect sequentially, introduce the residual structure and merge the input of the first weight layer with the output of the Unet convolutional neural network as the convolutional layer Input; Step S3: Merge the image to be enhanced in step S1 with the image preprocessed by histogram equalization, input the image into the low-light image enhancement network based on Residual-Unet in step S2, and output the enhanced image from the low-light image enhancement network based on Residual-Unet.

2. The low-light image enhancement method based on the Residual-Unet network according to claim 1, characterized in that: The first weight layer is composed of sequentially connected convolutional layers With activation layer The Unet convolutional neural network consists of a downsampling feature extraction sub-block, a channel expansion sub-block E and an upsampling feature extraction sub-block.

3. The low-light image enhancement method based on the Residual-Unet network according to claim 2, characterized in that: The downsampling feature extraction sub-block includes a downsampling feature extraction sub-block , downsampling feature extraction sub-block , downsampling feature extraction sub-block and downsampling feature extraction sub-block ; The upsampling feature extraction sub-block includes an upsampling feature extraction sub-block , upsampling feature extraction sub-block , upsampling feature extraction sub-block and upsampling feature extraction sub-block ; The Unet convolutional neural network consists of four downsampling feature extraction sub-blocks, one channel expansion sub-block E and four upsampling feature extraction sub-blocks connected in sequence.

4. The low-light image enhancement method based on the Residual-Unet network according to claim 3, characterized in that: The downsampling feature extraction sub-block Convolutional layers connected sequentially , Activate Layer , convolutional layer , Activate Layer With pooling layer Composition, downsampling feature extraction sub-block Convolutional layers connected sequentially , Activate Layer , convolutional layer , Activate Layer With pooling layer Composition, downsampling feature extraction sub-block Convolutional layers connected sequentially , Activate Layer , convolutional layer , Activate Layer With pooling layer Composition, downsampling feature extraction sub-block Convolutional layers connected sequentially , Activate Layer , convolutional layer , Activate Layer With pooling layer constitute; The channel expansion sub-block E is composed of sequentially connected convolutional layers , Activate Layer , convolutional layer With activation layer constitute; The upsampling feature extraction sub-block The transposed convolutional layers are connected sequentially , convolutional layer , Activate Layer , convolutional layer With activation layer Composition, upsampling feature extraction sub-block The transposed convolutional layers are connected sequentially , convolutional layer , Activate Layer , convolutional layer With activation layer Composition, upsampling feature extraction sub-block The transposed convolutional layers are connected sequentially , convolutional layer , Activate Layer , convolutional layer With activation layer Composition, upsampling feature extraction sub-block The transposed convolutional layers are connected sequentially , convolutional layer , Activate Layer , convolutional layer With activation layer Constructed and transposed convolutional layer 、 、 、 The output of the convolutional layer is respectively 、 、 、 The outputs of the convolutional layers are combined and 、 、 and input.

5. The low-light image enhancement method based on the Residual-Unet network according to claim 1, characterized in that: The spatial attention mechanism sub-block is composed of a dual-channel pooling module and a basic convolution module connected in sequence. The dual-channel pooling module is composed of a global average pooling layer. and global max pooling layer In parallel, and The output of will be merged as the output of the dual-channel pooling module; the basic convolution module consists of the convolution layer and activation layers The channel attention mechanism sub-block is composed of sequentially connected adaptive average pooling layers and feature dimension transformation module; the feature dimension transformation module is composed of convolution layer , Activate Layer , convolutional layer With activation layer constitute; The spatial attention mechanism and channel attention mechanism will generate spatial attention maps and channel attention maps respectively and act on the original input features respectively and then merge them and pass through the convolution layer. The pixel sum is processed with the original features, and the pixel sum result is used as the output of the dual-channel attention mechanism block.

6. The low-light image enhancement method based on the Residual-Unet network according to claim 2, characterized in that: The convolutional layer The convolution kernel size is , input channels are 6, output channels are 16, padding is 1; activation layer LeakyReLU and negative_slope is 0.2; convolutional layer The convolution kernel size is , input channel is 16, output channel is 32, padding is 1; maximum pooling layer The convolution kernel size is , stride is 2; convolutional layer The convolution kernel size is , input channel is 64, output channel is 32, padding is 1; convolution layer With convolutional layer Same parameters; convolutional layer The convolution kernel size is , input channel is 32, output channel is 32, padding is 1; activation layer is the ReLU function.

7. The low-light image enhancement method based on the Residual-Unet network according to claim 4, characterized in that: The convolutional layer With convolutional layer The parameters are the same: the convolution kernel size is , input channel is 32, output channel is 32, padding is 1; convolution layer The input channel is 32 and the output channel is 64; the convolution layer The input channel is 64 and the output channel is 64; the convolution layer The input channel is 64 and the output channel is 128; the convolution layer The input channel is 128 and the output channel is 128; the convolution layer The input channel is 128 and the output channel is 256; the convolution layer The input channel is 256 and the output channel is 256; the convolution layer The input channel is 256 and the output channel is 512; the convolution layer The input channel is 512, the output channel is 512; the convolution layer 、 、 、 The input channels and output channels of the convolutional layer are 、 、 and On the contrary; convolutional layer 、 、 and The input channels and output channels of the convolutional layer are 、 、 and Same; except for the input and output channels, the convolutional layer ~ , convolutional layer With convolutional layer ~ The parameters are the same: the convolution kernel size is , padding is 1; activation layer Use LeakyReLU block; pooling layer Using maximum pooling, the convolution kernel size is , stride is 2; the transposed convolution layer T uses ConvTranspose2d block, and the convolution kernel size is , stride are both 2, where The input channel is 512 and the output channel is 256. The input channel is 256 and the output channel is 128. The input channels are 128 and the output channels are 64. The input channel is 64 and the output channel is 32; the global average pooling layer and global max pooling layer Both are channel pooling operations, calculating the average value and the maximum value along the channel dimension of the feature map; convolutional layer The convolution kernel size is , input channel is 2, output channel is 1, padding is 2; activation layer Use Sigmoid block; Adaptive average pooling layer AdaptiveAvgPool2d block is used, and output_size is 1; convolutional layer The convolution kernel size is , input channels are 32, output channels are 4, padding is 0; activation layer Use ReLU block; convolutional layer The convolution kernel size is , input channel is 4, output channel is 32, padding is 0; convolution layer The convolution kernel size is , the input channel is 64, the output channel is 32, and the padding is 0; the interpolation sampling layer I uses the Interpolate block, and the scale_factor is 2; the convolution layer The convolution kernel size is , input channel is 32, output channel is 3, padding is 0; activation layer Use Sigmoid block.

8. The low-light image enhancement method based on the Residual-Unet network according to claim 1, characterized in that: The training process of the Residual-Unet low-light image enhancement network is as follows: The existing image dataset is used as the training set. During the data loading phase, the low-light images to be enhanced in the training set and the normal exposure images used as labels are randomly cropped, flipped, and rotated at the same position to obtain a dataset of size The original low-light image is preprocessed with histogram equalization, and the low-light image and the image obtained by histogram equalization preprocessing are used as input to form training samples; The loss function and training parameters are set, and the supervised batch learning method is used to train the low-light image enhancement network based on Residual-Unet.

9. The low-light image enhancement method based on the Residual-Unet network according to claim 8, characterized in that: The loss function is: ； Where L(x,y) represents the loss function, Ref(x,y) represents the label, G(x,y) represents the output image generated by the low-light image enhancement network based on Residual-Unet, (x,y) represents the pixel coordinates, and N is the size of the cropped image.

Images