A plug-and-play low-light image enhancement method for end-to-end object detection under low-light conditions

Abstract

The present invention discloses a plug-and-play low-light image enhancement method for end-to-end target detection under low-light conditions. The low-light image enhancement model consists of a parameter estimation module and an image processing module. The parameter estimation module regresses the hyperparameters required by the image processing module by learning the information fed back by the detection network. The image processing module that receives the hyperparameters adaptively enhances the input image to promote its detection effect under low-light conditions. The enhancement model and the detection model are integrated, end-to-end, complementary to each other, and benefit from each other. The image is enhanced according to the information fed back by the detection network, and the detection network is better trained based on the enhanced image. Compared with previous two-stage image enhancement models or other end-to-end methods, the plug-and-play low-light image enhancement model for end-to-end target detection under low-light conditions proposed by the present invention achieves better performance.

Description

Technical Field

[0001] The present invention relates to a plug-and-play low-light image enhancement model applied to end-to-end target detection under low-light conditions, belonging to the technical field of target detection. Background Art

[0002] Although deep learning-based object detection algorithms have been widely used in many scenarios, they still face challenges such as missed detection and false detection under non-ideal conditions such as low light. Therefore, object detection algorithms often adopt many image enhancement methods, but the enhancement model and the object detection model are usually independent of each other. This two-stage approach makes it difficult to unify the consistency of enhancement and detection. As a result, low-light enhancement operations do not always contribute to the subsequent object detection task. Recently, some works have attempted to integrate enhancement and detection in an end-to-end network, but they still face problems such as complex network structure, difficult training convergence, and lack of reference images for enhancement.

[0003] To address these issues, this paper proposes a plug-and-play low-light image enhancement model for end-to-end object detection in low-light conditions. This model can be easily integrated into existing object detection algorithms in an end-to-end fashion. The image enhancement module adaptively enhances images based on feedback from the subsequent detection network, generating images that are beneficial for subsequent detection tasks, thereby improving the performance of object detection algorithms. Summary of the Invention

[0004] The present invention proposes a plug-and-play low-light image enhancement method for end-to-end target detection under low-light conditions. The low-light image enhancement model consists of a parameter estimation module and an image processing module. The parameter estimation module regresses the hyperparameters required by the image processing module by learning the information fed back by the detection network. The image processing module, which receives the hyperparameters, adaptively enhances the input image to improve its detection effect under low-light conditions. The enhancement model and the detection model are integrated end-to-end, complementing each other and benefiting each other. The image is enhanced based on the information fed back by the detection network, and the detection network is better trained based on the enhanced image.

[0005] Specifically, the model for implementing the low-light image enhancement method consists of a parameter estimation network module and an image processing module, which in turn includes a pixel-level image enhancement operation unit and a sharpening operation unit.

[0006] In the pixel-level image enhancement operation unit, low-light images are analyzed from the aspects of image quality evaluation and image pixel imaging characteristics. At the same time, inspired by the curve adjustment tool in image editing software, a pixel-level image enhancement operation is designed, and its expression is as follows:

[0007] I'=A(1-I)I+I

[0008] Here, A is the parameter matrix obtained by the parameter estimation network. Unlike traditional 1×1 hyperparameters, the size of A is the same as the input image size. This means that each pixel in the original image can be locally adaptively adjusted using the parameter matrix A. I represents the original image, while I' represents the enhanced image. This expression is actually a quadratic equation. The matrix parameter A modifies the expressive power of the quadratic equation to control exposure. This equation is simple and differentiable. Furthermore, this operation can be further enhanced in low-light conditions by stacking multiple layers.

[0009] Sharpening operation unit: sharpens the image to highlight the image details. The sharpening operation is described by the following expression:

[0010] I_enhance=λ(I′−G(I′))+I′

[0011] Here, I' is the input image to the sharpening operation (i.e., the output of the pixel enhancement operation in the proposed framework), G(·) is the Gaussian filter, and λ is the scaling factor obtained by the parameter estimation network. I' - G(I') highlights the edge information in the image that should be enhanced. λ adaptively adjusts the degree of image sharpening. The sharpening operation automatically adds image edge details to the original image to increase sharpness. Similarly, this block can also be stacked.

[0012] In order to better balance the relationship between image enhancement and object detection, a parameter estimation network module is proposed to automatically learn the optimal enhancement hyperparameters. The input of the parameter estimation module is a low-light image, and the output is a set of key parameters required by the image enhancement model, namely the parameter matrix A required by the pixel-level image enhancement module and the scale factor λ in the sharpening operation. Figure 2 As shown in the figure, the parameter estimation network has a simple structure, with skip connections between multiple convolutional layers implemented through channel-wise connections. Each layer contains 32 convolutions with a kernel size of 3×3, a stride size of 1, and a Reluctant Unit (ReLU) activation function. To prevent information loss and fully consider the relationship between adjacent pixels, downsampling and batch normalization are not used. The final convolutional layer uses the Tanh activation function to generate the corresponding number of adjustable parameters for the subsequent pixel enhancement and sharpening operations. Finally, the parameter matrix output of the parameter estimation network is the same as the original image, and some parameters are average pooled to generate the 1x1 hyperparameters required for the sharpening operation.

[0013] Initialization: Parameter estimation network parameters P θ and SSD network parameters S β

[0014] Set the training phase: training iterations 60

[0015] Prepare relevant datasets (ExDark)

[0016] The schematic diagram of the embodiment is as follows Figure 1 As shown, the following steps are included:

[0017] Step 1: Resize the low-light image to (300, 300) to obtain image I;

[0018] Step 2: Calculate the output of the parameter estimation network module and get P N ;

[0019] Step 3: Pass Figure 2 The hyperparameters of the method are converted to obtain the hyperparameters of the corresponding format;

[0020] Step 4: Receive the corresponding hyperparameters and perform pixel-level image enhancement and sharpening operations on the original image to obtain an enhanced image I';

[0021] Step 5: The enhanced image I' is fed into the object detector SSD;

[0022] Step 6: Perform gradient backpropagation based on the SSD loss and update the parameter estimation network parameters P in real time θ and SSD network parameters S β ;

[0023] Step 7: Iterate the loop to complete the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The entire pipeline for image augmentation models and object detectors.

[0025] Figure 2 Schematic diagram of the generation and transformation of hyperparameters. DETAILED DESCRIPTION

[0026] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0027] Figure 1 The entire pipeline for the image enhancement model and object detector is constructed. To form the entire pipeline, the low-light image enhancement model can be inserted before any existing object detection model. The low-light image enhancement model primarily consists of a parameter estimation network module and an image processing module. The parameter estimation network module is a lightweight convolutional neural network that adaptively regresses the hyperparameters required by the image processing module using feedback from the entire network. The image processing module includes pixel-level image enhancement and sharpening operations, which adaptively enhance the image based on the hyperparameters.

[0028] Figure 2Figure 1. Schematic diagram of hyperparameter generation and transformation. The parameter estimation network generates an H×W parameter feature map. The resulting parameter matrix A is directly input into the pixel-level image enhancement operation, while the parameters for the sharpening operation are obtained through average pooling.

[0029] Take the SSD object detection algorithm as an example. In SSD, fixed bounding boxes are used for object proposals, and multi-layer feature information is used to detect objects at different scales. ResNet50 is used as the backbone network of SSD. It is worth noting that the proposed image enhancement model does not require any additional loss function to constrain it, so the loss function of the entire model is the same as that of the original SSD model, as shown below:

[0030] L cls_ssd =-log(softmax(Det cls_ssd (x)))

[0031] L reg_ssd =smooth L1 (Det reg_ssd (x, t)

[0032] L det_ssd =L cls_ssd +αL reg_ssd

[0033] Among them, Det cls_ssd and Det reg_ssd is the classifier and regressor of SSD. L cls_ssd is the classification loss L of SSD reg_ssd is the coordinate regression loss of SSD. L det_ssd is the total loss of the entire model for image enhancement and object detection. As can be seen, the entire pipeline is supervised by only one task-specific loss, which makes the proposed image enhancement model easily extendible to other computer vision tasks such as classification, semantic / instance segmentation, etc.

[0034] Implementation method.

[0035] Initialization: Parameter estimation network parameters P θ and SSD network parameters S β

[0036] Set the training phase: training iterations 60

[0037] Prepare relevant datasets (ExDark)

[0038] The schematic diagram of the embodiment is as follows Figure 1 As shown, the following steps are included:

[0039] Step 1: Resize the low-light image to (300, 300) to obtain image I;

[0040] Step 2: Calculate the output of the parameter estimation network module and get P N ;

[0041] Step 3: Pass Figure 2 The hyperparameters of the method are converted to obtain the hyperparameters of the corresponding format;

[0042] Step 4: Receive the corresponding hyperparameters and perform pixel-level image enhancement and sharpening operations on the original image to obtain an enhanced image I';

[0043] Step 5: The enhanced image I' is fed into the object detector SSD;

[0044] Step 6: Perform gradient backpropagation based on the SSD loss and update the parameter estimation network parameters P in real time θ and SSD network parameters S β ;

[0045] Step 7: Iterate the loop to complete the training process.

[0046] Experiment and analysis

[0047] This method has achieved promising results on low-light datasets such as ExDark. Compared to previous two-stage image enhancement models or other end-to-end approaches, the plug-and-play low-light image enhancement model proposed in this paper for end-to-end object detection in low-light conditions achieves superior performance. Experiments have shown that detectors incorporating the low-light image enhancement model achieve significant performance improvements.

Claims

1. A plug-and-play low-light image enhancement method for end-to-end object detection under low-light conditions, characterized in that: The model for implementing the low-light image enhancement method consists of a parameter estimation network module and an image processing module; The image processing module includes a pixel-level image enhancement operation unit and a sharpening operation unit; In the pixel-level image enhancement operation unit, the low-light image is analyzed from the aspects of image quality evaluation and image pixel imaging characteristics, and a pixel-level image enhancement operation is designed. Its expression is as follows: I'=A(1-I)I+I Where A is the parameter matrix obtained by the parameter estimation network. Unlike traditional 1×1 hyperparameters, the size of A is the same as the input image. Each pixel of the original image can be locally adaptively adjusted through the parameter matrix A. I represents the original image, and I' represents the enhanced image. This expression is actually a quadratic equation. The matrix parameter A modifies the expressive power of the quadratic equation and controls the exposure. Low-light enhancement is further achieved by stacking multiple layers. Sharpening operation unit: sharpens the image to highlight the image details; the sharpening operation is described by the following expression: I_enhance=λ(I′−G(I′))+I′ Where I' is the input image for the sharpening operation, G(·) is the Gaussian filter, and λ is the scale factor obtained by the parameter estimation network; I'-G(I') highlights the boundary information that should be enhanced in the image; λ adaptively adjusts the sharpness of the image; The parameter estimation network module is a lightweight convolutional neural network that adaptively regresses the hyperparameters required by the image processing module through feedback information from the entire network. The input of the parameter estimation network module is a low-light image, and the output is a set of key parameters required by the image enhancement model, namely the parameter matrix A required by the pixel-level image enhancement module and the scaling factor λ in the sharpening operation. The jump connection of multiple convolutional layers is realized through channel connection operations. Each convolutional layer contains 32 convolutions, the convolution kernel size is 3×3, the step size is 1, and the activation function is ReLU. The last convolution layer uses the Tanh activation function to generate the corresponding number of adjustable parameters for the subsequent pixel enhancement and sharpening operations. The parameter matrix output of the parameter estimation network module is the same as the original image, and the parameter matrix generates the 1x1 hyperparameters required for the sharpening operation through the average pooling operation.

2. The plug-and-play low-light image enhancement method for end-to-end target detection under low-light conditions according to claim 1, characterized in that: The following steps are involved: Step 1: Resize the low-light image to obtain image I; Step 2: Calculate the output of the parameter estimation network module and get P N ; Step 3: Obtain the hyperparameters in the corresponding format through hyperparameter conversion; Step 4: Receive the corresponding hyperparameters and perform pixel-level image enhancement and sharpening operations on the original image to obtain an enhanced image I'; Step 5: The enhanced image I' is fed into the object detector SSD; Step 6: Perform gradient backpropagation based on the SSD loss and update the parameter estimation network parameters P in real time θ and SSD network parameters S β ; Step 7: Iterate the loop to complete the training process.

Images