Unsupervised target detection-oriented low-illumination image re-enhancement method and device

Abstract

The embodiment of the invention provides an unsupervised target detection-oriented low-illumination image re-enhancement method and device, and the method comprises the steps: obtaining a low-illumination image data set which comprises a label-free low-illumination image and a paired-free normal-illumination image of a target object in a real scene; inputting the low-illumination image data set into a pre-trained low-illumination enhancement model to obtain a plurality of groups of preliminary enhancement images, inputting the plurality of groups of preliminary enhancement images into a pre-trained target detection model for reasoning, and determining a group of preliminary enhancement images with the best target detection precision; and constructing an image re-enhancement module, inputting a group of preliminarily enhanced images with the best target detection precision into the image re-enhancement module, and carrying out unsupervised re-enhancement facing the target detection task to obtain re-enhanced images.

Description

Technical Field

[0001] This document relates to the field of computer technology, and more particularly to an unsupervised low-light image enhancement method and device for target detection. Background Art

[0002] Most of the existing low-light image enhancement methods are based on human visual perception results, and are mainly based on image quality indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). They focus on the image quality after image enhancement. Although such methods can improve image quality to a certain extent, they also have many shortcomings. For example, the evaluation indicators of image enhancement (such as PSNR and SSIM) are inconsistent with the evaluation indicators of target detection (such as mAP and IOU), resulting in the effect of image enhancement and the effect of target detection not necessarily being positively correlated, and image enhancement may introduce some factors that are not conducive to target detection, such as noise, artifacts, color shift, etc.

[0003] There are already some methods that retrain the low-light enhancement module by adding semantic information guidance, guiding the image to restore in the direction of containing rich semantic information. This type of method can improve the target detection effect to a certain extent, but it requires retraining the existing low-light enhancement module, and the evaluation indicators are still not directly matched with the target detection results.

[0004] In summary, the problems existing in the prior art are as follows: the evaluation indicators of image enhancement (such as PSNR and SSIM) and the evaluation indicators of target detection (such as mAP and IOU) are inconsistent, resulting in the problem that the effect of image enhancement and the effect of target detection are not necessarily positively correlated. The existing low-light enhancement method for target detection tasks requires retraining the existing low-light enhancement module, and the evaluation indicators are still not directly matched with the target detection results, and cannot be plug-and-play in specific application scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide an unsupervised low-light image enhancement method and device for target detection, aiming to solve the above-mentioned problems in the prior art.

[0006] The present invention provides an unsupervised low-light image enhancement method for target detection, comprising:

[0007] Obtain a low-light image dataset containing unlabeled and unpaired normal-light images of target objects in real scenes;

[0008] Inputting the low-light image data set into a pre-trained low-light enhancement model to obtain multiple groups of preliminary enhanced images, and inputting the multiple groups of preliminary enhanced images into a pre-trained target detection model for reasoning to determine a group of preliminary enhanced images with the best target detection accuracy;

[0009] An image enhancement module is constructed, and a group of preliminary enhanced images with the best target detection accuracy are input into the image enhancement module for unsupervised target detection task-oriented enhancement to obtain enhanced images.

[0010] The present invention provides an unsupervised low-light image enhancement device for target detection, comprising:

[0011] An acquisition module is used to acquire a low-light image dataset containing unlabeled and unpaired normal-light images of target objects in real scenes;

[0012] A preliminary enhancement module, used to input the low-light image data set into a pre-trained low-light enhancement model to obtain multiple groups of preliminary enhanced images, and input the multiple groups of preliminary enhanced images into a pre-trained target detection model for reasoning to determine a group of preliminary enhanced images with the best target detection accuracy;

[0013] The re-enhancement module is used to construct an image re-enhancement module, and a group of preliminary enhanced images with the best target detection accuracy are input into the image re-enhancement module for unsupervised re-enhancement for target detection tasks to obtain re-enhanced images.

[0014] An embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the above-mentioned unsupervised low-light image enhancement method for target detection are implemented.

[0015] An embodiment of the present invention also provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by a processor, the steps of the above-mentioned unsupervised low-light image enhancement method for target detection are implemented.

[0016] By adopting the embodiment of the present invention, there is no need to retrain the existing enhancement module and detection module, and the image enhancement evaluation index is directly linked to the target detection evaluation index, so that image enhancement can be directly performed for the target detection task, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is a flow chart of an unsupervised low-light image enhancement method for target detection according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of an algorithm verification framework of an embodiment of the present invention;

[0020] Figure 3 is an architecture diagram of an enhancement module according to an embodiment of the present invention;

[0021] Figure 4 This is a comparison diagram of the effects of the unsupervised target detection-oriented low-light image enhancement method according to an embodiment of the present invention and the prior art;

[0022] Figure 5 This is a comparison diagram of the unsupervised target detection-oriented low-light image enhancement method according to an embodiment of the present invention and the prior art data;

[0023] Figure 6 is a schematic diagram of an unsupervised low-light image enhancement device for target detection according to an embodiment of the present invention;

[0024] Figure 7 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0026] Method Embodiment

[0027] According to an embodiment of the present invention, an unsupervised low-light image enhancement method for target detection is provided. Figure 1 is a flow chart of an unsupervised low-light image enhancement method for target detection according to an embodiment of the present invention. Figure 1As shown, the unsupervised low-light image enhancement method for target detection according to an embodiment of the present invention specifically includes:

[0028] Step S101, obtaining a low-light image dataset containing unlabeled and unpaired normal-light images of target objects in a real scene;

[0029] Step S102, inputting the low-light image data set into a pre-trained low-light enhancement model to obtain multiple groups of preliminary enhanced images, and inputting the multiple groups of preliminary enhanced images into a pre-trained target detection model for reasoning to determine a group of preliminary enhanced images with the best target detection accuracy;

[0030] Step S103, constructing an image enhancement module, inputting a group of preliminary enhanced images with the best target detection accuracy into the image enhancement module for unsupervised target detection task-oriented enhancement to obtain enhanced images. Constructing the image enhancement module specifically includes:

[0031] The first three layers of VGG pre-trained on ImageNet are used as encoders, and three layers of reversible residual blocks are used as decoders to jointly construct an image reconstruction network and design the loss function to obtain the image enhancement module. In the reversible residual block, the activation value of the current layer is calculated by the activation value of the next layer, and only the last layer activation value of each block needs to be stored.

[0032] The design of the loss function specifically includes:

[0033] The loss function is designed based on the number of detection boxes and confidence scores after non-maximum suppression output by inference, and then added to the cosine similarity loss to form the total loss function to guide image reconstruction so that the reconstructed image has higher detection accuracy in target detection tasks.

[0034] Specifically, the total loss function is determined according to Formula 1, Formula 2, and Formula 3:

[0035]

[0036] loss ent =α 1 ×m+α 2 ×score formula 2;

[0037] Total loss function = loss sim +loss ent Formula 3;

[0038] Among them, I en is an image that has been enhanced using the low-light enhancement module. It is the designed reconstruction module Ien The reconstructed image, loss sim is the cosine similarity loss, m is the format of the detection box predicted by the detector, score is the average confidence score predicted, and α 1 and α 2 are two given parameters.

[0039] As described above, the unsupervised low-light image re-enhancement method for object detection provided by the embodiments of the present invention does not require retraining the existing low-light enhancement module and detector, nor does it require using low-light image object detection labels. Based on a simple image reconstruction module, an entropy loss is designed to constrain the image reconstruction effect, so that the reconstructed image can adapt to the detector and improve the detection accuracy.

[0040] The above technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0041] As Figure 2 shown, the unsupervised low-light image re-enhancement method for object detection according to the embodiments of the present invention specifically includes the following steps:

[0042] 1. Obtain the most commonly used dataset for the performance of the current low-light enhancement algorithm in the object detection task, which is used to train and verify the effect of the enhancement module, and can fully prove the effectiveness of the algorithm. In the preferred example of the present invention, the dataset Dark Face is selected for algorithm verification, and the training set and the verification set are divided according to a ratio of 4:1 for training and verifying the effect of the enhancement module. DarkFace is a dataset for face detection in night (low-light) scenes and is also one of the most commonly used datasets for the performance of the current low-light enhancement algorithm in the object detection task, which can fully prove the effectiveness of the algorithm.

[0043] 2. Select existing pre-trained and relatively good low-light enhancement algorithms, and respectively send the above-obtained datasets into existing relatively good low-light enhancement models such as Zero-DC, RUAS, SCI, Pairlie, LLFlow, etc. for inference to obtain multiple groups of enhanced images.

[0044] 3. Select an existing widely used target detection model, train the target detection model with a normal lighting data set containing the target object, and obtain the pre-trained target detection model weights. In a preferred embodiment of the present invention, the Yolov5 algorithm is one of the most widely used target detection algorithms at present. It is the fifth-generation target detection model in the "You Only Look Once" (YOLO) series. It is based on the PyTorch framework and aims to provide high-speed and high-precision real-time target detection. Due to its efficient performance and ease of use, YOLOv5 has been widely used in various application scenarios, including monitoring systems, autonomous driving, and image recognition. Therefore, the present invention selects Yolov5 as the back-end detector of the algorithm verification framework. The detector is pre-trained on the normal lighting face detection data set widerface for testing on dareface.

[0045] 4. Send the above multiple sets of enhanced images to the pre-trained target detection model for inference, compare the target detection accuracy results, and select the best set as the training and test data set for the subsequent image enhancement module. Figure 4 According to the test results in , Zero-DCE performs better than other algorithms on Dark Face. Selecting Zero-DCE as the front-end low-light enhancement module of the algorithm verification framework of the present invention can not only better prove the effectiveness of the algorithm of the embodiment of the present invention, but also has more practical application value.

[0046] That is to say, the above-mentioned acquired data set is sent to multiple existing pre-trained low-light enhancement models, the results of the processed images on the target detection model are compared, and the processed image with the highest target detection accuracy is selected as the input of the next enhancement module.

[0047] 5. Build a network model. Figure 3 As shown in the figure, the first three layers of VGG pre-trained on ImageNet are used as encoders, and three layers of reversible residual blocks are used as decoders to jointly construct a simple image reconstruction network. The purpose of using the first three layers of pre-trained VGG is to better extract image features and ensure that the basic shallow features before and after image reconstruction will not change. It is only to perform adaptive optimization for the detector. In the reversible residual block, the activation value of the current layer can be calculated by the activation value of the next layer. Therefore, only the activation value of the last layer of each block needs to be stored, and a deeper network can be trained on limited video memory space.

[0048] 6. Loss function design. The output of the target detection model contains three types of information: category, coordinates of the target detection box, and confidence score, among which the confidence score can reflect the accuracy of target recognition and positioning. Therefore, without using labels, the present invention uses the two pieces of information of the number of detection boxes and confidence scores after non-maximum suppression (NMS) output by inference to design a loss function, and adds the cosine similarity loss to form a total loss, which guides image reconstruction and enables the reconstructed image to have higher detection accuracy in target detection tasks.

[0049] Specifically, the cosine similarity loss is used to keep the content of the image before and after basically unchanged. The entropy loss is constructed using the predicted output of the dataset using the pre-trained object detection model. First, all predicted annotation boxes are subjected to NMS (non-maximum suppression), the number and confidence scores of the remaining annotation boxes with higher confidence scores are counted, and the loss function is constructed.

[0050]

[0051] loss ent =α 1 ×m+α 2 ×score#(2)

[0052] Among them I en This is an image that has been enhanced using the low-light enhancement module Zero-DCE. It is the designed reconstruction module I en Reconstructed image, loss sim is the cosine similarity loss, m is the format of the detection box predicted by the detector, score is the average confidence score of the prediction, and α 1 and α 2 are two given parameters.

[0053] 7. Send the image obtained in step 4 after preliminary enhancement to the complete model constructed in steps 5 and 6 for training and reasoning, and finally send the output enhanced image to the pre-trained detector for detection to obtain the target detection accuracy map@.5. Experimental results show that the target detection accuracy of the image can be improved. In the preferred embodiment of the present invention, the training is a training set divided by darkface, which is enhanced by zero-dce and sent to the network constructed in steps 5 and 6 for training. Then, the test set is also enhanced by zero-dce and sent to the trained model for reasoning enhancement. Finally, the output image is sent to the pre-trained YOLOV5 detector for detection to obtain the target detection accuracy map@.5. Figure 3 and Figure 4The experimental results show that the enhanced image has no obvious change in the visual effect of the human eye, but can improve the target detection accuracy of the image, which fully proves the effectiveness of the algorithm of the embodiment of the present invention in theory and practical effects.

[0054] The beneficial effects brought by the present invention are:

[0055] The present invention realizes a new technical route. Without the need to retrain the existing low-light enhancement module and detector, and without the need to use low-light image target detection labels, the detection accuracy of the low-light enhanced image in the target detection task can be further improved, thereby achieving consistency between the image enhancement direction and the target detection task evaluation index. In order to achieve consistency between the image enhancement direction and the target detection task evaluation index, the present invention designs an entropy loss, and designs a loss function using the two pieces of information, the number of detection frames and the confidence score inferred by the detector. The loss is added to the cosine similarity loss to form a total loss, which is used to constrain the enhanced image to be able to detect more detection frames with high confidence scores, thereby improving the target detection accuracy. At the same time, a lightweight reconstruction module is also selected to ensure the speed, efficiency and practical application value of the entire algorithm. Comparison of the effects of the unsupervised low-light image re-enhancement method for target detection with the prior art Figure 4 As shown in Figure 2, the unsupervised low-light image enhancement method for object detection is compared with the existing technology data. Figure 5 shown.

[0056] Device Example 1

[0057] According to an embodiment of the present invention, an unsupervised low-light image enhancement device for target detection is provided. Figure 6 is a schematic diagram of an unsupervised low-light image enhancement device for target detection according to an embodiment of the present invention. Figure 6 As shown, the unsupervised low-light image enhancement device for target detection according to an embodiment of the present invention specifically includes:

[0058] An acquisition module 60 is used to acquire a low-light image dataset containing unlabeled and unpaired normal-light images of target objects in a real scene;

[0059] A preliminary enhancement module 62 is used to input the low-light image data set into a pre-trained low-light enhancement model to obtain multiple groups of preliminary enhanced images, and input the multiple groups of preliminary enhanced images into a pre-trained target detection model for reasoning to determine a group of preliminary enhanced images with the best target detection accuracy;

[0060] The re-enhancement module 64 is used to construct an image re-enhancement module, and input a group of preliminary enhanced images with the best target detection accuracy into the image re-enhancement module for unsupervised re-enhancement for target detection tasks to obtain re-enhanced images. Specifically used for:

[0061] Use the first three layers of VGG pre-trained on ImageNet as encoders and three layers of reversible residual blocks as decoders to jointly construct an image reconstruction network and design a loss function to obtain an image enhancement module. In the reversible residual block, the activation value of the current layer is calculated by the activation value of the next layer, and only the last layer activation value of each block needs to be stored. The loss function is designed using the number of detection boxes and confidence scores after non-maximum suppression output by inference, and then added to the cosine similarity loss to form a total loss function to guide image reconstruction so that the reconstructed image has higher detection accuracy in the target detection task. Determine the total loss function according to Formula 1, Formula 2, and Formula 3:

[0062]

[0063] loss ent =α 1 ×m+α 2 ×score formula 2;

[0064] Total loss function = loss sim +loss ent Formula 3;

[0065] Among them, I en is an image that has been enhanced using the low-light enhancement module. It is the designed reconstruction module I en Reconstructed image, loss sim is the cosine similarity loss, m is the format of the detection box predicted by the detector, score is the average confidence score of the prediction, and α 1 and α 2 are two given parameters.

[0066] The embodiment of the present invention is a device embodiment corresponding to the above-mentioned method embodiment. The specific operations of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0067] Device Example 2

[0068] An embodiment of the present invention provides an electronic device, such as Figure 7As shown, it includes: a memory 70, a processor 72, and a computer program stored in the memory 70 and executable on the processor 72. When the computer program is executed by the processor 72, the steps described in the method embodiment are implemented.

[0069] Device Example 3

[0070] An embodiment of the present invention provides a computer-readable storage medium, on which a program for implementing information transmission is stored. When the program is executed by the processor 72, the steps described in the method embodiment are implemented.

[0071] The computer-readable storage medium in this embodiment includes, but is not limited to, ROM, RAM, magnetic disk or optical disk, etc.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised low-light image enhancement method for target detection, characterized in that: include: Obtain a low-light image dataset containing unlabeled and unpaired normal-light images of target objects in real scenes; Inputting the low-light image data set into a pre-trained low-light enhancement model to obtain multiple groups of preliminary enhanced images, and inputting the multiple groups of preliminary enhanced images into a pre-trained target detection model for reasoning to determine a group of preliminary enhanced images with the best target detection accuracy; An image enhancement module is constructed, and a group of preliminary enhanced images with the best target detection accuracy are input into the image enhancement module for unsupervised target detection task-oriented enhancement to obtain enhanced images.

2. The method according to claim 1, characterized in that: Building an image enhancement module specifically includes: The first three layers of VGG pre-trained on ImageNet are used as encoders, and three layers of reversible residual blocks are used as decoders to jointly construct an image reconstruction network and design the loss function to obtain the image enhancement module. In the reversible residual block, the activation value of the current layer is calculated by the activation value of the next layer, and only the last layer activation value of each block needs to be stored.

3. The method according to claim 2, characterized in that The design of the loss function specifically includes: The loss function is designed based on the number of detection boxes and confidence scores after non-maximum suppression output by inference, and then added to the cosine similarity loss to form the total loss function to guide image reconstruction so that the reconstructed image has higher detection accuracy in target detection tasks.

4. The method according to claim 3, characterized in that The design of the loss function specifically includes: The total loss function is determined according to Formula 1, Formula 2 and Formula 3: loss ent = α1×m+α2×score Formula 2; Total loss function = loss sim +loss ent Formula 3; Among them, I en is an image that has been enhanced using the low-light enhancement module. It is the designed reconstruction module I en Reconstructed image, loss sim is the cosine similarity loss, m is the format of the detection box predicted by the detector, score is the predicted average confidence score, and α1 and α2 are two given parameters.

5. An unsupervised low-light image enhancement device for target detection, characterized in that: include: An acquisition module is used to acquire a low-light image dataset containing unlabeled and unpaired normal-light images of target objects in real scenes; A preliminary enhancement module, used to input the low-light image data set into a pre-trained low-light enhancement model to obtain multiple groups of preliminary enhanced images, and input the multiple groups of preliminary enhanced images into a pre-trained target detection model for reasoning to determine a group of preliminary enhanced images with the best target detection accuracy; The re-enhancement module is used to construct an image re-enhancement module, and a group of preliminary enhanced images with the best target detection accuracy are input into the image re-enhancement module for unsupervised re-enhancement for target detection tasks to obtain re-enhanced images.

6. The device according to claim 5, characterized in that The re-enhancement module is specifically used for: The first three layers of VGG pre-trained on ImageNet are used as encoders, and three layers of reversible residual blocks are used as decoders to jointly construct an image reconstruction network and design the loss function to obtain the image enhancement module. In the reversible residual block, the activation value of the current layer is calculated by the activation value of the next layer, and only the last layer activation value of each block needs to be stored.

7. The device according to claim 6, characterized in that The re-enhancement module is specifically used to design a loss function using the number of detection boxes and confidence scores after non-maximum suppression output by inference, and then add them to the cosine similarity loss to form a total loss function to guide image reconstruction so that the reconstructed image has higher detection accuracy in the target detection task.

8. The device according to claim 7, characterized in that The re-enhancement module is specifically used for: The total loss function is determined according to Formula 1, Formula 2 and Formula 3: loss ent = α1×m+α2×score Formula 2; Total loss function = loss sim +loss ent Formula 3; Among them, I en is an image that has been enhanced using the low-light enhancement module. It is the designed reconstruction module I en Reconstructed image, loss sim is the cosine similarity loss, m is the format of the detection box predicted by the detector, score is the predicted average confidence score, and α1 and α2 are two given parameters.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the unsupervised low-light image enhancement method for target detection as described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the unsupervised low-light image enhancement method for target detection as described in any one of claims 1 to 4 are implemented.

Images