An object detection method based on adaptive feature learning in low-light images

Abstract

The present invention relates to the technical field of image illumination enhancement, and specifically discloses a method for target detection based on adaptive feature learning of low-light images, the steps of which include inputting a target image into YOLOv5 and Zero_DCE respectively, obtaining features after removing illumination influence and target features, and fusing the two to obtain a result image; the fusion is as follows: the image processed by Zero_DCE is input into a backbone network for convolution to obtain a first intermediate result, and YOLOv5 convolves the image and upsamples it to obtain a second intermediate result, the first intermediate result and the second intermediate result have the same image size, and the two are fused. The present invention proposes a parallel target detection algorithm under low-light conditions, which uses an illumination enhancement network and a target detection network to encode image features in parallel, and the operation includes fusing illumination-related features with features for specific target detection to provide sufficient semantic information for subsequent detection tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of image illumination enhancement, and in particular to a target detection method for low-light image adaptive feature learning. Background Art

[0002] In the prior art, the problem of enhancing low-light images is mostly solved by machine learning methods, such as the method disclosed in the prior art patent (CN 115019340 A), which first selects the area to be enhanced based on the YoloV4 algorithm, and then performs light enhancement based on the Zero-DCE network. The aforementioned method processes the image to be processed serially, and the steps of obtaining image features and light enhancement are separated, which is prone to the following problems: since the image background and the target are not clearly distinguished, the light enhancement cannot be processed for the accurate target, resulting in poor results. Therefore, a technical solution that can solve the above problems is urgently needed to achieve a better target detection task. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides a target detection method for adaptive feature learning of low-light images, which makes the result better by processing the target detection algorithm and the illumination enhancement algorithm in parallel, overcomes the problem of sub-network inadaptability after the fusion of the two algorithms, and solves the problems mentioned in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions: a target detection method for adaptive feature learning of low-light images, wherein the target image is input into the target detection network YOLOv5 and the illumination enhancement network Zero_DCE respectively, and the features after removing the lighting influence and the target features are obtained, and the two types of features are fused to output the result image;

[0005] The fusion specifically includes: inputting the image processed by the illumination enhancement network into the backbone network for convolution to obtain a first intermediate result, and convolving and upsampling the target detection network image to obtain a second intermediate result; processing the first intermediate result and the second intermediate result to make the image sizes consistent (that is, the image size remains unchanged before and after the entire network processing), and fusing the images obtained by the first and second intermediate results to obtain a result image.

[0006] Preferably, the target detection network and the illumination enhancement network assign weights to the same processed image, dynamically adjust the learning weights between the two networks by detecting the reference model, and use a joint loss function to constrain the training of the two networks:

[0007] The target detection network loss functions are summed to obtain a joint loss function, which is:

[0008] Loss joint =Lossobject +Con2 / Con1·Loss ZeroDCE

[0009] Among them, Con1 represents the average confidence of a single image that has not been well trained in the parallel network, Con2 represents the average confidence of a single image in the pre-trained state, and Loss object is the target detection network loss function, Loss Zero_DCE Loss function for the illumination enhancement network.

[0010] Preferably, the first intermediate result is: the output result of the image after being processed by Zero_DCE and input into the shallow backbone network;

[0011] The second intermediate result is: the output result of the image after being processed by Zero_DCE and input into the deep backbone network and processed by the first C3 module.

[0012] Preferably, the size of the first intermediate result image is 80*80; the size of the second intermediate result image is 40*40.

[0013] Preferably, the fusion operation of fusing the images obtained from the first and second intermediate results is implemented in the Neck part of YOLOv5, and the fused results are output to the Head part for target detection and generation of bounding box coordinates to obtain the final result.

[0014] Preferably, the detection reference model realizes weight adjustment by constraining during the training process, and the pre-trained reference model obtains the relative confidence of the detection result;

[0015] The relative confidence is determined as a ratio of an average confidence of a single poorly trained image based on the parallel network to an average confidence of a single image in a pre-trained state.

[0016] Preferably, the target detection network loss function is:

[0017] Loss object =L loc +L conf +L class (1)

[0018] Class loss L loc and confidence loss L conf It is calculated using the binary cross entropy loss function, as shown in formulas (2) and (3):

[0019] L loc =1-GIoU (2)

[0020]

[0021]

[0022] Among them, k represents that the final network output feature map is divided into a grid of k×k cells, M represents the number of anchor boxes corresponding to each grid cell, represents the anchor box containing the object, and represents the anchor box that does not contain the object, l noobj Represents the confidence loss weight coefficient of the anchor box without an object, represents the predicted confidence of the target box, C represents the confidence of the actual box, and P represents the probability of the current category after activation. Indicates the actual value of the current category, which is 0 or 1.

[0023] Preferably, the Zero_DCE illumination enhancement loss function is:

[0024] Loss ZeroDCE =L spa +L exp +w1·L col +w2L ill (5)

[0025] Spatial consistency loss L spa The formula is as follows:

[0026]

[0027] Among them, w1 and w2 are the weights of color constancy loss and illumination smoothness loss, respectively, K represents the number of local regions, ω(i) represents the four adjacent regions centered on region i (from the top, bottom, left and right directions), and Y and I represent the average intensity values ​​of the local regions in the enhanced image and the input image, respectively:

[0028] Exposure control loss L exp The formula is as follows:

[0029]

[0030] Where Z represents the number of non-overlapping local regions of size 16×16, Y represents the average value, E is set to a constant of 0.6, and the exposure control loss is used to adjust the average light intensity value of each region in the enhanced image to make it close to the desired exposure level, thereby avoiding overexposure or underexposure;

[0031] Color constancy loss L col The formula is as follows:

[0032]

[0033] Among them, Jp represents the average intensity value of channel p in the enhanced image, J q represents the average intensity value of channel q in the enhanced image, (p,q) represents a pair of channels, and the color constancy loss function is used to ensure that the average color intensity of each channel in the enhanced image is close to the middle gray level, thereby correcting the color deviation;

[0034] Lighting smoothness loss L ill The formula is as follows:

[0035]

[0036] Where N is the number of iterations, A is the curve parameter diagram of Zero_DCE, x is the image channel, and Represent the horizontal and vertical gradient operations respectively. The lighting smoothness loss function is used to maintain the monotonicity of the curve parameter map between adjacent pixels and prevent the introduction of impurity textures.

[0037] The beneficial effects of the present invention are:

[0038] 1) Based on the problem of mutual isolation of sub-networks in the common two-stage model design, this paper proposes a parallel low-light target detection algorithm. First, the illumination enhancement network and the target detection network are used to encode image features in parallel. This operation includes fusing lighting-related features with features of specific target detection to provide sufficient semantic information for subsequent detection tasks;

[0039] 2) In order to enhance the feature fusion between the illumination enhancement network and the target detection network and improve the target detection performance under low-light conditions, the present invention introduces a "relative confidence weighted joint loss", uses this joint loss function to constrain network training, and realizes the design of a detection reference model; that is, using relative confidence (CON2 / CON1) as the method of illumination enhancement loss weight instead of simply relying on absolute confidence (CON1), can effectively prevent the network from disproportionately increasing the loss weight of samples that are easy to learn, thereby avoiding the bias in detecting simple samples, and effectively preventing the model from being biased towards learning objects that are easy to detect, thereby enhancing the performance and applicability of the model in various scenarios.

[0040] 3) In order to enhance the mutual adaptation between the two parallel illumination enhancement and object detection networks and achieve higher performance of object detection under low illumination, the confidence obtained by dividing the confidence con2 of the detection network by the confidence con1 of the standard detection network is designed as the weight of the enhancement loss function. This not only ensures the mutual adaptation between the networks, but also avoids the situation where the network tends to learn simple samples that are easier to detect when only the confidence con2 of the detection network is used as the loss weight. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the main framework of the network model in an embodiment of the present invention;

[0042] Figure 2 It is a generalized model framework diagram in an embodiment of the present invention;

[0043] Figure 3 1. is a comparison diagram of the target detection method of low-light image adaptive feature learning according to an embodiment of the present invention and the prior art;

[0044] Figure 4 This is a data comparison diagram of the target detection method for low-light image adaptive feature learning according to an embodiment of the present invention and the prior art. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Example 1

[0047] In order to solve the problem of limited relevance and low processing efficiency of the traditional two-stage network when combining illumination enhancement and object detection, the present invention proposes the following Figure 1 The network model shown in Figure 1. The illumination-enhanced Zero_DCE network is combined with the YOLOv5 network for target detection. This design uses a parallel integrated network framework, which not only improves network efficiency, but also enhances the correlation between the two models, and promotes the mutual fusion and adaptation of features between the illumination-enhanced network and the target detection network. Therefore, the target detection network can make full use of the illumination-enhanced network to achieve excellent performance.

[0048] YOLOv5 is one of the widely used object detection algorithms in the YOLO series, which achieves a better balance between detection speed and accuracy. It is one of the commonly used object detection methods today. Its basic principle regards object detection as a regression problem of positioning and classification. It uses an end-to-end convolutional neural network to directly predict the positioning and classification probabilities of objects in the input image. Its main architecture includes a backbone network, a neck network, and a head module. YOLOv5 introduces a focus mechanism in the backbone network for slicing the input image, thereby reducing the number of parameters of the network. The neck network is located between the backbone network and the head module and adopts a path aggregation network (PAN) structure. This structure helps to fuse the features extracted by the backbone network at multiple scales, thereby enhancing the feature expression capability. In the head module, the network uses the features obtained from the neck network for multi-scale detection. These detection results help generate the final detection results of the image.

[0049] Zero_DCE is a zero-reference deep learning algorithm for enhancing low-light images. Unlike existing methods that regard image enhancement as an image-to-image mapping, Zero_DCE regards it as a parameter mapping problem of a specific curve, customizing the curve for each image. Specifically, this method designs an illumination enhancement curve that is able to perform pixel-level adaptive adjustments to the image. In order to learn the mapping relationship between the input image and the optimal curve, a lightweight deep network called DCE_Net is proposed. Importantly, in order to achieve no-reference learning, this method introduces a series of differentiable no-reference loss functions that implicitly evaluate the enhancement effect. These designs enable Zero_DCE to perform unsupervised learning, eliminating the need for paired training data and avoiding the overfitting problem. Therefore, it exhibits strong generalization ability under different lighting conditions.

[0050] The parallel training method using Zero_DCE and YOLOv5 takes full advantage of Zero_DCE's unsupervised learning method, which eliminates the need for paired datasets. This makes it possible to train the integrated network using existing open source datasets.

[0051] In order to improve the efficiency of adaptive learning and prevent the model from biased towards learning simple samples in target detection tasks, the present invention discloses a new detection reference model, which dynamically adjusts the learning weights of the two networks, thereby enhancing the network's ability to encode target-related information under low-light conditions.

[0052] The detection reference network provides confidence scores through a pre-trained object detection network, which, when combined with the confidence scores of the parallel network, form a relative confidence score. This relative confidence score is used to control the weights of the joint loss function, enabling joint training of the entire network and strengthening the correlation between the two parallel networks.

[0053] The original YOLOv5 loss function is expressed by formula (1), which includes three parts: positioning loss, confidence loss and category loss. Among them, the positioning loss is shown in formula (2), which is obtained by subtracting the generalized intersection and union of the predicted box and the standard box from 1.

[0054] Loss object =L loc +L conf +L class (1)

[0055] L loc =1-GIoU (2)

[0056]

[0057]

[0058] The category loss and confidence loss are calculated using the binary cross entropy loss function, as shown in equations (2) and (3). In the above two equations, k represents that the final network output feature map is divided into a grid of k×k cells. M represents the number of anchor boxes corresponding to each grid cell. represents the anchor box containing the object, and Indicates an anchor box that does not contain an object. noobj Represents the confidence loss weight coefficient of the anchor box without an object. represents the predicted confidence of the target box, C represents the confidence of the actual box. P represents the probability of the current category after activation, Indicates the actual value of the current category (0 or 1).

[0059] The loss function consisting of three independent loss terms effectively constrains the object detection network and achieves relatively fast convergence during training.

[0060] The enhancement loss function of Zero_DCE consists of four parts: spatial consistency loss, exposure control loss, color constancy loss, and illumination smoothness loss. As shown in Formula 5, w1 and w2 are the weights of color constancy loss and illumination smoothness loss, which are set to 1 and 0.5 respectively. The components of the enhancement loss function of Zero_DCE are presented in Formulas (6) to (9).

[0061] Loss ZeroDCE =L spa +L exp +w1·L col +w2L ill (5)

[0062] Spatial consistency loss L spaThe formula of is as follows, which aims to preserve the difference between the input image and the enhanced image in adjacent regions. In the formula, K represents the number of local regions, and ω(i) represents the four adjacent regions centered on region i (from the top, bottom, left, and right directions). Y and I represent the average intensity values ​​of the local regions in the enhanced image and the input image, respectively.

[0063]

[0064] The definition of exposure control loss is shown in formula (7), where Z represents the number of non-overlapping local regions of size 16×16, Y represents the average value, and E is set to a constant of 0.6. The exposure control loss aims to adjust the average light intensity value of each region in the enhanced image to make it close to the desired exposure level, thereby avoiding overexposure or underexposure.

[0065]

[0066] The color constancy loss is defined as shown in formula (8), where J p represents the average intensity value of channel p in the enhanced image, J q represents the average intensity value of channel q in the enhanced image, and (p,q) represents a pair of channels. This loss function is designed to ensure that the average color intensity of each channel in the enhanced image is close to the mid-gray level, thereby correcting the color deviation.

[0067]

[0068] The lighting smoothness loss is shown in formula (9), where N represents the number of iterations, A is the curve parameter map of Zero_DCE, x is the image channel, and Represent the horizontal and vertical gradient operations respectively. This loss function is used to maintain the monotonicity of the curve parameter map between adjacent pixels and prevent the introduction of impurity textures.

[0069]

[0070] In order to enhance the feature fusion between the illumination enhancement network and the target detection network and improve the target detection performance under low light conditions, the relative confidence weight joint loss is introduced. This joint loss function is used to constrain network training and realize the design of the detection reference model. The definition of this joint loss function is shown in Equation 10:

[0071] Loss joint =Loss object +Con2 / Con1·Loss ZeroDCE (10)

[0072] Among them, Con1 represents the average confidence of a single image that has not been well trained in the parallel network, and Con2 represents the average confidence of a single image in the pre-trained state (such as Figure 1 This design effectively prevents the model from biased towards learning objects that are easy to detect, thereby enhancing the performance and applicability of the model in various scenarios.

[0073] In this embodiment, the method of using relative confidence (CON2 / CON1) as the lighting enhancement loss weight, rather than simply relying on absolute confidence (CON1), can effectively prevent the network from disproportionately increasing the loss weight of samples that are easy to learn, thereby avoiding the bias in detecting simple samples, and effectively preventing the model from being biased towards learning objects that are easy to detect, thereby enhancing the performance and applicability of the model in various scenarios.

[0074] In this embodiment, the loss function Loss of the target detection network object And the illumination enhancement function Loss ZeroDCE In order to enhance the mutual adaptation of the two parallel networks and improve the detection performance of the target detection network under low-light conditions, the relative confidence of Con2 / Con1 is used as the weight of the illumination enhancement loss. Among them, Con2 is the average confidence of a single image of the parallel target detection network, and Con1 is the relative confidence of the pre-trained standard target detection network. Using relative confidence as a weight can make the features extracted by the enhanced illumination enhancement network more suitable for the target detection network. And the design of relative confidence can avoid the situation when only a single confidence (Con2) is used. When facing relatively simple samples during network training, Con2 will be larger, resulting in a larger loss weight for simple samples, which makes the network tend to learn simple sample features. Therefore, the joint loss function designed by Sun will increase the illumination enhancement network to be more adaptable to target detection tasks when constraining network training, and avoid the tendency of the network to learn simple samples.

[0075] Comparative Verification

[0076] like Figure 3 As shown, using the above-mentioned parallel network ( Figure 3 (f)) and only the detection network ( Figure 3 (a)) and other serial networks (light enhancement followed by target detection, Figure 3 (b) to Figure 3 Visual display of the detection results in (e).

[0077] It can be seen that the separate target detection network has missed detections, and the results in the serial network also have varying degrees of missed detections, false detections, or confidence levels lower than our method.

[0078] like Figure 4As shown, the method of the present application is compared with the objective performance of the detection network only (yolov5) and the serial network when performing detection tasks on the ExDark dataset. It can be seen that the model of the present invention can effectively improve the performance of the target detection network under low light conditions, and proves that the performance of the proposed parallel network is better than that of the two-stage serial network.

[0079] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for object detection based on adaptive feature learning in low-light images, characterized in that: The target image is input into the target detection network YOLOv5 and the illumination enhancement network Zero_DCE respectively to obtain the features after removing the lighting influence and the target features. The two types of features are fused and the result image is output. The fusion specifically includes: inputting the image processed by the illumination enhancement network into the backbone network for convolution to obtain a first intermediate result, and convolving and upsampling the image processed by the target detection network to obtain a second intermediate result; processing the first intermediate result and the second intermediate result so that the image sizes are consistent, and fusing the images obtained by the first and second intermediate results to obtain a result image; The target detection network and the illumination enhancement network assign weights to the same processed image, dynamically adjust the learning weights between the two networks by detecting the reference model, and use a joint loss function to constrain the training of the two networks: The target detection network loss functions are summed to obtain a joint loss function, which is: Loss joint =Loss object +Con2 / Con1·Loss ZeroDCE Among them, Con1 represents the average confidence of a single image that has not been well trained in the parallel network, Con2 represents the average confidence of a single image in the pre-trained state, and Loss object is the target detection network loss function, Loss Zero_DCE Enhance the network loss function for illumination; The first intermediate result is: the output result of the image after being processed by Zero_DCE and input into the shallow backbone network; The second intermediate result is: the output result of the image after being processed by the target detection network and then input into the deep backbone network and processed by the first C3 module.

2. The object detection method for low-light image adaptive feature learning according to claim 1, characterized in that: The size of the first intermediate result image is 80*80; the size of the second intermediate result image is 40*40.

3. The object detection method for low-light image adaptive feature learning according to claim 1, characterized in that: The fusion operation of fusing the images obtained from the first and second intermediate results is implemented in the Neck part of YOLOv5, and the fused results are output to the Head part for target detection and generation of bounding box coordinates to obtain the final result.

4. The object detection method for low-light image adaptive feature learning according to claim 1, characterized in that: The detection reference model implements weight adjustment by constraining during the training process, and the pre-trained reference model obtains the relative confidence of the detection result; The relative confidence is determined as a ratio of an average confidence of a single poorly trained image based on the parallel network to an average confidence of a single image in a pre-trained state.

5. The object detection method for low-light image adaptive feature learning according to claim 1, characterized in that: The target detection network loss function is: Loss object =L loc +L conf +L class (1) Class loss L loc and confidence loss L conf It is calculated using the binary cross entropy loss function, as shown in formulas (2) and (3): L loc =1-GIoU (2) Among them, k represents that the final network output feature map is divided into a grid of k×k cells, M represents the number of anchor boxes corresponding to each grid cell, represents the anchor box containing the object, and represents the anchor box that does not contain the object, λ noobj Represents the confidence loss weight coefficient of the anchor box without an object, represents the predicted confidence of the target box, C represents the confidence of the actual box, and P represents the probability of the current category after activation. Indicates the actual value of the current category, which is 0 or 1.

6. The object detection method for low-light image adaptive feature learning according to claim 1, characterized in that: Zero_DCE illumination enhancement loss function is: Loss ZeroDCE =L spa +L exp +w1·L col +w2L ill (5) Spatial consistency loss L spa The formula is as follows: Among them, w1 and w2 are the weights of color constancy loss and illumination smoothness loss, respectively, K represents the number of local regions, ω(i) represents the four adjacent regions centered on region i, and Y and I represent the average intensity values ​​of the local regions in the enhanced image and the input image, respectively: Exposure control loss L exp The formula is as follows: Where Z represents the number of non-overlapping local regions of size 16×16, E is set to a constant of 0.6, and the exposure control loss is used to adjust the average light intensity value of each region in the enhanced image to make it close to the desired exposure level, thereby avoiding overexposure or underexposure; Color constancy loss L col The formula is as follows: Among them, J p represents the average intensity value of channel p in the enhanced image, J q represents the average intensity value of channel q in the enhanced image; (p, q) represents a pair of channels, and the color constancy loss function is used to ensure that the average color intensity of each channel in the enhanced image is close to the middle gray level, thereby correcting the color deviation; Lighting smoothness loss L ill The formula is as follows: in, N represents the number of iterations, A is the curve parameter diagram of Zero_DCE, ξ is the image channel, and Represent the horizontal and vertical gradient operations respectively. The lighting smoothness loss function is used to maintain the monotonicity of the curve parameter map between adjacent pixels and prevent the introduction of impurity textures.

Images