A dark target detection method and system based on dark transform feature equivalence

Abstract

The present application relates to the technical field of target detection, and proposes a dark target detection method and system based on feature isometry of dark transform, which comprises the following steps: inputting a normal light image into a physical noise model obtained by analyzing and modeling physical sensor noise generated in an imaging process to obtain a low light image subjected to dark transform; inputting the normal light image and the low light image into a detector, wherein the detector is configured with a feature isometry conversion network, which is used to generate a first feature vector corresponding to the normal light image and the low light image, and a second feature vector containing nonlinear information converted from the first feature vector; determining a training loss of the detector by using a consistency loss for feature isometry determined according to the first feature vector and the second feature vector, optimizing the training loss, and updating the detector through the optimized training loss; and inputting an image to be detected into the updated detector to obtain a dark target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and more particularly to a dark target detection method and system based on dark transform feature isometry. BACKGROUND

[0002] Target detection is an important task in computer vision, which aims to automatically identify and locate the position and category of objects in images or videos, and it is widely used in medical image analysis, autonomous driving and object recognition. With the rapid development of convolutional neural networks, target detection methods based on convolutional networks such as Faster R-CNN, YOLO, etc. have become classic algorithms in the field of target detection. However, the good performance of these algorithms is only limited to the scene of normal light, because in poor lighting conditions, the representation of the edge, texture, color and other detailed features of the target will be affected, and using these target detection algorithms cannot extract sufficient detailed features, which will cause problems such as insufficient accuracy of target detection results, increased false detection rate and missed detection rate.

[0003] In order to solve this problem, there are two aspects of research at present:

[0004] (1) Research on enhancing low-light images:

[0005] The purpose of its research is to improve the image quality in low-light scenes, so that the image is suitable for the target detection algorithm for detecting normal light images, thereby improving the effect of target detection; the main goal of low-light enhancement method is to reduce noise and increase contrast, and to present more details and information of low-light images. However, the low-light enhancement method is usually used as a preprocessing stage of the detector, which not only consumes time, but also may destroy the information of the original input image; because the low-light enhancement method improves the overall visual effect of the target image, but in the process of low-light enhancement, the problem of over-enhancement or a large amount of noise interference generated in the enhancement process is easy to occur, which causes the low-light image to lose the original target detailed features.

[0006] (2) Research on domain adaptation:

[0007] The purpose of its research is to generate synthetic low-light images by learning the distribution of low-light scenes, and to train the detector using the synthetic low-light images. However, this adaptive method is not enough to solve the huge semantic information gap caused by the dramatic change of light, which causes the generated low-light images to have a gap with the actual low-light images corresponding to the scene, and the effect of target detection using this method is not ideal.

[0008] In summary, the existing methods improve the dark target detection algorithm from the perspective of reconstructing image data, do not explore the internal relationship between normal light and low-light images, and have the disadvantage that reconstructing image data causes the loss of original features of the image, thereby affecting the effect of target detection. SUMMARY

[0009] The present application provides a dark target detection method and system based on dark transform feature isometry to overcome the defects of the prior art that the reconstructed image data causes the loss of original features of the image, thereby affecting the target detection effect.

[0010] To solve the above technical problems, the technical scheme of the present application is as follows:

[0011] A dark target detection method based on dark transform feature isometry, characterized in that it comprises the following steps:

[0012] S1: analyzing and modeling the physical sensor noise generated in the imaging process to obtain a physical noise model; inputting a normal light image into the physical noise model to obtain a low light image subjected to dark transform;

[0013] S2: inputting the normal light image and the low light image into a detector for dark target detection, wherein the detector is configured with a feature isometry conversion network for generating a first feature vector corresponding to the normal light image and the low light image, and a second feature vector containing nonlinear information converted from the first feature vector;

[0014] S3: determining a consistency loss for feature isometry according to the first feature vector and the second feature vector, and determining a training loss of the detector using the consistency loss, optimizing the training loss, and updating the detector through the optimized training loss;

[0015] S4: inputting an image to be detected into the updated detector to obtain a dark target detection result.

[0016] Preferably, the present application further provides a dark target detection system based on dark transform feature isometry applied to the dark target detection method based on dark transform feature isometry described above. The dark target detection system based on dark transform feature isometry comprises a preprocessing module, a feature extraction module, and a detection module connected in sequence.

[0017] In the technical scheme, the preprocessing module is used to obtain a normal light image dataset and generate a low light image dataset using the normal light image dataset;

[0018] The feature extraction module is used to train a neural network through the normal light image and low light image datasets, input an image to be detected into the trained neural network, and extract features of the image to be detected;

[0019] The detection module is used to output a detection result by inputting the features of the image to be detected.

[0020] Compared with the prior art, the beneficial effects of the technical scheme of the present application are:

[0021] According to the imaging principle of the camera, the physical sensor noise generated in the imaging process is analyzed and modeled, which can truly simulate the imaging process in the actual scene, so as to generate a corresponding realistic low-light image through dark transformation of the normal light image, and then learn the invariant features of the images in the low-light environment and the normal light environment by using feature equivariance, so that the detector can explore the internal relationship between the normal light and weak light images, capture a more compact representation of the dark image, and improve the effect of target detection.

[0022] At the same time, since the present method focuses on constraining the detector to learn more light-dark invariant features, rather than improving the performance of the detector through additional modules or special structure design, the present method is suitable for various mainstream target detectors, does not affect the good performance of the target detector in detecting normal light images, and can maintain superior performance in both normal light environment and low-light environment. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 Flowchart of the dark target detection method based on dark transformation feature equivariance of embodiment 1.

[0024] Figure 2 Flowchart of the dark target detection method based on dark transformation feature equivariance applied to the YOLO detector of embodiment 2.

[0025] Figure 3 Overall framework diagram of the dark target detection system based on dark transformation feature equivariance of embodiment 3. DETAILED DESCRIPTION

[0026] The drawings are only used for illustrative description and cannot be understood as a limitation of the present patent;

[0027] In order to better illustrate the present embodiment, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product;

[0028] For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0029] The technical scheme of the present application will be further described below in combination with the drawings and embodiments.

[0030] Embodiment 1

[0031] The present embodiment proposes a dark target detection method and system based on dark transformation feature equivariance, as shown in Figure 1 Flowchart of the dark target detection method based on dark transformation feature equivariance of the present embodiment.

[0032] The dark target detection method based on feature equivariance of dark transformation provided in the embodiment comprises the following steps:

[0033] S1: analyzing and modeling the physical sensor noise generated in the imaging process to obtain a physical noise model; inputting a normal light image into the physical noise model to obtain a low light image subjected to dark transformation;

[0034] S2: inputting the normal light image and the low light image into a detector for dark target detection, wherein the detector is configured with a feature equivariance conversion network for generating a first feature vector corresponding to the normal light image and the low light image and a second feature vector containing nonlinear information converted from the first feature vector;

[0035] S3: determining a consistency loss for feature equivariance according to the first feature vector and the second feature vector, determining a training loss of the detector by using the consistency loss, optimizing the training loss, and updating the detector by using the optimized training loss;

[0036] S4: inputting an image to be detected into the updated detector to obtain a dark target detection result.

[0037] In the specific implementation process, the imaging process in the actual scene is more realistically simulated by analyzing and modeling the physical sensor noise generated in the imaging process; the normal light image is subjected to dark transformation processing by using the physical sensor noise to generate a realistic low light image; the normal light image and the corresponding low light image generated therefrom are used as training data of the detector to optimize the training loss of the detector; and the image to be detected is input into the optimized detector to output the detection result.

[0038] The feature equivariance conversion network configured in the detector is used to learn the invariant features of images in low light environment and normal light environment, so that the detector can explore the internal relationship between normal light images and low light images, capture a more compact representation of dark images, and improve the target detection effect; meanwhile, the good performance of the detector in detecting normal light images is not affected, so that the detector can maintain superior performance in normal light environment and low light environment.

[0039] In an optional embodiment, the S1 step comprises:

[0040] S1.1: obtaining simulated lens noise by adding random noise subject to Poisson distribution; wherein the lens noise δ s The calculation formula of the lens noise δ

[0041]

[0042] wherein, S is the light signal collected by the camera;

[0043] S1.2: Generate random noise subject to Gaussian distribution with zero mean and fixed variance, and superimpose it with the original image to obtain the simulated read noise;

[0044] wherein the read noise δ r is subject to Gaussian distribution, and the expression thereof includes:

[0045] δ r ~N(0, 1);

[0046] S1.3: Use constant w, lens noise δ s and read noise δ r to constitute a physical noise model x noise ; wherein the expression of x noise includes:

[0047]

[0048] wherein the constant w represents the factor of linear light decay, and x represents the pixel true value of the input image;

[0049] S1.4: Input the constant light image into the physical noise model to obtain the low light image xD after dark transformation:

[0050] x D = wx+x noise .

[0051] In the specific implementation process, the pixel value of the input image is linearly attenuated by a constant w.

[0052] wherein optionally, in the step S1.3, the constant w is randomly sampled in the range of [0.01, 1.0].

[0053] In the embodiment, the lens noise and the read noise are the main noises that occur when the photons pass through the lens to reach the camera sensor and are converted into an analog voltage signal during the imaging process; wherein the lens noise is caused by the uncertainty of the photons captured in the camera, i.e. the fluctuation of the photons; due to the very limited number of photons, random fluctuations occur during the capture process, and such random fluctuations appear as irregular ripples in brightness on the image, i.e. lens noise; in the specific implementation process, the number of photons of each pixel point is regarded as an independent event, and it is assumed that the occurrence probability of each photon is equal, and the Poisson distribution is used to describe the distribution of the lens noise.

[0054] The read noise is the noise that occurs when the electrons are converted into digital signals through the preamplifier; the size of the read noise is irrelevant to any light signal.

[0055] In an optional embodiment, the expression of the consistency loss for feature isometry in the S2 step comprises:

[0056]

[0057] wherein p and p D are the first feature vectors corresponding to the normal light image and the low light image respectively, z and z D are the second feature vectors obtained after the conversion of p and p D respectively, and cos(·,·) represents the cosine similarity function, and detach(·) is the gradient stop operation.

[0058] Optimizing the consistency loss, i.e., making the consistency loss smaller, can make the cosine similarity between the first feature vector and the second feature vector larger, i.e., can make the feature similarity between the normal light image and the low light image larger; minimizing the consistency loss, i.e., can achieve the purpose of feature isometry.

[0059] In an optional embodiment, the specific process of the S3 step comprises:

[0060] S3.1: adding a supervised detection loss L S , combining the supervised detection loss L S with the consistency loss L DTE to determine the training loss of the detector; wherein the expression of the supervised detection loss L S comprises:

[0061] L S = L box + L cla + L obj

[0062] wherein L box , L cls and L obj represent the position loss, the classification loss and the confidence loss respectively;

[0063] The expression of the training loss of the detector comprises:

[0064] L = L S + a·L DTE

[0065] wherein a is a balance factor of a preset value;

[0066] S3.2: using the stochastic gradient descent method to optimize the training loss, updating the detector with the parameters that minimize the training loss to obtain the updated detector.

[0067] In this embodiment, the consistency loss and the supervised detection loss are simultaneously optimized. By optimizing the consistency loss, the feature equivariance is achieved, the feature distance between the normal light image and the low light image is minimized, the detector can explore the internal relationship between the normal light image and the low light image, capture a more compact representation of the dark image, and improve the target detection effect; by optimizing the supervised detection loss, the detection ability of the detector is improved.

[0068] In an optional embodiment, the feature equivariance conversion network comprises a backbone neural network, a projection layer and a prediction layer; wherein the backbone neural network is used for feature extraction, the projection layer is used for mapping the first feature vector to the same feature space, and the prediction layer is used for converting the first feature vector mapped to the same feature space to obtain the second feature vector containing nonlinear information.

[0069] Optionally, the backbone neural network comprises DarkNet-53.

[0070] Further optionally, the projection layer comprises a projection MLP head, and the prediction layer comprises a prediction MLP head.

[0071] Embodiment 2

[0072] This embodiment applies the dark target detection method based on dark transformation feature equivariance proposed in embodiment 1, and proposes the following performance comparison examples.

[0073] As shown in Figure 2 , it is a flowchart of applying the dark target detection method based on dark transformation feature equivariance to the YOLO detector in this embodiment.

[0074] The dark target detection method based on dark transformation feature equivariance is compared with other excellent dark target detection methods on the ExDark dataset, and the other excellent dark target detection methods include MBLLEN (multi-branch low-light enhancement network model) and Zero-DCE (Zero-Reference Deep Curve Estimation, zero-reference deep curve estimation).

[0075] When comparing, the dark target detection method based on dark transformation feature equivariance is instantiated on the YOLOv3 detector together with other excellent dark target detection methods; wherein the YOLOv3 detector has a DarkNet-53 backbone neural network and a YOLO detection head.

[0076] The quantitative experimental results (AP) and the total average results (mAP) of this embodiment are shown in Table 1.

[0077] As can be seen from Table 1, the method proposed in the present application achieves the best results in most categories compared with other methods. Moreover, the method proposed in the present application significantly improves the basic detector YOLOv3, achieving a 3.4% mAP improvement, which shows that after applying the method proposed in the present application, the detector can learn more advantageous information for dark object detection, and the results further prove that learning dark transform and other variant features is beneficial to improving the feature learning ability of the detector for low-light images.

[0078] Table 1 Comparison of results of various methods

[0079]

[0080]

[0081] Embodiment 3

[0082] The present embodiment proposes a dark target detection system based on dark transform feature invariance.

[0083] As Figure 3 shown, it is the overall framework diagram of the dark target detection system based on dark transform feature invariance of the present embodiment.

[0084] The dark target detection system based on dark transform feature invariance comprises:

[0085] The preprocessing module is configured to acquire a normal-light image dataset and generate a low-light image dataset using the normal-light image dataset.

[0086] The feature extraction module is configured to train a neural network through the normal-light image dataset and the low-light image dataset, input an image to be detected into the trained neural network, and extract features of the image to be detected.

[0087] The detection module is configured to input the features of the image to be detected and output a detection result.

[0088] Optionally, the dark target detection system based on dark transform feature invariance further comprises a visualization module configured to visually display the detection result output by the detection module, so that a user can intuitively know the detection result and conveniently operate the system.

[0089] The same or similar reference numerals correspond to the same or similar components;

[0090] The terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present patent;

[0091] Obviously, the above embodiments of the present application are merely exemplary but not intended to limit the embodiments of the present application. Based on the above description, any other variations or changes can be made by those skilled in the art without departing from the spirit and principles of the present application. It is not necessary to list all the embodiments here. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall fall within the scope of the claims of the present application.

Claims

1. A method for detecting dark targets based on dark transform feature equivariance, characterized in that, Includes the following steps: S1: Analyze and model the physical sensor noise generated during the imaging process to obtain a physical noise model; input the normal light image into the physical noise model to obtain a low-light image after dark transformation; S2: Input the normal light image and the low light image into a detector for dark target detection, wherein the detector is configured with a feature equivariance transformation network for generating a first feature vector corresponding to the normal light image and the low light image, and a second feature vector containing nonlinear information obtained by transforming the first feature vector. S3: Determine the consistency loss for feature equivariance based on the first feature vector and the second feature vector, and use the consistency loss to determine the training loss of the detector, optimize the training loss, and update the detector using the optimized training loss; S4: Input the image to be detected into the updated detector to obtain the dark target detection result; In step S2, the expression for the consistency loss used for feature equivariance includes: in, and These are the first feature vectors corresponding to the normal light image and the low light image, respectively. and They are respectively and The second feature vector obtained after transformation Represents the cosine similarity function. It is gradient stopping computation; The specific process of step S3 includes: S3.1: Add supervised detection loss Combined with monitoring and detection of losses With consistency loss Determine the training loss of the detector; wherein, the supervised detection loss The expressions include: in, and These represent location loss, classification loss, and confidence loss, respectively. The training loss expression for the detector includes: in, The balance factor is a preset value; S3.2: Optimize the training loss using stochastic gradient descent, and update the detector using the parameters that minimize the training loss to obtain the updated detector; The feature equivariance transformation network includes a backbone neural network, a projection layer, and a prediction layer; wherein, the backbone neural network is used for feature extraction, the projection layer is used to map the first feature vector to the same feature space, and the prediction layer is used to transform the first feature vector mapped to the same feature space to obtain a second feature vector containing nonlinear information. The backbone neural network includes DarkNet-53; The projection layer includes a projection MLP head, and the prediction layer includes a prediction MLP head.

2. The dark target detection method based on dark transform feature equivariance according to claim 1, characterized in that, Step S1 includes: S1.1: By adding random noise that follows a Poisson distribution, simulated lens noise is obtained; where, lens noise The calculation formulas include: Where S is the light signal collected by the camera; S1.2: Generate random noise that follows a Gaussian distribution with zero mean and fixed variance, and superimpose it on the original image to obtain simulated reading noise; Among them, reading noise It follows a Gaussian distribution, and its expression includes: ； S1.3: Using constants Lens noise and reading noise Composition of physical noise model ;in, The expressions include: Where, constant The factor representing linear precipitation. Represents the true pixel values ​​of the input image; S1.4: Input the normal light image into the physical noise model to obtain the low-light image after dark transformation. : 。 3. The dark target detection method based on dark transform feature equivariance according to claim 2, characterized in that, In step S1.3, the constant Random sampling is performed within the range of [0.01, 1.0].

4. A dark target detection system based on dark transform feature equivalence, employing the dark target detection method based on dark transform feature equivalence as described in any one of claims 1 to 3, characterized in that, include: The preprocessing module is used to acquire the normal light image dataset and generate the low light image dataset using the normal light image dataset; The feature extraction module is used to train a neural network using normal light and low light image datasets, input the image to be detected into the trained neural network, and extract the features of the image to be detected. The detection module is used to output detection results by taking the features of the image to be detected as input.

5. The dark target detection system based on dark transform feature equivariance according to claim 4, characterized in that, It also includes a visualization module for visually displaying the detection results output by the detection module.

Images