An all-weather vehicle detection method based on one-to-many adversarial network

Abstract

The present invention discloses an all-weather vehicle detection method based on a one-to-many adversarial network. Considering the characteristic that the night scene is not single due to the light source distribution and the difference in light intensity, the gray histogram distribution and the clustering algorithm are used to realize the grading of the light intensity of natural light and background light in the night scene; the purpose of proposing this one-to-many adversarial network is to convert a daytime image into night images under different lighting environments according to the set natural light and background light intensities, and use the daytime image carrying label information and the synthesized night images to jointly train the vehicle detection model, effectively alleviating the problem that the existing all-weather vehicle detection technology methods have insufficient generalization ability in the night environment with different lighting due to the scarcity of night images with label information or the lack of diversity of synthesized night images.

Description

Technical Field

[0001] The present invention relates to the field of vehicle monitoring, and particularly to an all-weather vehicle detection method based on a one-to-many adversarial network Background Technique

[0002] The vehicle detection task based on monitoring is the perception basis for many tasks in the field of transportation engineering (such as traffic parameter extraction, congestion discrimination, etc.), and its research has important practical significance. In recent years, vehicle detection methods based on deep learning have gradually become the mainstream due to advantages such as high accuracy and strong robustness. However, due to the data-driven nature, deep learning methods require a large amount of high-quality data as input to achieve training convergence. At present, there are a large number of daytime scene datasets with annotation information. The vehicle detectors trained based on this dataset have high detection accuracy in daytime scenes, but low detection accuracy in night scenes. Due to the lack of corresponding night scene datasets, the detection effects of current mainstream models in night scenes are not ideal

[0003] Recently, the research "Domain adaptation from daytime to nighttime: A situation-sensitive vehicle detection and traffic flow parameter estimation framework" uses the generative adversarial network CycleGAN to convert daytime pictures with annotation information into synthetic night pictures; then, uses the synthetic night pictures to train the Faster R-CNN model to achieve vehicle detection in night scenes. This method avoids large-scale annotation of night pictures and effectively saves human resources, but does not consider the diversity of night scenes, resulting in the model trained based on synthetic night pictures being difficult to meet the vehicle detection tasks under different lighting conditions at night. Similarly, the research "GAN-Based Day-to-Night Image Style Transfer for Nighttime Vehicle Detection" proposes a generative adversarial network model AugGAN that realizes one-to-one mapping from daytime pictures to night pictures. This network can convert individual daytime pictures into real high-quality synthetic night pictures, but lacks consideration of the diversity of night scene lighting (intensity of natural light and background light)

[0004] Due to the intensity influence of natural light (moonlight) and background light (headlights, streetlights, etc.), different night stages of the same scene should present different contrasts and brightness. The main drawback of the existing technology is that it does not consider the diversity of night scenes. The generative adversarial network models constructed by these methods can only achieve a one-to-one mapping from day to night, resulting in the generated night pictures lacking diversity, thereby affecting the accuracy of vehicle detection in night scenes. Summary of the Invention

[0005] To solve the technical problems mentioned in the above background art, the present invention proposes an all-weather vehicle detection method based on a one-to-many adversarial network.

[0006] To achieve the above technical objectives, the technical solution of the present invention is as follows:

[0007] An all-weather vehicle detection method based on a one-to-many adversarial network, comprising the following steps:

[0008] S1. Use the sky segmentation algorithm to segment the collected night surveillance pictures into a sky region S and a non-sky region NS;

[0009] S2. Obtain the gray-scale histogram distributions G S and G NS of the sky region S and the non-sky region NS, and construct low-dimensional feature vectors f S and f NS from the gray-scale histogram distributions G S and f NS ;

[0010] S3. Use the clustering algorithm to cluster the low-dimensional feature vectors f S and f NS , and respectively obtain the illumination vectors of natural light and background light of all night pictures according to the clustering results;

[0011] S4. Construct a one-to-many adversarial network, and build a joint loss function based on the illumination vectors of natural light and background light obtained from the night pictures to train the one-to-many adversarial network;

[0012] S5. The trained one-to-many adversarial network takes day pictures and adjustable illumination vectors as inputs, and generates night synthetic pictures under different natural light and background light conditions;

[0013] S6. Use the day pictures and the generated night synthetic pictures to jointly train the vehicle detection model;

[0014] S7. Import the trained vehicle detection model into the roadside computing device or the cloud computing center to perform the vehicle detection task for the entire period under roadside monitoring.

[0015] Preferably, step 2 specifically includes:

[0016] Gray-scale the sky region S and the non-sky region NS, and count the number of pixels with gray values in the range [0, 255] to form a gray-scale histogram distribution G S and G NS ; Divide the gray-scale pixel range [0, 255] into 10 equally-spaced intervals, and count the number of pixel values falling into each interval from the distribution results of G S and G NS to obtain a gray-scale distribution histogram g S and g NS ; Count the gray-scale distribution histogram g S and g NS in all night data, denoted as the g S set and the g NS set; Calculate the mean and variance of the g S set and the g NS set according to the statistical results, and use the obtained mean and variance to standardize the data of g S and g NS as follows:

[0017]

[0018]

[0019] In the formula: and are the means of the g S set and the g NS set respectively, and are the variances of the g S set and the g NS set respectively; f S and f NS are low-dimensional feature vectors.

[0020] Preferably, step 3 specifically includes:

[0021] S31. Use the DBSCAN clustering algorithm to cluster the obtained low-dimensional feature vectors f S and f NS ;

[0022] S32. Use the elbow method to determine the optimal number of clusters for the clustering algorithm;

[0023] S33. Visualize the night pictures of each cluster, and assign illumination labels according to the light intensity of the night pictures in the cluster, where natural light is divided into three levels; background light is divided into five levels;

[0024] S34. Generate unique codes for natural light and background light according to the assigned levels, which are the illumination labels.

[0025] Preferably, step 4 specifically includes:

[0026] S41. Construct a day-to-night image generation network GD 2N , the purpose of this network is to convert a day image d into multiple night images n under different lighting environments according to the set natural light and background light intensity z;

[0027] S42. Construct a night-to-day image generation network G N2D , this network converts all generated night images n into reconstructed day images

[0028] S43. Construct a cyclic consistency loss function to make the input day image d and the reconstructed day image close enough, thereby empowering the generation network G D2N to generate night images n that are consistent with the content of the input day image d: This cyclic consistency loss function L c is defined as follows:

[0029]

[0030]

[0031] In the formula: D is the day image dataset, Z is the selection space of natural light and background light intensity; G D2N (d|z) is the night image generated with the lighting adjustment vector z and the day image d as inputs; E d～D,z～Z is the expectation of the outputs for all day images d and all lighting vectors z;

[0032] S44. To make the night images generated by the generation network G D2N real enough and have natural light and background light under the lighting vector z constraint conditions, introduce an image authenticity discriminator D N and a lighting level discriminator C;

[0033] S45. Introduce an adversarial loss to train the generation network G D2N and the image authenticity discriminator D N , making the generation network G D2N generate night images that are real enough; with minimizing this loss as the learning objective while the discriminator D N uses maximizing this loss as the learning objective for the generation network G D2N is expressed as follows:

[0034] L adv = Ε n～N log(D N (n)) + Ε d～D,z～Z log(1 - DN (G D2N (dz)))

[0035] In the formula, L adv is the adversarial loss function; E n～N is the expectation of the outputs for all night pictures n;

[0036] S46. Introduce the illumination classification loss and prompt the generation network G D2N to generate natural light and background light under the illumination vector z constraint condition for the generated night pictures; The illumination classification loss is specifically defined as follows:

[0037]

[0038]

[0039] In the formula, z n and z a are the recognition results of the illumination level discriminator C for the natural light and background light brightness levels of the generated night picture n respectively; The generation network G D2N takes minimizing the loss as the learning objective, while the discriminator C takes minimizing the loss as the learning objective; c n is the illumination vector of the natural light of the night picture n; c a is the illumination vector of the background light.

[0040] Beneficial effects brought by adopting the above technical solutions:

[0041] (1) The present invention first considers the characteristic that the night scene is not single due to the light source distribution and illumination intensity difference, and first uses the gray histogram distribution and clustering algorithm to realize the classification of the illumination intensity of the natural light and background light in the night scene;

[0042] (2) The present invention proposes a one-to-many generative adversarial network. This network can convert a daytime picture into night pictures in different illumination environments according to the set natural light and background light intensities, greatly enriching the number of generated night pictures;

[0043] (3) The present invention uses the daytime pictures carrying label information and the synthesized night pictures to jointly train the vehicle detection model, effectively alleviating the problem that the existing all-weather vehicle detection technology method has insufficient generalization ability in the night environment with different illuminations due to the scarcity of labeled night pictures or the lack of diversity of synthesized night pictures. Description of the Drawings

[0044] Figure 1It is the general flowchart of an all-weather vehicle detection method based on a one-to-many adversarial network. Specific implementation mode

[0045] The technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings.

[0046] To make up for the deficiencies of the existing technology, the present invention proposes a vehicle detection method based on a one-to-many generative adversarial network. Considering the characteristics that the night scene is not single due to the light source distribution and the difference in light intensity, the gray histogram distribution and the clustering algorithm are used to realize the classification of the light intensity of natural light and background light in the night scene; the purpose of proposing this one-to-many generative adversarial network is to convert a daytime picture into a night picture under different lighting environments according to the set natural light and background light intensities, and use the daytime picture with label information and the synthesized night picture to jointly train the vehicle detection model, effectively alleviating the problem that the existing all-weather vehicle detection technology method has insufficient generalization ability in the night environment with different lighting due to the scarcity of night pictures with label information or the lack of diversity of synthesized night pictures, as Figure 1 shown, the specific steps are as follows:

[0047] A vehicle detection method based on a one-to-many generative adversarial network includes the following steps:

[0048] S1. Use the sky segmentation algorithm to segment the collected night surveillance pictures into a sky area S and a non-sky area NS;

[0049] S2. Gray-scale the sky area S and the non-sky area NS, and count the number of pixels with gray values in the interval [0, 255] to form a gray histogram distribution G S and G NS ; divide the gray pixel interval [0, 255] into 10 equally spaced intervals, and respectively count the number of pixel values falling into each interval from the distribution results of G S and G NS to obtain the gray distribution histograms g S and g NS ;

[0050] Count the gray distribution histograms g S and g NS in all night data, and denote them as the g S set and the g NS set. Calculate the mean and variance of the g S set and the g NS set according to the statistical results, and use the obtained mean and variance to realize the data standardization of g S and g NS according to the following formula:

[0051]

[0052] In the formula: and are the mean values of the g S set and the g NS set respectively, and are the variances of the g S set and the g NS set respectively. f S and f NS are low-dimensional feature vectors.

[0053] S3. Use the DBSCAN clustering algorithm to cluster the obtained low-dimensional feature vectors f S and f NS ;

[0054] Use the elbow method to determine the optimal number of clusters for the clustering algorithm; experiments show that the optimal number of cluster groups for the low-dimensional feature vector f S is 3 and the optimal number of cluster groups for f NS is 5;

[0055] Visualize the night pictures of each cluster group, and assign lighting labels according to the light intensity of the night pictures in the cluster group, where natural light is divided into three levels: weak, medium, and strong, and background light is divided into five levels: very weak, weak, medium, strong, and very strong;

[0056] Generate one-hot encodings for natural light and background light according to the assigned levels, which are the lighting labels;

[0057] S4. Construct a day-to-night picture generation network G D2N , the purpose of this network is to convert a day picture d into multiple night pictures n under different lighting environments according to the set natural light and background light intensity z;

[0058] S42. Construct a night-to-day picture generation network G N2D , this network converts all the generated night pictures n into reconstructed day pictures

[0059] S43. Construct a cyclic consistency loss function to make the input day picture d and the reconstructed day picture close enough, and then empower the generation network G D2N to generate night pictures n that are consistent with the content of the input day picture d: This cyclic consistency loss function L c is defined as follows:

[0060]

[0061] In the formula: D is the day picture dataset, Z is the selection space of natural light and background light intensity, GD2N (d|z) is the generated night image with the light adjustment vector z and the daytime image d as inputs; E d～D,z～Z is the expectation of the outputs for all daytime images d and all light vectors z;

[0062] S44. To prompt the generation network G D2N to generate night images that are realistic enough and have natural light and background light under the condition of the light vector z, an image authenticity discriminator D N and a light level discriminator C are introduced;

[0063] S45. Introduce an adversarial loss to train the generation network G D2N and the image authenticity discriminator D N to prompt the generation network G D2N to generate night images that are realistic enough. The generation network G D2N takes minimizing this loss as the learning objective while the discriminator D N takes maximizing this loss as the learning objective:

[0064]

[0065] where L adv is the adversarial loss function; E n～N is the expectation of the outputs for all night images n;

[0066] S46. Introduce a light classification loss and prompt the generation network G D2N to generate night images that have natural light and background light under the condition of the light vector z. The light classification loss is specifically defined as follows:

[0067]

[0068] where z n and z a are the recognition results of the light level discriminator C for the brightness levels of the natural light and background light of the generated night image n respectively; the generation network C D2N takes minimizing the loss as the learning objective while the discriminator C takes minimizing the loss as the learning objective; c n is the light vector of the natural light of the night image n; c a is the light vector of the background light.

[0069] S5. The trained one-to-many generative adversarial network takes the daytime image and the adjustable light vector as inputs and generates night synthetic images under different natural light and background light conditions;

[0070] S6. Use the daytime images and the generated nighttime images to jointly train the vehicle detection model EfficientDet;

[0071] S7. Import the trained EfficientDet vehicle detector into the roadside computing device or the cloud computing center for all-weather and all-time vehicle detection tasks under roadside monitoring;

[0072] As shown in Table 1, Training Mode 1 is to directly train the model using the daytime dataset with label information; Training Mode 2 is to train the model using the nighttime dataset generated by CycleGAN; Training Mode 3 is to train the model using the nighttime dataset generated by the one-to-many generative adversarial network proposed in the present invention.

[0073] Table 1. Performance of the model in the nighttime scenario

[0074]

[0075]

[0076] According to Table 2, the training methods based on Training Modes 2 and 3 significantly improve the detection performance of the vehicle detector in the nighttime scenario. Among them, Training Mode 3 obtains the best results compared to Training Models 1 and 2 in the experiments with multiple detectors as the base detectors. In particular, the effect of the vehicle detector trained with Training Mode 3 based on the EfficientDet detector reaches the current optimal level. This optimal level is 10.01% higher than the training method using Mode 2, indicating the superiority of the one-to-many generative adversarial network proposed in the present invention for vehicle detection tasks in the nighttime scenario.

[0077] Table 2. Performance of the model in the all-weather scenario

[0078]

[0079] During the experimental verification process of the all-weather vehicle detection task, the present invention jointly trains a vehicle detection model using daytime images with annotation information and nighttime images generated by one to multiple generative adversarial networks, thereby realizing the all-day vehicle detection task. Table 2 reflects the detection accuracies of the trained model in daytime scenes, nighttime scenes, and all-day scenes respectively. Compared with the nighttime scene detection, the average detection accuracy of the vehicle detection in the daytime scene by the detection model is better because although the generative adversarial network proposed by the present invention can generate nighttime images that are difficult to distinguish between true and false, the differences between daytime images are still smaller than the differences between the synthesized nighttime images and the real nighttime images. In summary, the average detection accuracy of the all-day vehicle detection method proposed by the present invention reaches 78.95%, the accuracy rate reaches 98.03%, and the recall rate reaches 79.23%, indicating the rationality and superiority of this method applied to all-weather and all-time vehicle detection.

[0080] The embodiments are only for illustrating the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made on the basis of the technical solution according to the technical idea proposed by the present invention falls within the protection scope of the present invention.

Claims

1. An all-weather vehicle detection method based on a one-to-many adversarial network, characterized in that Including the following steps: S1. Use the sky segmentation algorithm to segment the collected night surveillance images into a sky region S and a non-sky region NS; S2. Obtain the gray - level histogram distributions G of the sky region S and the non - sky region NS S and G NS , and construct a low - dimensional feature vector f from the gray - level histogram distributions G S and G NS ; S and f NS ; S3. Use the clustering algorithm to cluster the low-dimensional feature vectors f S and f NS and respectively obtain the illumination vectors of natural light and background light of all night pictures according to the clustering results; S4. Construct a one-to-many adversarial network, and build a joint loss function based on the illumination vectors of natural light and background light obtained from night images to train the one-to-many adversarial network; S5. The trained one-to-many adversarial network takes daytime images and adjustable illumination vectors as inputs to generate night synthetic images under different natural light and background light conditions; S6. Use the daytime images and the generated night synthetic images to jointly train the vehicle detection model; S7. Import the trained vehicle detection model into the roadside computing device or the cloud computing center to perform vehicle detection tasks for all time periods under roadside surveillance; Step 2 specifically includes: Gray-scale the sky region S and the non-sky region NS, and count the number of pixels with gray values in the range [0, 255] to form a gray-scale histogram distribution G S and G NS ; Divide the gray-scale pixel range [0, 255] into 10 equally spaced intervals, and count the number of pixel values falling into each interval from the distribution results of G S and G NS to obtain the gray-scale distribution histogram g s and g NS ; Count the gray-scale distribution histogram g s and g NS in all night data, denoted as the g S set and the g NS set; Calculate the mean and variance of the g S set and the g NS set according to the statistical results, and use the obtained mean and variance to standardize the data of g S and g NS as follows: Wherein: and are the means of the G S set and the g NS set, respectively; and are the variances of the g s set and the g NS set, respectively; f S and f NS are low-dimensional feature vectors; Step 4 specifically includes: S41. Construct a day-to-night image generation network G D2N , whose purpose is to convert a daytime image d into multiple night-time images n under different lighting environments according to the set natural light and background light intensity z; S42. Construct a night-to-day image generation network G N2D , which converts all generated night images n into reconstructed day images S43. Construct a cyclic consistency loss function to prompt the input daytime image d to be close enough to the reconstructed daytime image so as to empower the generation network G D2N to generate a nighttime image n that is consistent with the content of the input daytime image d: This cyclic consistency loss function L c is defined as follows: Where: D is the daytime image dataset, Z is the selection space of natural light and background light intensity; N is the real nighttime image dataset; G D2N (d|z) is the nighttime image generated with the illumination adjustment vector z and the daytime image d as inputs; E d～D,z～Z is the expectation of the outputs for all daytime images d and all illumination vectors z; S44. To prompt the generation of the generation network G D2N The generated night pictures are realistic enough and have natural light and background light under the condition of the illumination vector z limitation. An image authenticity discriminator D is introduced N and an illumination level discriminator C; S45. Introduce adversarial loss to train the generator network G D2N and the image authenticity discriminator D N , prompting the generator network G D2N to generate sufficiently realistic night pictures; with minimizing this loss as the learning objective, while the discriminator D N has maximizing this loss as the learning objective for the generator network G D2N is expressed as follows: L adv = Ε n～N log(D N (n)) + Ε d～D,z～Z log(1 - D N (G D2N (d|z))) Where L adv is the adversarial loss function; E n～N is the expectation of the output for all night pictures n; S46. Introduce the illumination classification loss and prompt the generation network G D2N to generate a night image with natural light and background light under the illumination vector z constraint condition; the illumination classification loss is specifically defined as follows: where z n and z a are the recognition results of the light level discriminator c for the brightness levels of natural light and background light in generating the night picture n, respectively; the generation network G D2N minimizes the loss as the learning objective, while the discriminator c minimizes the loss as the learning objective; c n is the light vector of natural light in the night picture n; c a is the light vector of the background light.

2. The all-weather vehicle detection method based on a one-to-many adversarial network according to claim 1, wherein Step 3 specifically includes: S31. Cluster the obtained low-dimensional feature vectors f S and f NS using the DBSCAN clustering algorithm; S32. Use the elbow method to determine the optimal number of clusters for the clustering algorithm; S33. Visualize the night images of each cluster group, and assign illumination labels according to the illumination intensity of the night images in the cluster group, where natural light is divided into three levels; background light is divided into five levels; S34. Generate unique codes for natural light and background light according to the assigned levels, which are the illumination labels.

Images