A nighttime vehicle recognition method based on CycleGAN

Abstract

The invention discloses a nighttime vehicle recognition method based on CycleGAN, comprising the following steps: obtaining a data set of vehicle images, dividing the data set into a training set and a test set; obtaining an improved YOLOv4 based on CycleGAN and YOLOv4; training CycleGAN based on nighttime vehicle images and daytime vehicle images to obtain a trained converter; training the improved YOLOv4 based on nighttime vehicle images and the trained converter to obtain a trained YOLOv4; testing the test set based on the trained YOLOv4 to obtain a nighttime vehicle recognition result. The invention can improve the contrast between the vehicle and the background and suppress the interference of ambient light, more accurately identify the vehicle at night, effectively enhance the vehicle features, and improve the accuracy of nighttime vehicle detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of nighttime vehicle detection, and in particular relates to a nighttime vehicle recognition method based on CycleGAN. Background Art

[0002] Vehicle detection is an important application in the field of target detection. In autonomous driving systems and assisted driving systems, the accuracy of vehicle detection in day and night conditions directly affects driving safety. Under night conditions, due to the poor visibility of urban roads and vehicles ahead, less visual information can be used, resulting in an increased probability of traffic accidents. At the same time, under the complex lighting environment of urban roads, most existing target detection algorithms will be interfered by various other lights, causing confusion between the vehicle's headlights and taillights and resulting in misjudgment, which poses a major challenge to vision-based night vehicle detection and seriously affects the safety of autonomous driving systems and assisted driving systems.

[0003] Headlights and taillights are obvious features of vehicles at night. Traditional nighttime vehicle detection methods locate headlights or taillights by segmenting images to obtain the vehicle position and predict the vehicle's bounding box. First, image segmentation, connected domain analysis and other methods are used to obtain candidate regions, then lights are selected from the information in the candidate regions, and finally the symmetry of the lights is used to pair them to obtain the predicted position and predicted bounding box of the vehicle to determine whether there is a vehicle in the candidate region. Traditional nighttime vehicle detection methods have two disadvantages: (1) Due to the complex lighting conditions in urban areas, including vehicle lights, street lights, building lights, and lights reflected by vehicles, the vehicle detection performance is poor and even increases the false positive rate; (2) When the vehicle may be blocked by other vehicles or only the side of the vehicle is captured, the lights may not be matched, which increases the missed detection rate. Summary of the invention

[0004] With the development of deep learning, the vehicle detection method given to convolutional neural network has greatly improved its accuracy compared with traditional methods. In view of the above technical problems, the present invention adopts YOLOv4 to perform night vehicle recognition tasks, and adds CycleGAN conversion method to it to enhance the characteristics of night vehicles.

[0005] To achieve the above object, the present invention provides a nighttime vehicle recognition method based on CycleGAN, comprising the following steps:

[0006] Obtain a data set of vehicle images, and divide the data set into a training set and a test set, wherein the training set includes a training set of CycleGAN and a training set of YOLOv4;

[0007] Based on CycleGAN and YOLOv4, an improved YOLOv4 is obtained;

[0008] Training the CycleGAN based on the training set of the CycleGAN to obtain weights of the trained converter;

[0009] Training the improved YOLOv4 based on the training set of the YOLOv4 and the trained converter to obtain a trained YOLOv4;

[0010] The test set is tested based on the trained YOLOv4 to obtain nighttime vehicle recognition results.

[0011] Optionally, the training set of the CycleGAN is nighttime vehicle images and daytime vehicle images;

[0012] The training set of YOLOv4 is nighttime vehicle images.

[0013] Optionally, the trained converter is a converter that converts nighttime vehicle features into daytime vehicle features.

[0014] Optionally, the step of training the improved YOLOv4 based on the training set of the YOLOv4 and the trained converter includes:

[0015] The improved YOLOv4 is trained based on the training set of the YOLOv4 to obtain a multi-scale nighttime vehicle feature map;

[0016] Based on the trained converter and the multi-scale nighttime vehicle feature map, a multi-scale fused enhanced nighttime vehicle feature map is obtained, and the training of the improved YOLOv4 is completed.

[0017] Optionally, the multi-scale nighttime vehicle feature map includes: a first-scale feature map, a second-scale feature map, and a third-scale feature map;

[0018] The first scale feature map, the second scale feature map and the third scale feature map are feature maps of three different scales.

[0019] Optionally, the multi-scale fused enhanced nighttime vehicle feature map includes: an enhanced first-scale feature map, an enhanced second-scale feature map, and an enhanced third-scale feature map;

[0020] The steps of obtaining a multi-scale fused enhanced nighttime vehicle feature map include:

[0021] Performing maximum pooling and stacking processing on the third scale feature map to obtain a processed third scale feature map;

[0022] Performing feature fusion based on the second scale feature map and the processed third scale feature map to obtain a fused second scale feature map;

[0023] Converting the first scale feature map based on the trained converter to obtain a converted first scale feature map, and performing feature fusion on the converted first scale feature map based on the fused second scale feature map to obtain the enhanced first scale feature map;

[0024] Performing feature fusion based on the enhanced first scale feature map and the fused second scale feature map to obtain the enhanced second scale feature map;

[0025] Feature fusion is performed based on the enhanced second scale feature map and the processed third scale feature map to obtain the enhanced third scale feature map.

[0026] Optionally, training the improved YOLOv4 based on the training set of the YOLOv4 and the trained converter further comprises: setting a loss function to train the improved YOLOv4;

[0027] The loss function is the sum of the loss function of CycleGAN and the loss function of YOLOv4.

[0028] Optionally, the loss function of the YOLOv4 includes:

[0029]

[0030]

[0031]

[0032] Among them, K 2 is the total number of grids, The meaning value indicating that the target exists in the candidate box, is the predicted probability, P i (c) is the true probability that the target in the grid is of class c, M is the number of candidate boxes generated in each grid, is the actual confidence, λ noobj is the weight of the candidate box where there is no target, Indicates that the target does not exist in the candidate box, C i is the prediction confidence, w gt With h gt is the width and height of the true box, IOU is the ratio of the intersection and union of the true box and the candidate box, d is the Euclidean distance, c is the diagonal distance, α is the trade-off parameter, and ν is the consistency of the aspect ratio.

[0033] Optionally, the loss function of the CycleGAN includes:

[0034]

[0035]

[0036] Among them, G night The generator for converting night images into day images, G day Generator for converting daytime images into nighttime images, D day To determine whether the daytime image is real or fake.

[0037] Compared with the prior art, the present invention has the following advantages and technical effects:

[0038] The present invention adds a CycleGAN converter to YOLOv4, uses CycleGAN to achieve style conversion, converts night features into day features, and enhances night vehicle features. After passing through the CycleGAN converter, the obtained daytime vehicle features and nighttime vehicle features are subjected to multi-scale feature fusion, which enhances vehicle features while reducing the missed detection rate and misjudgment rate of long-distance vehicles. The loss function becomes the sum of the CycleGAN loss and the YOLOv4 loss to improve the performance of night feature conversion and daytime feature conversion. The present invention can effectively enhance vehicle features and improve the accuracy of night vehicle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 This is a flow chart of a method for nighttime vehicle recognition based on CycleGAN according to Embodiment 1 of the present invention;

[0041] Figure 2 This is a schematic diagram of the YOLOv4 network structure of the first embodiment of the present invention;

[0042] Figure 3 Schematic diagram of a CycleGAN model according to the first embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the improved YOLOv4 network structure of Example 1 of the present invention. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Embodiment 1

[0047] like Figure 1 As shown, the present invention provides a nighttime vehicle recognition method based on CycleGAN, comprising the following steps:

[0048] YOLOv4

[0049] YOLOv4 is mainly composed of three parts: Backbone, Neck, and Head. The overall network structure is as follows Figure 2 shown.

[0050] Backbone uses the CSPDarknet53 network to extract image features. CSPDarknet53 is an improved version of Darknet53, mainly composed of darknet53 and residual network. Darknet53 will extract input image features and use residual network to fuse features. Neck uses a combination of spatial pyramid pooling (SPP) and path aggregation network (PANet) modules to strengthen the composition of the extraction network. The SPP structure stacks features of different scales after maximum pooling, and PANet combines convolution, upsampling, downsampling and other modules to fuse features of feature layers of different scales to obtain enhanced features. Head, as the output layer of YOLOv4, extracts features to predict results, classify and locate targets.

[0051] Compared with YOLOv3, the loss function of YOLOv4 only uses CIOU loss instead of MSEloss for bounding box regression loss. CIOU loss focuses on improving the ratio of the rectangular box, which further improves the detection effect.

[0052]

[0053]

[0054]

[0055] Among them, K 2 is the total number of grids, The meaning value indicating that the target exists in the candidate box, is the predicted probability, P i (c) is the true probability that the target in the grid is of class c, M is the number of candidate boxes generated in each grid, is the actual confidence, λ noobj is the weight of the candidate box where there is no target, Indicates that the target does not exist in the candidate box, C i is the prediction confidence, w gt With h gt is the width and height of the true box, IOU is the ratio of the intersection and union of the true box and the candidate box, d is the Euclidean distance, c is the diagonal distance, α is the trade-off parameter, and ν is the consistency of the aspect ratio.

[0056] CycleGAN

[0057] GAN has achieved good results in some applications in the field of deep learning. Most of these applications use GAN to enhance images to obtain "fake" images similar to real images, and the "fake" images are better than real images for training, thus achieving the purpose of enhancing the training effect. Cycle generative adversarial network (CycleGAN) is a variant of GAN, which is used to convert images from one scene to another. Unlike Pix2pix, the images trained by CycleGAN do not need to be paired, which makes it more convenient to perform image style transfer.

[0058] CycleGAN consists of two generators and two discriminators. Unlike GAN, CycleGAN does not need to rely on one-to-one corresponding images, and CycleGAN introduces the concept of cycle, which solves the problem that the mapping learned by GAN cannot convert different images into the same image. The concept of cycle is to map the generated "fake" image back to the real image, which will produce a "fake" real image, and use the discriminator to judge whether it is similar to the real image, so that a one-to-one mapping can be learned instead of a one-to-many mapping. Two transformations G and F are defined in the CycleGAN model. The G transformation represents the transformation from the input domain X to the target domain Y, and the F transformation represents the transformation from the target domain Y to the input domain X, which also shows that F is the inverse transformation of G. And in the input domain X and the target domain Y, there are two discriminators D x and D y To determine whether the converted images of F and G are similar to the images of input domain X and target domain Y. The CycleGAN model structure is as follows Figure 3 shown.

[0059] For night images, the image should be able to return to the original image through the conversion cycle of CycleGAN, and its loss function is:

[0060]

[0061]

[0062] Among them, G night The generator for converting night images into day images, G day Generator for converting daytime images into nighttime images, D day To determine whether the daytime image is real or fake.

[0063] In the task of identifying vehicles at night in cities, if the road lighting is very complex, the traditional algorithm that relies on the characteristics of vehicle lights is not very effective, and the nighttime vehicle features extracted in YOLOv4 cannot meet the requirements of autonomous driving for detecting vehicles. The daytime vehicle images generated by CycleGAN will lose some nighttime vehicle information, and even some distant vehicles may be mistaken for background or cannot be detected. Therefore, the present invention adds CycleGAN to YOLOv4, so that the nighttime features are converted into daytime features after passing through the CycleGAN converter, which reduces the impact of nighttime features on vehicle detection and reduces complex lighting information. The improved YOLOv4 network structure is shown in the figure below. Figure 4 shown.

[0064] Step 1. Obtain a dataset of vehicle images and divide the dataset into a training set and a test set;

[0065] The training sets include the training sets of CycleGAN and YOLOv4. The training sets of CycleGAN are night vehicle images and day vehicle images, and the training sets of YOLOv4 are night vehicle images.

[0066] In this embodiment, the vehicle image dataset uses the Berkeley Deep Drive (BDD) 100K dataset, and a total of 11,549 images are obtained, of which 836 night images and 846 daytime images are used as training sets for the CycleGAN network, 7,865 night images are used as training sets for the YOLOv4 network, and 2,002 night images are used as test sets.

[0067] Step 2. Train CycleGAN based on the CycleGAN training set to obtain the trained converter.

[0068] In this embodiment, the scale size of the data set is 416*416, and the 416*416 nighttime vehicle images and the corresponding daytime vehicle images are input into CycleGAN for training to obtain the nighttime to daytime converter and the corresponding weight w.

[0069] Step 3. Based on CycleGAN and YOLOv4, obtain the improved YOLOv4.

[0070] In this embodiment, the improved YOLOv4 is that a CycleGAN converter is added to the Neck of YOLOv4 and the corresponding weight w is called to enhance the nighttime vehicle features.

[0071] Step 4. Train the improved YOLOv4 based on the YOLOv4 training set and the trained converter to obtain the trained YOLOv4.

[0072] The steps of training the improved YOLOv4 include: training the improved YOLOv4 based on the training set of YOLOv4 to obtain a multi-scale nighttime vehicle feature map; obtaining a multi-scale fused enhanced nighttime vehicle feature map based on the trained converter and the multi-scale nighttime vehicle feature map, and completing the training of the improved YOLOv4.

[0073] The multi-scale nighttime vehicle feature map includes: a first-scale feature map, a second-scale feature map and a third-scale feature map; the first-scale feature map, the second-scale feature map and the third-scale feature map are feature maps of three different scales.

[0074] The multi-scale fused enhanced nighttime vehicle feature map includes: an enhanced first-scale feature map, an enhanced second-scale feature map, and an enhanced third-scale feature map;

[0075] The steps of obtaining a multi-scale fused enhanced nighttime vehicle feature map include:

[0076] Performing maximum pooling and stacking processing on the third-scale feature map to obtain a processed third-scale feature map;

[0077] Perform feature fusion based on the second scale feature map and the processed third scale feature map to obtain a fused second scale feature map;

[0078] The first scale feature map is converted based on the trained converter to obtain a converted first scale feature map, and feature fusion is performed based on the fused second scale feature map and the converted first scale feature map to obtain an enhanced first scale feature map;

[0079] Perform feature fusion based on the enhanced first-scale feature map and the fused second-scale feature map to obtain an enhanced second-scale feature map;

[0080] Feature fusion is performed based on the enhanced second-scale feature map and the processed third-scale feature map to obtain an enhanced third-scale feature map.

[0081] Training the improved YOLOv4 based on the training set of YOLOv4 and the trained converter also includes: setting a loss function to train the improved YOLOv4; the loss function is the sum of the loss function of CycleGAN and the loss function of YOLOv4.

[0082] In this implementation, a nighttime vehicle image with an image size of 416*416 is input into the improved YOLOv4 network for training. First, the image is input into the CSPDarknet53 feature extraction network to extract nighttime vehicle features and obtain three nighttime feature maps of different scales, which are 52*52, 26*26, and 13*13, corresponding to three branches. The 52*52 feature map is the first scale feature map, the 26*26 feature map is the second scale feature map, and the 13*13 feature map is the third scale feature map.

[0083] 4.1. The 13*13 feature map is input into the SPP module for maximum pooling and stacking operations.

[0084] 4.2. The 26*26 feature map branch is downsampled and fused with the 13*13 feature map that has been upsampled in step 4.1 to obtain a fused 26*26 feature map.

[0085] 4.3, add the trained converter, that is, multiple resnet blocks, to the 52*52 feature map branch and call the corresponding weight w for conversion to obtain a 52*52 feature map during the day. The fused 26*26 feature map obtained in step 4.2 is upsampled and fused with the 52*52 feature map during the day to obtain an enhanced 52*52 feature map, which strengthens the nighttime vehicle features.

[0086] 4.4. After downsampling, the enhanced 52*52 feature map in step 4.3 is fused with the 26*26 feature map fused in step 4.2 to obtain an enhanced 26*26 feature map. Similarly, after downsampling, the enhanced 26*26 feature map is fused with the feature map obtained in step 4.1 to obtain an enhanced 13*13 feature map. After the above operations, the training model is obtained.

[0087] In the present invention, the loss function becomes the sum of the CycleGAN loss and the YOLOv4 loss to improve the performance of nighttime feature conversion and daytime feature conversion.

[0088] The loss function is:

[0089] L=L cls +L conf +L loc +L cyc_night +L D_night .

[0090] Step 5. Test the test set based on the trained YOLOv4 to obtain the night vehicle recognition results.

[0091] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A nighttime vehicle recognition method based on CycleGAN, characterized in that: The following steps are involved: Obtain a data set of vehicle images, and divide the data set into a training set and a test set, wherein the training set includes a training set of CycleGAN and a training set of YOLOv4; Based on CycleGAN and YOLOv4, an improved YOLOv4 is obtained; Training the CycleGAN based on the training set of the CycleGAN to obtain a trained converter; Training the improved YOLOv4 based on the training set of the YOLOv4 and the trained converter to obtain a trained YOLOv4; The test set is tested based on the trained YOLOv4 to obtain a nighttime vehicle recognition result; The training set of CycleGAN is nighttime vehicle images and daytime vehicle images; The training set of YOLOv4 is nighttime vehicle images; The trained converter is a converter that converts nighttime vehicle features into daytime vehicle features; The step of training the improved YOLOv4 based on the training set of the YOLOv4 and the trained converter comprises: The improved YOLOv4 is trained based on the training set of the YOLOv4 to obtain a multi-scale nighttime vehicle feature map; Based on the trained converter and the multi-scale nighttime vehicle feature map, a multi-scale fused enhanced nighttime vehicle feature map is obtained, and the training of the improved YOLOv4 is completed; The multi-scale nighttime vehicle feature map includes: a first-scale feature map, a second-scale feature map, and a third-scale feature map; The first scale feature map, the second scale feature map and the third scale feature map are feature maps of three different scales; The multi-scale fused enhanced nighttime vehicle feature map includes: an enhanced first-scale feature map, an enhanced second-scale feature map, and an enhanced third-scale feature map; The steps of obtaining a multi-scale fused enhanced nighttime vehicle feature map include: Performing maximum pooling and stacking processing on the third scale feature map to obtain a processed third scale feature map; Performing feature fusion based on the second scale feature map and the processed third scale feature map to obtain a fused second scale feature map; Converting the first scale feature map based on the trained converter to obtain a converted first scale feature map, and performing feature fusion on the converted first scale feature map based on the fused second scale feature map to obtain the enhanced first scale feature map; Performing feature fusion based on the enhanced first scale feature map and the fused second scale feature map to obtain the enhanced second scale feature map; Feature fusion is performed based on the enhanced second scale feature map and the processed third scale feature map to obtain the enhanced third scale feature map.

2. The nighttime vehicle recognition method based on CycleGAN according to claim 1, characterized in that: Training the improved YOLOv4 based on the training set of the YOLOv4 and the trained converter further includes: setting a loss function to train the improved YOLOv4; The loss function is the sum of the loss function of CycleGAN and the loss function of YOLOv4.

3. The nighttime vehicle recognition method based on CycleGAN according to claim 2, characterized in that: The loss function of YOLOv4 includes: Among them, K 2 is the total number of grids, The meaning value indicating that the target exists in the candidate box, is the predicted probability, P i (c) is the true probability that the target in the grid is of class c, M is the number of candidate boxes generated in each grid, is the actual confidence, λ noobj is the weight of the candidate box without the target, Indicates that the target does not exist in the candidate box, C i is the prediction confidence, w gt With h gt is the width and height of the true box, IOU is the ratio of the intersection and union of the true box and the candidate box, d is the Euclidean distance, c is the diagonal distance, α is the trade-off parameter, and ν is the consistency of the aspect ratio.

4. The nighttime vehicle recognition method based on CycleGAN according to claim 2, characterized in that: The loss function of CycleGAN includes: Among them, G night The generator for converting night images into day images, G day Generator for converting daytime images into nighttime images, D day To determine whether the daytime image is real or fake.

Images