A vehicle tracking and license plate recognition joint detection method based on improved YOLOv8

Abstract

The present application relates to the technical field of small target detection and multi-target tracking, and particularly relates to a vehicle tracking and license plate recognition joint detection method based on improved YOLOv8, which fuses vehicle detection tracking and license plate recognition in a unified system, and realizes joint detection through the following steps: firstly, an improved YOLOv8 algorithm is used to detect vehicles and license plates, then a BOT-SORT tracking algorithm based on the Hungarian algorithm is used for vehicle target tracking, and LPRNet is used for license plate recognition; by using deformable convolution operation instead of standard convolution, the receptive field size can be changed according to the image, and the MHSA attention network is added, which greatly improves the object detection accuracy while ensuring lightweight, and the present application integrates vehicle multi-target tracking and license plate recognition in a system, which is convenient to use and easy to deploy in edge devices.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of small target detection and multi-target tracking, and particularly relates to a vehicle tracking and license plate recognition joint detection method based on an improved YOLOv8. BACKGROUND

[0002] With the continuous rise in the number of motor vehicles, the road traffic environment has become increasingly complex, and traffic safety problems have become more prominent. Therefore, intelligent transportation systems have emerged to alleviate traffic safety problems. In the process of realizing traffic intelligence, accurately identifying motor vehicles on the road is one of the most important tasks.

[0003] In the past, target recognition algorithms were mainly based on traditional machine learning algorithms, which had certain limitations in accuracy and performance. With the development and widespread application of deep learning technology, the accuracy and performance of target recognition algorithms have been greatly improved. However, existing target detection technologies still have defects and shortcomings. First, the target detector is not suitable for small target detection. When the target occupies a small proportion in the picture and exceeds the detectable size of the detector, the probability of missed detection or misjudgment increases. Second, the target detector is sensitive to light, rotation, and occlusion. Changes in light, rotation, or occlusion of the target often lead to a decline in detector performance, and the accuracy and speed of target detection have a huge impact on target tracking. In scenarios such as background changes, light changes, and target deformation, the decline in target detection accuracy further leads to the failure of the tracker. There is still a lot of room for improvement in target detection and target tracking technology. SUMMARY

[0004] The purpose of the present application is to provide a vehicle tracking and license plate recognition joint detection method based on an improved YOLOv8, which aims to improve the accuracy of small target license plate recognition through a lightweight network while integrating multi-target tracking functions to improve ease of use.

[0005] To achieve the above purpose, the present application provides a vehicle tracking and license plate recognition joint detection method based on an improved YOLOv8, comprising the following steps:

[0006] Step 1: input picture data sets with both vehicle bounding boxes and license plate bounding boxes for training;

[0007] Step 2: use the improved YOLOv8 network to detect and locate the vehicle and license plate targets in the input vehicle image, and obtain the vehicle detection result and the license plate detection result, respectively;

[0008] Step 3: use the BOT-SORT tracking algorithm based on the Hungarian algorithm to calculate and process the vehicle detection result, and output the bounding box and ID of each vehicle in the picture;

[0009] Step 4: Crop the license plate area from the license plate detection results and input it into the LPRNet network for license plate recognition;

[0010] Step 5: Match the vehicle ID obtained in Step 3 with the license plate information obtained in Step 4 by calculating the intersection-union ratio of the vehicle bounding box and the license plate bounding box.

[0011] Optionally, the image dataset uses the annotated open-source dataset CCPD of Chinese city license plates. The original dataset only contains license plate annotation information. A portion of the dataset was reconstructed, and the vehicle label information in the dataset was re-annotated, thereby generating two types of labels: vehicle and license plate.

[0012] Optionally, the improvements in the improved YOLOv8 network specifically involve replacing the standard convolutions in the original YOLOv8 model's backbone network with DCN deformable convolutions, adding an MHSA multi-head attention network below the SPPF layer, and adding a feature map of size 160x160 for detecting small targets larger than 4x4.

[0013] Optionally, the process of detecting and locating both the vehicle and license plate in the input vehicle image includes the following steps:

[0014] The input image is preprocessed and then fed into the Backbone network for further processing. After two layers of standard convolution, shallow feature extraction is performed to obtain a feature map, which is then fed into C2F and DCN convolutional layers for further feature extraction. Finally, the feature map is converted into a fixed-size feature vector through the SPPF layer and fed into MHSA for attention operation, which improves performance while reducing time complexity.

[0015] Vehicle multi-scale feature fusion prediction is performed using a PAN-FPN network. A 160x160 feature output is added to the second layer of the backbone network as an additional small object detection layer. After convolution operations on two branches, the classification loss is calculated using VFL and the regression loss is calculated using CIOU. The formula for the overall loss function is as follows:

[0016] L = L VFL +L CIOU

[0017] Where L VFL L represents the VFL loss function. CIOU CIOU loss is represented by three components: center point distance, aspect ratio, and overlap area.

[0018] Optionally, the process of calculating and processing the vehicle detection results using the BOT-SORT tracking algorithm based on the Hungarian algorithm includes the following steps:

[0019] The vehicle target objects extracted in step 2 are assigned a unique ID. The Simamese network is used to calculate the feature vector and the position and size information of each target. Then, the Hungarian algorithm is used to match the targets in different frames using the feature vector.

[0020] The matching results are updated by estimating the state of each target using a Kalman filter and updating the position, velocity, and acceleration states of the tracker.

[0021] The SBS and ResNeSt50 in FastReID are used as the backbone network, and an exponential moving average update strategy is used to assist in target re-identification.

[0022] Optionally, the LPRNet network's license plate recognition process includes the following steps:

[0023] The license plate recognition frame generated in step 2 is cropped, and the cropped license plate image is resized to 94x24 and sent to the Backbone backbone network for processing.

[0024] The backbone network takes RGB images as input. Unlike LSTM based on RNN, the LPRNet backbone network uses a wide convolutional kernel with a kernel size of 1x13 to extract local character context information.

[0025] The CTC loss method is used for training, and the loss is calculated on the input X, as shown in the following formula:

[0026]

[0027] Where p t (a t |X) represents the posterior probability, t=1,…,T represents the time variable, and then the Greedy Search algorithm is used to obtain the final prediction result. Its principle is to take the node with the highest probability in each time slice as the output. The Greedy Search algorithm formula is:

[0028]

[0029] Optionally, the formula for calculating the intersection-union ratio is as follows:

[0030]

[0031] Where B1 and B2 represent the license plate bounding box and the vehicle target box, respectively, and the threshold is set to 0.7. When the IOU is greater than 0.7, it means that the license plate and the vehicle ID match.

[0032] This invention provides a joint detection method for vehicle tracking and license plate recognition based on an improved YOLOv8 algorithm. It integrates vehicle detection and tracking with license plate recognition into a unified system. The joint detection is achieved through the following steps: first, the improved YOLOv8 algorithm is used to detect vehicles and license plates; then, the BOT-SORT tracking algorithm based on the Hungarian algorithm is used for vehicle target tracking; and simultaneously, LPRNet is used for license plate recognition. By using deformable convolution operations instead of standard convolution, the receptive field size can be adjusted according to the image. Furthermore, an MHSA attention network is added, significantly improving object detection accuracy while maintaining a lightweight design. This invention also integrates multi-target vehicle tracking and license plate recognition into a single system, making it convenient to use and easy to deploy on edge devices. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating a vehicle tracking and license plate recognition joint detection method based on an improved YOLOv8 according to the present invention.

[0035] Figure 2 This is a schematic diagram of vehicle and license plate detection results in a specific embodiment of the present invention.

[0036] Figure 3 This is a network training PR curve diagram in a specific embodiment of the present invention. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0038] This invention provides a joint detection method for vehicle tracking and license plate recognition based on an improved YOLOv8, comprising the following steps:

[0039] S1: Input an image dataset with both vehicle bounding boxes and license plate bounding boxes for training;

[0040] S2: Use the improved YOLOv8 network to detect and locate the two targets, vehicle and license plate, in the input vehicle image, and obtain the vehicle detection results and license plate detection results respectively.

[0041] S3: The vehicle detection results are processed using the BOT-SORT tracking algorithm based on the Hungarian algorithm, and the bounding box and ID of each vehicle in the image are output.

[0042] S4: Crop out the license plate area from the license plate detection results and input it into the LPRNet network for license plate recognition;

[0043] S5: By calculating the intersection-union ratio of the vehicle bounding box and the license plate bounding box, match the vehicle ID obtained in step S3 with the license plate information obtained in step S4.

[0044] The specific execution flowchart is as follows: Figure 1 As shown, the following provides further explanation in conjunction with specific implementation steps:

[0045] The dataset in step S1 uses the annotated open-source dataset CCPD of Chinese city license plates. Since the original dataset only contains license plate labeling information, part of the dataset was reconstructed and the labeling information of the vehicles in the dataset was re-annotated.

[0046] Furthermore, the target detection and localization steps for vehicles and license plates in step S2 are as follows:

[0047] 2.1) First, the image input in step S1 is preprocessed and then fed into the Backbone network for further processing. After two layers of standard convolutions, shallow feature extraction is performed to obtain a feature map. Then, it is fed into C2F and DCN convolutional layers for further feature extraction. Finally, the feature map is converted into a fixed-size feature vector through an SPPF layer and fed into MHSA for attention operations, which improves performance while reducing time complexity.

[0048] 2.2) Multi-scale feature fusion prediction of vehicles is performed using a PAN-FPN network. A 160x160 feature output is added to the second layer of the backbone as an additional small target detection layer. Finally, after standard convolution operations are performed on the two branches, the classification loss is calculated using VFL and the regression loss is calculated using CIOU. The loss function is shown in formula (1):

[0049] L = L VFL +L CIOU (1)

[0050] Here, VFL (varifocal loss) represents the loss function, and CIOU represents the loss, which includes three terms: center point distance, aspect ratio, and overlap area.

[0051] Secondly, the step of tracking the vehicle target using the BOT-SORT tracking algorithm based on the Hungarian algorithm in step S3 is as follows:

[0052] 4.1) Assign a unique ID to the target object extracted in step S2. Calculate the feature vector and information such as the target object's position and size for each target using the Simamese network. Then, use the feature vector to match targets in different frames using the Hungarian algorithm. In this invention, a matching cost matrix C(i,j) is established between the predicted trajectory i of the previous frame and the current detection result j. The elements of this matrix are weighted and fused from the motion and appearance features of i and j, quantized into a metric value D(i,j). The global optimal solution of the feature cost matrix is ​​solved to obtain the matching relationship between targets.

[0053] 4.2) Update the state of the matching results in 4.1. Use the Kalman filter to estimate the state of each target and update the position, velocity, and acceleration of the tracker. In Bot-SORT, the state vector of the Kalman filter is defined as an 8-tuple, and the state equation is as shown in formula (2):

[0054]

[0055] In the formula, represents the system state at the previous moment, is the state transition matrix, is the input control matrix, and is the process noise of the random variable.

[0056] 4.3) During the tracking process, target occlusion, disappearance, or the appearance of new targets may occur, requiring target re-identification (Re-ID). To extract Re-ID features more effectively, SBS and ResNeSt50 from FastReID are used as the backbone network, and the feature update strategy for updating the appearance state of the i-th trajectory in the current frame is updated using the exponential moving average (EMA) method, expressed as formula (3):

[0057]

[0058] Where represents the tracklets embedding of the k-th frame, represents the momentum term, and represents the detection embedding of the k-th frame. Step S4, using LPRNet for license plate recognition, is as follows:

[0059] 5.1) First, the vehicle bounding box generated in step 3) is cropped, and the cropped license plate image is resized to 94x24 and sent to the Backbone backbone network for processing.

[0060] 5.2) The backbone network takes RGB images as input. Unlike LSTM based on RNN, the LPRNet Backbone backbone network uses a wide convolutional kernel with a kernel size of 1x13 to extract local character context information.

[0061] 5.3) Since the decoder output and the target character sequence have different lengths in step 5.2), the CTC loss method is used for training. The loss function is calculated with X as input, as shown in the following formula:

[0062]

[0063] Where represents the posterior probability, and represents the time variable. The Greedy Search algorithm is then used to obtain the final prediction result. Its principle is to take the node with the highest probability in each time slice as the output, as shown in the formula:

[0064]

[0065] The steps in step 6) are as follows:

[0066] 6.1) Calculate the intersection-union ratio (IUU) of the license plate recognition frame and the vehicle target frame with a fixed ID. The formula is:

[0067]

[0068] Here, represents the license plate bounding box and the vehicle target box, respectively. The threshold is set to 0.7. When the IOU is greater than 0.7, it means that the license plate and the vehicle ID match.

[0069] Furthermore, the present invention also proposes a specific embodiment, such as... Figure 2 and Figure 3 As shown. Figure 2 The target detector in the image can accurately identify large targets such as vehicles, and can also detect small targets such as license plates with high accuracy and identify the content of the license plates. Figure 3 The PR curve during model training is shown. It can be seen that the curves are very close to the upper right corner, indicating that the model has high accuracy and recall, and the model performs well in detecting small targets such as vehicles and license plates.

[0070] In summary, compared with existing methods, the present invention has the following advantages:

[0071] 1. This invention uses deformable convolution operations instead of standard convolution, which can change the receptive field size according to the image. At the same time, it adds an MHSA attention network, which greatly improves the object detection accuracy while ensuring lightweight design.

[0072] 2. This invention adds a small target detection layer to YOLOv8 to meet the requirements of license plate recognition, thereby improving the accuracy of small target recognition for license plates.

[0073] 3. This invention integrates multi-target vehicle tracking and license plate recognition into one system, which is convenient to use and easy to deploy in edge devices. It has good performance in multi-target vehicle tracking and license plate detection in road surveillance videos and is more convenient to use.

[0074] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A joint detection method for vehicle tracking and license plate recognition based on improved YOLOv8, characterized in that, Includes the following steps: Step 1: Train the image dataset by inputting an image dataset with both vehicle bounding boxes and license plate bounding boxes as labels; Step 2: Use the improved YOLOv8 network to detect and locate the vehicle and license plate targets in the input vehicle image, and obtain the vehicle detection results and license plate detection results respectively; The improvements in the improved YOLOv8 network specifically involve replacing the standard convolutions in the original YOLOv8 model's backbone network with DCN deformable convolutions, adding an MHSA multi-head attention network below the SPPF layer, and adding a 160x160 feature map as a small object detection layer in the second layer of the backbone network. The process of detecting and locating both the vehicle and its license plate in the input vehicle image includes the following steps: The input image is preprocessed and then fed into the Backbone network for further processing. After two layers of standard convolution, shallow feature extraction is performed to obtain a feature map, which is then fed into C2F and DCN convolutional layers for further feature extraction. Finally, the feature map is converted into a fixed-size feature vector through the SPPF layer and fed into MHSA for attention operation, which improves performance while reducing time complexity. Vehicle multi-scale feature fusion prediction is performed using the PAN-FPN network. A 160x160 feature map is added as a small object detection layer in the second layer of the backbone network. After convolution operations in two branches, the classification loss is calculated using VFL and the regression loss is calculated using CIOU. The formula for the overall loss function is as follows: L=L VFL +L CIOU Where L VFL L represents the VFL loss function. CIOU Indicates CIOU loss; Step 3: Use the BOT-SORT tracking algorithm based on the Hungarian algorithm to calculate and process the vehicle detection results, and output the bounding box and ID of each vehicle in the image; Step 4: Crop the license plate area from the license plate detection results and input it into the LPRNet network for license plate recognition; Step 5: Match the vehicle ID obtained in Step 3 with the license plate information obtained in Step 4 by calculating the intersection-union ratio of the vehicle bounding box and the license plate bounding box.

2. The vehicle tracking and license plate recognition joint detection method based on improved YOLOv8 as described in claim 1, characterized in that, The image dataset uses the annotated open-source dataset CCPD of Chinese city license plates. The original dataset only contained license plate annotation information. A portion of the dataset was reconstructed, and the vehicle label information in the dataset was re-annotated, thus generating two types of labels: vehicle and license plate.

3. The vehicle tracking and license plate recognition joint detection method based on improved YOLOv8 as described in claim 2, characterized in that, The process of calculating and processing the vehicle detection results using the BOT-SORT tracking algorithm based on the Hungarian algorithm includes the following steps: The vehicle target objects extracted in step 2 are assigned a unique ID. The Simamese network is used to calculate the feature vector and the position and size information of each target. Then, the Hungarian algorithm is used to match the targets in different frames using the feature vector. The matching results are updated by estimating the state of each target using a Kalman filter and updating the position, velocity, and acceleration states of the tracker. The SBS and ResNeSt50 in FastReID are used as the backbone network, and an exponential moving average update strategy is used to assist in target re-identification.

4. The vehicle tracking and license plate recognition joint detection method based on improved YOLOv8 as described in claim 3, characterized in that, The process of license plate recognition using the LPRNet network includes the following steps: The license plate recognition frame generated in step 2 is cropped, and the cropped license plate image is resized to 94x24 and sent to the Backbone backbone network for processing. The backbone network takes RGB images as input. Unlike LSTM based on RNN, the LPRNet backbone network uses a wide convolutional kernel with a kernel size of 1x13 to extract local character context information. The CTC loss method is used for training, and the loss is calculated on the input X, as shown in the following formula: Where p t (a t |X) represents the posterior probability, t=1,…,T represents the time variable, and then the Greedy Search algorithm is used to obtain the final prediction result. Its principle is to take the node with the highest probability in each time slice as the output. The Greedy Search algorithm formula is: 。 5. The vehicle tracking and license plate recognition joint detection method based on improved YOLOv8 as described in claim 4, characterized in that, The formula for calculating the intersection-union ratio is as follows: Where B1 and B2 represent the license plate bounding box and the vehicle target box, respectively, and the threshold is set to 0.

7. When the IOU is greater than 0.7, it means that the license plate and the vehicle ID match.

Images