License plate detection and license plate recognition method and system

Abstract

The invention discloses a license plate detection and license plate recognition method and system, and the method comprises the steps: collecting and marking an image, and constructing a license plate detection data set; detecting the position of a license plate based on an improved yolov5 network and carrying out positioning identification; cutting out a license plate image through a related cutting function so as to construct a license plate recognition data set; and constructing a license plate recognition network based on an LPRNet network, training the license plate recognition network by using the license plate recognition data set, obtaining an optimal license plate recognition model, extracting license plate character features by using the optimal license plate recognition model, and continuously optimizing by using a CTCLoss loss function so as to obtain a license plate recognition result. According to the method, license plate character features can be extracted without segmenting license plate numbers into single characters, license plate characters are output, the recognition accuracy and efficiency are improved, and the method can be effectively applied to the field of intelligent traffic management.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic management, and in particular to a license plate detection and recognition system. In particular, when used for license plate detection and recognition problems in complex scenarios such as long distances and rainy and snowy weather, the present invention improves the accuracy of license plate detection and recognition. Background Art

[0002] License plate detection and recognition technology is a technology that accurately locates and extracts vehicle license plates in complex backgrounds and accurately recognizes and analyzes the characters on license plates. This technology has become increasingly mature in the field of intelligent traffic management. In large urban traffic management areas, especially in scenes prone to traffic accidents such as intersections and crowded public places, license plate detection and recognition systems can provide accurate and effective information support for urban traffic management.

[0003] The traditional license plate detection and recognition system mainly consists of two parts: license plate detection and license plate character recognition. The license plate detection part is used to detect the position of the license plate, which is achieved by the license plate positioning method; the license plate character recognition part is used to recognize and output the character information of the license plate, which uses the segmentation algorithm to segment the characters in the license plate, and then recognizes the segmented characters in order and finally outputs them.

[0004] However, traditional license plate detection technology usually uses HOG+SVM algorithm and DPM algorithm. It is difficult for HOG+SVM algorithm to solve problems such as image occlusion and large amplitude of pedestrian body movement, and because of the use of gradient, it is greatly affected by noise. The excitation features of the DMP algorithm are set manually, the workload is large, and the universality is poor. Traditional license plate recognition technology includes two stages: character segmentation and character classification. Character segmentation is the process of dividing multiple license plate characters of the original license plate image into single characters and converting them into images again. The whole process is greatly affected by the quality of the image. In the character classification stage, the segmented characters are generally divided into Chinese characters, numbers and letters through the reverse error propagation neural network, and then the characters are recognized through the convolutional neural network, but there are problems of insufficient generalization ability and high computing resource requirements.

[0005] It should be noted that the information disclosed in the above background technology section is only used to deepen the understanding of the background of the present invention, and therefore may include information that is not prior art known to ordinary technicians in the field. Summary of the invention

[0006] In view of the defects of the prior art, the present invention provides a license plate detection and license plate recognition method and system to meet the application in the field of intelligent traffic management in complex scenarios such as long distances and rainy and snowy weather.

[0007] In order to meet the above application requirements, the present invention provides a license plate detection and license plate recognition method, comprising the following steps:

[0008] Use the CCPD dataset and annotate it to build a license plate detection dataset;

[0009] Based on the yolov5 network, it adds functions such as enhanced data, enhanced feature extraction capability, and enhanced feature fusion capability. The improved yolov5 network is trained using the above-constructed license plate detection dataset to obtain the license plate detection model, based on which the license plate position is detected and located.

[0010] A license plate recognition dataset is constructed based on the CCPD dataset. The license plate detection model is used to detect the license plate position and locate it, obtain the position area coordinate information of the license plate, and use the relevant cropping function to crop the license plate from the picture to obtain the license plate image.

[0011] A license plate recognition network is constructed based on the LPRNet network, and an output sequence is obtained with maximum probability using beam search. The license plate recognition network is trained with the license plate recognition data set, and license plate character features are extracted using the LPRNet network. The CTC_Loss loss function is used for continuous optimization to obtain a license plate recognition result.

[0012] The present invention adopts an improved yolov5 algorithm to detect the license plate, detects the license plate position and obtains the position area coordinate information of the license plate, obtains the front license plate image through the relevant clipping function, and extracts the license plate character features without segmenting the license plate number into single characters by using the license plate recognition network constructed by LPRNet. Finally, the license plate characters are output, which improves the recognition accuracy and efficiency.

[0013] In a more specific technical solution, the improved yolov5 network adds functions such as enhanced data, enhanced feature extraction capability, and enhanced feature fusion capability. The improved yolov5 network is trained using the above-constructed license plate detection data set to obtain a license plate detection model, thereby detecting the license plate position, positioning and identifying it, and outputting the license plate image, including:

[0014] The functions of Mosaic data enhancement, adaptive image scaling and adaptive anchor frame calculation are set at the input end of the yolov5;

[0015] The Focus structure and CSPNet structure are set in the backbone network of yolov5 to enhance the ability to extract image features;

[0016] The CSP2_X structure is introduced into the neck network of yolov5 to further enhance the feature fusion capability;

[0017] Construct the yolov5 improved network, and use the GIOU_Loss loss function to add the overlapping area, center point distance, and aspect ratio between the prediction box and the target box;

[0018] The license plate detection data set is used to train the yolov5 improved network, thereby obtaining the optimal license plate detection model;

[0019] The location area coordinate information of the license plate is obtained using the yolov5 improved network.

[0020] In a more specific technical solution, the input end of the yolov5 is provided with the functions of Mosaic data enhancement, adaptive image scaling and adaptive anchor frame calculation, specifically including:

[0021] Mosaic data enhancement randomly selects 4 images and combines them by cropping, deforming, and placing them, so that the smallest batch can be appropriately reduced, reducing the amount of calculation. Mosaic data enhancement greatly enriches the image features, and has better detection effects for small targets such as license plates.

[0022] In the original yolov5 algorithm, the original image needs to be specified in a standard size before being input into the network model. However, the CCPD dataset has clear images and only contains cars, so the adaptive image scaling is used to improve the training speed.

[0023] The adaptive anchor frame calculation obtains a predicted frame based on a given anchor frame during model training, compares the real frame with the predicted frame, calculates the error value between the two, and then updates in reverse, so that the network parameters are continuously optimized and updated.

[0024] In a more specific technical solution, the GIOU_Loss loss function adds the overlapping area, center point distance, and aspect ratio between the prediction box and the target box, including:

[0025]

[0026] Where IOU is the intersection-over-union ratio of the target anchor box and the predicted anchor box, v is a parameter to measure the consistency of the aspect ratio, Dis_a and Dis_b are the Euclidean distance between the predicted anchor box and the center point of the standard anchor box and the diagonal distance of the minimum circumscribed rectangle, respectively.

[0027] The GIOU_Loss loss function of the present invention measures the overlap between bounding boxes and also includes their closure areas, and can provide effective gradient updates even in the case of no overlap, which makes model training more stable.

[0028] In a more specific technical solution, the license plate recognition network is constructed based on the LPRNet network, the output sequence is obtained with maximum probability by using the beam search, the license plate recognition network is trained with the license plate recognition data set, the license plate character features are extracted with the LPRNet network, and the CTC_Loss loss function is continuously optimized to obtain the license plate recognition result, wherein the license plate recognition network includes:

[0029] STN localization network with spatial transformer layer, lightweight CNN backbone network, character classification head for each position, character classification probability of decoded sequence and post-filtering processing.

[0030] In a more specific technical solution, the overall network structure of the LPRNet includes:

[0031] Input layer, C in ×H×W feature map;

[0032] Convolutional layer, #C out / 4 1×1, step size is 1;

[0033] Convolutional layer, #C out / 4 3×1, the step length in height and width direction is 1;

[0034] Convolutional layer, #C out / 4 1×3, the step length in height and width direction is 1;

[0035] Convolutional layer, #C out 1×1, step size is 1;

[0036] Output layer, C out ×H×W feature map.

[0037] In a more specific technical solution, the small basic block structure of the backbone network of the LPRNet includes:

[0038] Input layer, 94×24 pixel RGB image;

[0039] Convolutional layer, #64, 3×3, stride 1;

[0040] Max pooling layer, #64, 3×3, stride 1;

[0041] Small basic block, #128, 3×3, step size 1;

[0042] Max pooling layer, #64, 3×3, stride (2,1);

[0043] Small basic block, #256, 3×3, step size 1;

[0044] Small basic block, #256, 3×3, step size 1;

[0045] Max pooling layer, #64, 3×3, stride (2,1);

[0046] Dropou, 0.5;

[0047] Convolutional layer, #256, 4×1, stride 1;

[0048] Dropout, 0.5;

[0049] Convolutional layer, #number of classes, 1×13, stride 1.

[0050] In a more specific technical solution, the license plate recognition data set is used to train the license plate recognition network, and the LPRNet network is used to extract license plate character features. The process includes:

[0051] Construct an STN positioning network with a spatial transformer layer, a lightweight CNN backbone network, character classification heads for each position, character classification probabilities for the decoded sequence, and an LPRNet network for post-filtering processing;

[0052] Randomly initializing the license plate recognition network;

[0053] The license plate recognition model is trained by continuously optimizing the CTC_Loss loss function;

[0054] Inputting the license plate image of the license plate recognition data set, training the license plate recognition network, and obtaining the optimal license plate recognition network model;

[0055] Extracting image features through the convolutional layer in the license plate recognition network;

[0056] In a more specific technical solution, the CTC_Loss loss function includes:

[0057]

[0058] Among them, M represents the input, W represents the probability obtained by path C after k transformations, and t represents the time.

[0059] The CTC_Loss loss function of the present invention can be trained end-to-end without segmenting characters, thereby greatly reducing the workload of data alignment and labeling, and avoiding the errors caused by segmenting characters.

[0060] In a more specific technical solution, a license plate detection and license plate recognition system is characterized in that the system includes:

[0061] The license plate detection dataset construction module is used to collect and annotate vehicle images to construct a license plate detection dataset;

[0062] The yolov5 improved network construction module is used to construct an improved yolov5 network model, which is trained using the license plate detection dataset constructed above to obtain a license plate detection model, thereby detecting the license plate position and positioning it;

[0063] The license plate recognition data set construction module is used to crop the license plate from the image according to the location area coordinate information of the license plate through the relevant cropping function to obtain the license plate image, and then construct the license plate recognition data set;

[0064] The license plate recognition network module builds a license plate recognition network based on the LPRNet network, trains the license plate recognition network with the license plate recognition data set, obtains a license plate recognition model, extracts license plate character features, and obtains license plate recognition results.

[0065] Compared with the prior art, the present invention has the following advantages: First, the present invention adopts the CCPD data set and optimizes the annotation to construct a high-quality license plate detection data set. Based on the improved yolov5 network, the data processing, feature extraction and feature fusion capabilities are enhanced, and the accuracy and efficiency of license plate detection are significantly improved compared with the existing research. Then, the improved license plate detection model is used to identify and locate the license plate from the CCPD data set, the license plate area coordinates are obtained, and the license plate image is accurately extracted through the clipping function. Finally, the LPRNet network and CTC_Loss loss function are used, combined with the beam search algorithm, to optimize the license plate recognition process and improve the accuracy and robustness of character recognition. On the whole, the present invention has obvious advantages in the rapid positioning detection and high-accuracy recognition of license plates. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A model training flow chart of a license plate detection model and a license plate recognition model in an embodiment of the present invention;

[0067] Figure 2 A flow chart of a license plate detection and license plate recognition method in an embodiment of the present invention;

[0068] Figure 3 The figure is a schematic diagram of the license plate image cropping and extraction process in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to more clearly illustrate the purpose, technical solution and advantages of the present invention, several embodiments of the present invention are described in detail below. It should be clear that only some embodiments of the present invention are described here, not all. Based on these embodiments, all other embodiments that can be derived by ordinary technicians in the technical field without creative work are within the protection scope of the present invention.

[0070] Example 1

[0071] This implementation example provides a method for obtaining a license plate detection model based on an improved yolov5, including:

[0072] Step S1: Figure 1 As shown in the figure, a vehicle sample image dataset is collected and annotated using the LabelImg tool. The vehicle sample image dataset mainly comes from the blue license plate dataset CCPD2019 and the green license plate dataset CCPD2020, with a total of 10,000 images used for license plate detection.

[0073] Step S1.2, for the license plate detection dataset, 8,000 blue license plate images were randomly selected from CCPD2019 (the proportion of images in the Base scene was 70%, and the proportion of images in other scenes was 30%), and 2,000 green license plate images were randomly selected from CCPD2020. On the basis of the normal environment, pictures taken in 9 types of scenes such as rainy days and blur were also included, which enhanced the generalization ability of the license plate detection model.

[0074] Step S2: Build an improved yolov5 network, initialize the network weights, use the license plate detection dataset built in step S1, train after data enhancement, and stop after 60 epochs of training. The specific process is as follows:

[0075] Step S2.1, improve the original yolov5 network architecture; Mosaic data enhancement function is designed at the input end of yolov5, 4 images are randomly selected, and they are combined by cropping, deformation, placement, etc., so that the smallest batch can be appropriately reduced to reduce the amount of calculation; an adaptive image scaling function is designed to appropriately reduce the size to increase the training speed, so the standard image size used is 416×416×3; an adaptive anchor frame calculation function is designed. During training, the network will get the predicted frame based on the given anchor frame, compare the real frame with it, calculate the error value between the two, and then update it in reverse, so that the network parameters are continuously optimized and updated; the skeleton of yolov5 The stem network designs the Focus structure and CSPNet structure to enhance the ability to extract image features. After the image is input in the form of 416×416×3, it is first converted into a 208×208×12 feature-oriented image after slicing, and then becomes a 208×208×32 image after a convolution operation with a convolution kernel number of 32; the CSP2_X structure introduced in the neck network of yolov5 further enhances the ability of feature fusion. After inputting an image of a specified size of 416×416×3, it outputs 52×52, 26×26 and 13×13 feature tensors through multiple downsampling, which correspond to the large scale, medium scale and small scale in multi-scale target detection respectively;

[0076] In this embodiment, the GIOU_Loss loss function is used to add the overlapping area, center point distance, and aspect ratio between the prediction box and the target box, which are defined as follows:

[0077]

[0078] The GIOU_Loss loss function measures the overlap between bounding boxes and their closure areas, providing effective gradient updates even in the case of no overlap, which makes model training more stable.

[0079] Step S2.2, construct the yolov5 improved network and initialize the parameters of the pre-trained model;

[0080] Step S2.3, using the image samples of the license plate detection data set, inputting the improved network of yolov5, first performing data augmentation operation, and then starting training, stopping after completing the set 60 epochs training, and extracting the weights of the model;

[0081] Example 2

[0082] This implementation example provides a method for building a license plate recognition network based on an LPRNet network, including:

[0083] Step S1: Figure 1 As shown, a license plate recognition dataset is constructed, and the trained license plate detection model is used to detect the license plate position in the vehicle image, and the license plate is located and marked to obtain the position information of the license plate. The license plate is cropped out of the image through the relevant cropping function to obtain the license plate image;

[0084] Step S1.1, the license plate is cropped from the picture through the relevant cropping function to obtain the license plate image. Specifically, the license plate is cropped from the picture through the crop() function of the pillow module in Python. Finally, all the license plate character information is used as the picture name and saved as a picture of size 94×24.

[0085] Step S2: Build a license plate recognition network based on LPRNet, which consists of an STN positioning network and a lightweight CNN backbone network. Randomly initialize the license plate recognition network and input the license plate recognition data set in step S1 for training. During the training process, the CTC_Loss loss function is used for continuous optimization. After 60 eporch are completed, the license plate recognition model is obtained.

[0086] Step S3: extract image features through the convolutional layer in the license plate recognition network constructed in step S2, and finally output the license plate number.

[0087] Example 3

[0088] This implementation example provides a license plate detection and license plate recognition method, which uses a pre-trained improved YOLOv5 license plate detection model and an LPRNet license plate recognition model to achieve fast and accurate license plate processing. This embodiment uses a pre-trained YOLOv5 model to detect the license plate of the input vehicle image, and the model can accurately locate the license plate area in the image. The detected license plate area is then accurately cropped and saved as a separate image file. These license plate images are then fed into the pre-trained LPRNet license plate recognition network. The LPRNet network recognizes the characters in the license plate image through its lightweight CNN backbone network and STN positioning network, and outputs the final license plate number. This embodiment directly uses a powerful pre-trained model for rapid detection and accurate recognition, which is suitable for real-time license plate processing applications, and improves processing efficiency and practicality.

[0089] The steps of the license plate detection and license plate recognition method are as follows: Figure 2 As shown, including:

[0090] Step S1: parsing the video to be processed obtained in traffic management to obtain the required image;

[0091] Step S2: Input the image into the yolov5 model, scale the image size to the standard size of 416×416×3 through adaptive image scaling, output the image feature tensor after a series of slicing and convolution operations of the backbone network, and finally obtain the detection frame and license plate position information of the license plate area based on GIOU_Loss loss function optimization and non-maximum suppression calculation;

[0092] Step S3: Figure 3 As shown, according to the license plate position information, the license plate is cropped from the picture to obtain the license plate image, which is saved as a picture of size 94×24;

[0093] Step S4: extracting image features from the license plate image through a license plate recognition network based on LPRNet;

[0094] Step S5: outputting the license plate number based on the extracted license plate image features;

[0095] In this embodiment, the recognition network uses a lightweight convolutional neural network (CNN) backbone, combined with a spatial transformer network (STN) to accurately locate the license plate position, showing a high degree of robustness when processing license plate images under different angles and conditions. In addition, LPRNet does not require predefined character segmentation steps, and directly learns and recognizes characters end-to-end from the entire license plate image, which greatly simplifies the processing flow and improves the recognition speed.

[0096] In summary, the license plate detection and license plate recognition method proposed in the present invention has several key steps in its process. First, the video to be processed obtained in traffic management is parsed to extract the required images. Then, these images are input into the improved yolov5 model, in which the adaptive image scaling technology adjusts the image to the standard size of 416×416×3, and through a series of slicing and convolution operations, the GIOU_Loss loss function and non-maximum suppression technology are used to accurately locate the license plate area. After that, according to the positioning information, the license plate is accurately cropped and saved as an image of 94×24 size. Then, these images are processed by the license plate recognition network based on LPRNet, and finally the license plate number characters are output. The network adopts a lightweight CNN backbone and STN, which not only improves the ability to accurately locate the license plate position, but also due to its end-to-end character recognition design, no predefined character segmentation steps are required, which significantly simplifies the processing flow and speeds up the recognition speed. Overall, this method effectively improves the processing efficiency and accuracy by integrating advanced license plate detection and recognition technologies, and is suitable for real-time traffic management systems.

Claims

1. A license plate detection and license plate recognition method, characterized in that: The following steps are involved: Step 1: Dataset construction: Use the CCPD dataset and annotate it, and construct the dataset through image enhancement processing; Step 2: Improve YOLOv5 license plate detection model training: Improve the yolov5 network by adding functions such as data enhancement and adaptive anchor frame calculation. Use the constructed license plate detection dataset to train the improved yolov5 network to obtain the license plate detection model. Step 3: License plate detection: Use the improved YOLOv5 model trained in step S2 to detect the license plate position and locate it, obtain the location information of the license plate, and use the relevant cropping function to crop the license plate from the image to obtain the license plate image; Step 4: Build the LPRNet license plate recognition network: Build the license plate recognition network based on the LPRNet network, use beam search and CTC_Loss loss function to train the license plate recognition network, extract the character features in the license plate image, and output the license plate number.

2. The license plate detection and license plate recognition method according to claim 1, characterized in that: The improved yolov5 network includes the functions of setting Mosaic data enhancement, adaptive image scaling and adaptive anchor frame calculation at the input end.

3. The license plate detection and license plate recognition method according to claim 1, characterized in that: The cropping function used in the license plate detection step is the crop() function in Python's Pillow module.

4. The license plate detection and license plate recognition method according to claim 1, characterized in that: The LPRNet license plate recognition network includes a spatial transformer layer (STN), a lightweight CNN backbone network, a character classification head, and a character classification probability module for a decoding sequence.

5. The license plate detection and license plate recognition method according to claim 1, characterized in that: The CTC_Loss loss function is used to optimize the character recognition network during the training process without pre-defining the character segmentation step.

6. A system applicable to the license plate detection and license plate recognition method according to any one of claims 1 to 5, characterized in that: The system comprises: The dataset construction unit is used to collect and annotate vehicle images to construct a license plate detection dataset; the improved YOLOv5 model training unit is used to make improvements based on the yolov5 network and use the constructed dataset for model training; the license plate detection unit is used to use the trained YOLOv5 model to detect the license plate position and crop the license plate image; the license plate recognition network unit is used to use the LPRNet network to extract the license plate character features and output the license plate number.

7. The system according to claim 6, characterized in that The license plate detection unit includes a Mosaic data enhancement module, an adaptive picture scaling module and an adaptive anchor frame calculation module, which are used to enhance image features and improve detection accuracy.

Images