License plate recognition method based on simplified yolov5 combined with halcon

Abstract

The application discloses a license plate recognition method based on combination of simplified yolov5 and halcon, and comprises the following steps: image acquisition: a picture containing a license plate is collected through a camera; license plate positioning: a simplified yolov5 network is used to identify the picture to generate the position of the license plate, i.e. the coordinates of the license plate; license plate preprocessing: the license plate picture is segmented through the license plate coordinates to obtain an image containing only the license plate, then image enhancement, license plate flip correction and license plate character region approximation are performed, and finally a region and an image containing only characters are obtained; character segmentation: each character of the license plate is segmented; OCR recognition: according to the characters extracted from the character segmentation module, OCR recognition is performed on the characters from left to right one by one, and finally the string of the license plate number is output. The application avoids the influence of a complex environment on license plate positioning, and simultaneously proposes a'step-by-step approximation' character segmentation method according to the similar size of license plate characters, greatly increases the robustness of the algorithm, and enables the algorithm to well recognize the license plate.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to a license plate recognition method based on a simplified combination of YOLOv5 and Halcon. Background Technology

[0002] Current license plate recognition mainly includes non-character segmentation recognition based on neural networks and character segmentation license plate recognition based on digital image processing. Existing technologies include research on license plate localization and recognition based on the YOLO deep learning algorithm and research on license plate detection algorithms under unconstrained conditions based on improved YOLO v5. Both require training with tens of thousands of images, with the former undergoing 1800 training iterations, resulting in massive data and long training times. Additionally, there is research on license plate recognition based on HALCON, which uses RGB images to convert to HSV images to extract the license plate, but it does not consider vehicles with similar colors to the license plate. HALCON-based license plate recognition technology uses histograms for segmentation; however, the histogram for each vehicle is not identical, and it cannot handle situations where multiple vehicles appear in the same frame simultaneously. Summary of the Invention

[0003] To address the shortcomings of the existing technology, the technical solution of this invention is a license plate recognition method based on a simplified combination of YOLOv5 and Halcon, comprising the following steps:

[0004] S10, Image Acquisition: Acquire images containing license plates using a camera;

[0005] S20, License Plate Localization: The image is recognized using a simplified YOLOv5 network to generate the location of the license plate, i.e., the coordinates of the license plate;

[0006] S30, License plate preprocessing: The license plate image is segmented by license plate coordinates to obtain an image containing only the license plate. Then, image enhancement, license plate flip correction and license plate character region approximation are performed to finally obtain a region and image containing only characters.

[0007] S40, Character Segmentation: Segments each character of the license plate;

[0008] S50, OCR recognition: Based on the characters extracted by the character segmentation module, OCR recognition is performed one by one from left to right, and finally the license plate number string is output.

[0009] Preferably, in S20, the simplified YOLOv5 network for license plate localization includes:

[0010] A pair of Conv->C3 modules were deleted from backbone;

[0011] The neck part is simplified by changing the three-layer feature pyramid structure in the original YOLOv5 network into a two-layer feature pyramid structure.

[0012] The k-means++ algorithm is used to set the initial anchor box values.

[0013] Preferably, S30, license plate preprocessing, specifically includes the following steps:

[0014] S31, convert the image to grayscale, extract the license plate, and perform image enhancement on the license plate;

[0015] S32, preliminary threshold segmentation, that is, first apply mean filtering to the license plate, then perform adaptive threshold segmentation, and finally filter by the area of ​​the region and the minimum bounding rectangle 2 of the region, that is, the length and width of the bounding rectangle in any direction to remove impurities.

[0016] S33, Rotate and straighten the license plate: Extract the character area based on the size and direction of the characters, and then calculate the average slope based on the coordinates of the center point of the character area to calculate the required rotation angle;

[0017] S34, perform adaptive threshold segmentation on the corrected license plate, break down the resulting region, then find the minimum horizontal bounding rectangle of the broken region and eliminate interference;

[0018] S35, based on the principle that the area and width of the horizontal bounding rectangle of each character are similar, find the region that may be the character;

[0019] S36, extract the region containing the character and crop out the character region.

[0020] Preferably, the halcon operators used in S31 include rgb1_to_gray, reduce_domain, and emphasize.

[0021] Preferably, in step S32, before adaptive threshold segmentation, adaptive license plate color judgment is performed. Adaptive threshold segmentation is performed on the license plate. In Halcon, the operator is dyn_threshold, and its last parameter is either 'light' or 'dark'. The color of the license plate is determined by comparing the areas of the two regions obtained by changing the last parameter. If the area obtained by the 'light' parameter is smaller than that obtained by the 'dark' parameter, it is a blue license plate; otherwise, it is a green license plate.

[0022] Preferably, the halcon operators used in S33 include count_obj, select_obj, smallest_rectangle2, remove_obj, and area_center.

[0023] Preferably, the halcon operators used in S34 include mean_image, dyn_threshold, connection, smallest_rectangle1, gen_smallest_rectangle1, and select_shape.

[0024] Preferably, the halcon operators used in S35 include tuple_remove, tuole_solt, and select_shape.

[0025] Preferably, the halcon operators used in S36 include union1, smallest_rectangle1, gen_rectangle1, and corp_domain.

[0026] Preferably, in step S40, character segmentation involves performing adaptive threshold segmentation on the cropped image to obtain character regions, and then using a connection operator to break them down, eliminating regions smaller than the threshold area.

[0027] The beneficial effects of this invention are as follows: This invention proposes a simplified YOLO model for license plate localization, which can avoid the influence of complex environment on license plate localization and does not require a large number of images for training. At the same time, it proposes a "stepwise approximation" character segmentation method based on the similarity of license plate character size, which greatly increases the robustness of the algorithm and enables it to recognize license plates well. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the steps of a license plate recognition method based on a simplified combination of YOLOv5 and Halcon, according to an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the existing YOLOv5 network structure;

[0030] Figure 3 This is a simplified YOLOv5 model diagram of the license plate recognition method based on the combination of simplified YOLOv5 and Halcon according to an embodiment of the present invention.

[0031] Figure 4 This is a schematic diagram of the S31 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0032] Figure 5 This is a schematic diagram of the S32 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0033] Figure 6 This is a schematic diagram of the S33 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0034] Figure 7 This is a schematic diagram of the S34 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon in an embodiment of the present invention.

[0035] Figure 8 This is a schematic diagram of the S35 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0036] Figure 9 This is a schematic diagram of the S36 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0037] Figure 10 This is a schematic diagram of the S40 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0038] Figure 11 This is a schematic diagram of the S50 process and effect of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon according to an embodiment of the present invention.

[0039] Figure 12 This is a schematic diagram of the model folder structure used for OCR recognition in S50 of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon in an embodiment of the present invention.

[0040] Figure 13 shows the F1 score and confidence curves of the license plate recognition method based on the simplified combination of YOLOv5 and Halcon in the prior art and the embodiments of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0043] See Figure 1 The technical solution of this invention, as an embodiment of the invention, is a flowchart of a license plate recognition method based on a simplified combination of YOLOv5 and Halcon, including the following steps:

[0044] S10, Image Acquisition: Acquire images containing license plates using a camera;

[0045] S20, License Plate Localization: The image is recognized using a simplified YOLOv5 network to generate the location of the license plate, i.e., the coordinates of the license plate;

[0046] S30, License plate preprocessing: The license plate image is segmented by license plate coordinates to obtain an image containing only the license plate. Then, image enhancement, license plate flip correction and license plate character region approximation are performed to finally obtain a region and image containing only characters.

[0047] S40, Character Segmentation: Segments each character of the license plate;

[0048] S50, OCR recognition: Based on the characters extracted by the character segmentation module, OCR recognition is performed one by one from left to right, and finally the license plate number string is output.

[0049] The license plate localization method of this invention is based on the YOLOv5 object detection algorithm. YOLOv5 has four models: YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x, representing small, medium, large, and super-large license plates, respectively. As the model size increases, the mesh size decreases, increasing detection accuracy but significantly reducing training speed. This invention is a modification of YOLOv5s, which has the smallest model size and the fastest training speed. The four versions of YOLOv5 share the same network structure, consisting of an input network, a backbone network, a neck network, and a head network. See [link to relevant documentation]. Figure 2 .

[0050] Yolo's input performs image preprocessing, including Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling. Mosaic data augmentation involves randomly selecting four images, randomly scaling, randomly cropping, and randomly flipping them, and then stitching them together into a single image. This enriches the dataset. Adaptive image scaling resizes the images to 640×640 pixels, padding the shorter side with zeros if necessary.

[0051] The backbone mainly consists of the regular convolutional modules Conv and C3, and finally an SPFF module. The Conv modules (except for the first Conv) use 3×3 kernels with a stride of 2, batch normalization with BatchNorm2d, and the SiLu activation function. Their function is to extract and organize feature maps, performing downsampling, normalization, and adding non-linearity. The C3 module, composed of three Conv modules and a Bottleneck module, is a more important feature extraction module. The SPFF module is a spatial pyramid pooling module, primarily used to fuse multi-scale features.

[0052] The Neck module is a feature pyramid structure that combines shallower features obtained from the backbone with deeper semantic features to obtain more complete features and improve the accuracy of object detection.

[0053] The Head module consists of three detect modules, which perform the detection function.

[0054] S20, a simplified YOLOv5 network for license plate localization, includes: removing a pair of Conv->C3 modules from the backbone; simplifying the neck part by reducing the original three-layer feature pyramid structure of the YOLOv5 network to a two-layer feature pyramid structure; and using the k-means++ algorithm to set the initial anchor box values. See also... Figure 3 Since YOLO only performs license plate localization and doesn't need to be highly precise—as long as the bounding box can enclose the license plate—a simplified version of the YOLOv5 model was implemented to reduce the number of model parameters while still ensuring accurate license plate detection. Experiments have shown that this simplification method effectively reduces the number of model parameters without significantly decreasing detection accuracy, although a slight decrease in accuracy may occur.

[0055] S30, license plate preprocessing, specifically includes the following steps:

[0056] S31 converts the image to grayscale, extracts the license plate, and performs image enhancement on the license plate; the Halcon operators used in this process are rgb1_to_gray, reduce_domain, and embeddage. See also Figure 4 The diagram illustrates its process and effects.

[0057] S32, preliminary threshold segmentation, involves first applying mean filtering to the license plate, then adaptive threshold segmentation, and finally filtering by the area of ​​the region and the minimum bounding rectangle 2 (the length and width of the bounding rectangle in any direction) to remove impurities. Before adaptive threshold segmentation, adaptive license plate color determination is performed. In Halcon, the operator is `dyn_threshold`, whose last parameter is either 'light' or 'dark'. The color of the license plate is determined by comparing the areas of the two regions obtained after changing the last parameter. If the area obtained with the 'light' parameter is smaller than that obtained with the 'dark' parameter, it is a blue license plate; otherwise, it is a green license plate. See also... Figure 5 The diagram illustrates its process and effects.

[0058] S33, Rotation and Alignment of License Plate: Based on the size and orientation of the characters, the character region is extracted. Then, the average slope is calculated based on the coordinates of the center point of the character region to determine the required rotation angle. Halcon operators used in this process include count_obj, select_obj, smallest_rectangle2, remove_obj, area_center, etc. See [link / reference needed]. Figure 6 The diagram illustrates the process and its effects. The functions `delete_long_rec2`, `delete_small_area`, `find_nearest_number`, and `figure_out_phi` are user-defined functions. `delete_long_rec2` deletes regions where the longer side is more than three times the shorter side. `delete_small_area` deletes small areas to eliminate interference. `find_nearest_number` finds the most frequent value in the input array, which can be an area, length, angle, etc., and returns its average. For example, to find the average area of ​​a character, the input parameters are an array of all area values ​​and 'area'. Since the areas of the smallest bounding rectangle of a character are close, this function finds these close values, calculates their average, and outputs the result. The `figure_out_phi` function calculates the average angle required for rotation from the center of the obtained character region.

[0059] S34: Adaptive threshold segmentation is performed on the corrected license plate, and the resulting region is broken down. Then, the minimum horizontal bounding rectangle of the broken region is calculated, and interference is eliminated. The main Halcon operators used in this process are mean_image, dyn_threshold, connection, smallest_rectangle1, gen_smallest_rectangle1, and select_shape. See also... Figure 7 The diagram illustrates its process and effects.

[0060] S35. Based on the principle that the area and width of the horizontal bounding rectangle of each character are similar, identify the regions that might contain the character. Before starting this step, eliminate interference from regions with large aspect ratios. The aspect ratio of the character's bounding rectangle is generally around 2, so we can first eliminate regions with aspect ratios greater than 2.5 or less than 0.5 (here, length refers to the length of the region on the y-axis, and width refers to the length of the region on the x-axis). Next, eliminate the largest area. If the second largest area is more than twice the size of the third largest area, eliminate that as well. If the third largest area is more than twice the size of the fourth largest area, eliminate that as well. After these three layers of elimination, we can eliminate regions much larger than the area of ​​the character's horizontal bounding rectangle. After eliminating some small areas, we can find the region containing the character's horizontal bounding rectangle. Finally, based on the similarity of widths, we further determine the region where the character is located. The Halcon operators used in this process mainly include tuple_remove, tuole_solt, and select_shape. See also... Figure 8 The diagram illustrates the process and its effects. The functions `delete_long_rec1`, `delete_large_area`, `delete_small_area`, and `find_nearest_number` are all user-defined functions. The purpose of `delete_long_rec1` is to delete long rectangles. While it can accurately extract the horizontal bounding rectangles of characters, it cannot guarantee that the horizontal bounding rectangles of all characters can be extracted, so further extraction is required.

[0061] S36: Extract and crop the region containing the characters. This step mainly determines the number of missing characters based on the number of rectangular regions obtained in S35. The blue card actually has 7 characters, and the green card has 8. Then, for each missing character, add the corresponding number of characters to the left and right, adhering to the principle of "better to add too many than too few." The gaps between characters must also be considered; these gaps can be roughly calculated based on the ratio between the area on the image and the actual area. The main Halcon operators used in this process include union1, smallest_rectangle1, gen_rectangle1, and corp_domain. See also... Figure 9 The diagram illustrates its process and effects.

[0062] S40, Character Segmentation: Adaptive thresholding is applied to the cropped image to obtain character regions. Then, a connection operator is used to break them down, excluding areas smaller than the threshold. Because Chinese characters have complex strokes, and some strokes are not connected, excluding small regions might reduce the stroke count and eliminate the entire character. The second character of a license plate can only be an English letter, excluding "I" and "O." English letters are connected and have simple strokes, so the second character can be used as a starting point. First, find the second character, then move one character to the left plus the width of a gap to find the Chinese character. Finally, look to the right for other characters; blue plates have 5 characters, and green plates have 6. This way, all the characters on the license plate can be found.

[0063] The second character can be found by determining the distance between the center points of adjacent characters, since the distance between the second and third characters is 2 to 3 times the distance between the centers of other characters. The opening operation can also be used to exclude some consecutive characters. The Halcon operators used in this process mainly include dyn_threshold, opening_circle, connection, remove_obj, select_obj, etc. See also... Figure 10 The diagram illustrates its process and effects.

[0064] S50, OCR character recognition. Characters extracted from S40 character segmentation are recognized one by one from left to right using OCR. Since the first character can only be Chinese and the second character can only be an English letter, to improve accuracy, a model containing only Chinese characters is used for one character, a model containing only English characters is used for the second character, and a model combining numbers and English letters is used for the remaining characters. During the recognition process, a "white background with black text" image is first created based on the characters to improve accuracy. Model training will be described in detail later. The main process of this step can be found in [link to relevant documentation]. Figure 11 .

[0065] So far, the entire license plate recognition is basically completed. By inputting the recognition result into the detect module of yolov5, the detected result can be displayed. The following describes the method for training the model used in the OCR recognition process:

[0066] First, create a folder on the desktop with a structure like Figure 12 Then, store the pictures containing characters in the corresponding folders. Put the character '京' into the folder '00京'.

[0067] Then, create a for loop that can loop 3 times. The first time, train the pictures in the folder '0 Chinese characters', the second time, train the pictures in the folder '1 Letters', and the third time, train the pictures in the folder '2 Numbers and letters'. It should be noted that during the training process, pictures with a white background and black characters also need to be created for training, rather than directly using the original pictures, so as to exclude the influence of factors such as lighting. After training, three models will be generated, which are used to recognize Chinese characters, the second character, and the remaining other characters in the license plate respectively. In fact, Chinese characters, letters, and numbers can also be trained together to make a model, but the recognition accuracy of the model obtained in this way is not as high as that of the three models obtained by subdivision. Therefore, three models are adopted here.

[0068] Experiments and Analyses of the Present Invention

[0069] Data Collection

[0070] Part of the image data required for license plate positioning comes from pictures downloaded from the Internet and part comes from pictures taken by mobile phones. There are a total of 68 training pictures, including 6 background pictures, that is, pictures of cars without license plates, and 17 verification pictures, including 3 background pictures.

[0071] The pictures used for OCR character recognition are all obtained by cropping pictures from the Internet. Among them, there are 124 Chinese character pictures (31 Chinese characters, 4 pictures for each character), 40 number pictures, and 96 English letter pictures, for a total of 260 pictures.

[0072] Experimental Equipment and Parameter Settings

[0073] The processor of the experimental server in this experiment is Intel(R) Core(TM) i5-8250U CPU@1.60GHz 1.80GHz, and the GPU is MX150. The parameter settings during training are --epochs 300 --batch-size 2, that is, the number of pictures passed into the neural network each time is 2, and it is trained for 300 rounds.

[0074] Comparison between the Simplified Model of the Present Invention and the Existing Technology Model

[0075] Figure 13 compares the F1 score and confidence curves of the simplified model of this invention with those of the prior art model. The F1 score curve represents the harmonic mean of precision and recall, ranging from 0 to 1. A value closer to 1 indicates better confidence inference performance. (a) shows the F1 score curve of the simplified model of this invention, and (b) shows the F1 score curve of the prior art model. The simplified F1 score of this invention is 0.687, while the prior art model has an F1 score of 0.664, indicating that the simplified model of this invention is slightly better than the prior art model. The confidence curve is directly proportional to precision, with higher precision being better. (c) shows the confidence curve of the simplified model of this invention, and (d) shows the confidence curve of the prior art model. The confidence score of the simplified model of this invention is 0.727, indicating that when the confidence score reaches 0.727, the model's inference accuracy reaches 1; the original model has a confidence score of 0.709. Here, the simplified model is also slightly better than the prior art model.

[0076] The simplified model of this invention is compared with the model of the prior art in terms of mAP50, mAP50-95, model size, and number of floating-point operations (GFLOPS) as shown in Table 1 below.

[0077] Table 1 Comparison between the simplified model of this invention and existing models.

[0078] mAP50 mAP50-95 Model size GFLOPS Simplified model 0.995 0.820 3.71MB 14.0 Existing technology model 0.995 0.852 13.74MB 16.6

[0079] As shown in the table, the simplified mAP50-95 of this invention is slightly smaller than the existing model, but the model size is much smaller and the GFLOPS is also smaller, making it more suitable for embedded small devices.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A license plate recognition method based on a simplified combination of YOLOv5 and Halcon, characterized in that, Includes the following steps: S10, Image Acquisition: Acquire images containing license plates using a camera; S20, License Plate Localization: The image is recognized using a simplified YOLOv5 network to generate the location of the license plate, i.e., the coordinates of the license plate; S30, License plate preprocessing: The license plate image is segmented by license plate coordinates to obtain an image containing only the license plate. Then, image enhancement, license plate flip correction and license plate character region approximation are performed to finally obtain a region and image containing only characters. S40, Character Segmentation: Segments each character of the license plate; S50, OCR recognition: Based on the characters extracted by the character segmentation module, OCR recognition is performed one by one from left to right, and finally the license plate number string is output; S20, the simplified YOLOv5 network for license plate localization, includes: A pair of Conv->C3 modules were deleted from backbone; The neck part is simplified by changing the three-layer feature pyramid structure in the original YOLOv5 network into a two-layer feature pyramid structure. The k-means++ algorithm is used to set the initial anchor box values; S30, license plate preprocessing, includes the following steps: S31, convert the image to grayscale, extract the license plate, and perform image enhancement on the license plate; The halcon operators used in S31 include rgb1_to_gray, reduce_domain, and emphasize.

2. The method according to claim 1, characterized in that, S30, license plate preprocessing, also includes the following steps: S32, preliminary threshold segmentation, that is, first apply mean filtering to the license plate, then perform adaptive threshold segmentation, and finally filter by the area of ​​the region and the minimum bounding rectangle 2 of the region, that is, the length and width of the bounding rectangle in any direction to remove impurities. S33, Rotate and straighten the license plate: Extract the character area based on the size and direction of the characters, and then calculate the average slope based on the coordinates of the center point of the character area to calculate the required rotation angle; S34, perform adaptive threshold segmentation on the corrected license plate, break down the resulting region, then find the minimum horizontal bounding rectangle of the broken region and eliminate interference; S35, based on the principle that the area and width of the horizontal bounding rectangle of each character are similar, find the region that may be the character; S36, extract the region containing the character and crop out the character region.

3. The method according to claim 2, characterized in that, In step S32, before adaptive threshold segmentation, adaptive license plate color judgment is performed. Adaptive threshold segmentation is performed on the license plate. In Halcon, the operator is dyn_threshold, and its last parameter is either 'light' or 'dark'. The color of the license plate is determined by comparing the areas of the two regions obtained by changing the last parameter. If the area obtained by the 'light' parameter is smaller than that obtained by the 'dark' parameter, it is a blue license plate; otherwise, it is a green license plate.

4. The method according to claim 2, characterized in that, The halcon operators used in S33 include count_obj, select_obj, smallest_rectangle2, remove_obj, and area_center.

5. The method according to claim 2, characterized in that, The halcon operators used in S34 include mean_image, dyn_threshold, connection, smallest_rectangle1, gen_smallest_rectangle1, and select_shape.

6. The method according to claim 2, characterized in that, The halcon operators used in S35 include tuple_remove, tuole_solt, and select_shape.

7. The method according to claim 2, characterized in that, The halcon operators used in S36 include union1, smallest_rectangle1, gen_rectangle1, and corp_domain.

8. The method according to claim 2, characterized in that, In step S40, character segmentation involves adaptive threshold segmentation of the cropped image to obtain character regions, which are then broken down using a connection operator to exclude regions smaller than the threshold area.

Images