Single-layer and double-layer character rapid reconstruction method in license plate recognition process

Abstract

The invention provides a single-layer and double-layer character rapid reconstruction method in a license plate recognition process. The method comprises the following steps: firstly, detecting the position, the type and the angular point of a license plate in a vehicle image through a deep learning network model; then affine transformation correction is carried out on the license plate image to eliminate shooting inclination; inputting the corrected image into a character detection network, and obtaining positions, contents and confidence information of all characters at one time; then, dynamically judging whether the license plate is of a single-layer structure or a double-layer structure according to the longitudinal distribution statistical characteristics of the character center points; if the license plate is a double-layer license plate, the license plate is divided into an upper layer and a lower layer according to character longitudinal coordinates, and sequential splicing reconstruction is performed after respective sorting; and if the license plate is a single-layer license plate, directly outputting in sequence. According to the method, detection of all characters is completed through single reasoning, intelligent judgment and reconstruction are carried out by utilizing a character space geometrical relationship, and the problems of repeated calculation, character relevance loss and order disorder caused by image cutting in a traditional method are effectively avoided.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and automatic vehicle license plate recognition technology, specifically to a method for rapid reconstruction of single and double-layer characters in the license plate recognition process. Background Technology

[0002] License plate recognition technology is one of the core technologies of intelligent transportation systems and has been widely used in scenarios such as road checkpoints, parking lot management, vehicle tracking, and automatic toll collection. A typical license plate recognition process includes steps such as license plate localization, image correction, character segmentation, character recognition, and result output.

[0003] Given that my country's vehicle license plates commonly have two structures—single-layer (for small vehicles) and double-layer (for large trucks, etc.)—existing recognition methods still face numerous challenges when dealing with double-layer license plates, mainly including: Character segmentation based on projection analysis: This method locates character regions by horizontally and vertically projecting binary license plate images, and performs well under conditions of uniform lighting and upright shooting. However, in complex real-world scenarios, license plate images often suffer from tilting, occlusion, and uneven lighting, leading to blurred projection peaks and valleys, character segmentation failure, and consequently affecting recognition accuracy.

[0004] Layered Recognition and Result Stitching Method: This method, upon detecting a double-layered license plate, first physically segments the license plate image into two independent upper and lower regions. Then, it calls character recognition models to perform recognition on each region separately, and finally stitches the two results together as the final output. The drawbacks of this approach are: it requires two complete character recognition inferences, increasing the system's computational overhead and latency; furthermore, it severs the spatial relationship between the upper and lower layer characters in the original image, easily leading to misclassification of character attribution, resulting in duplicate character output or missed recognition.

[0005] End-to-end deep learning recognition methods: In recent years, end-to-end recognition models based on deep neural networks (such as the Transformer architecture) have been applied to license plate recognition. These models directly output the entire license plate character sequence without explicit character segmentation steps. However, when the training data coverage is insufficient, such models are not robust to the arrangement of characters on double-layer license plates, especially the spatial interleaving of characters caused by the shooting angle, and are prone to character order disorder.

[0006] Therefore, existing technologies have significant shortcomings in terms of recognition accuracy, processing efficiency, and adaptability to complex scenarios when processing single and double-layer license plates, especially double-layer license plates. Thus, there is an urgent need for a novel recognition method that can efficiently and accurately distinguish between single and double-layer license plates and automatically complete the correct sorting and reconstruction of all characters with only a single character detection. Summary of the Invention

[0007] The purpose of this invention is to provide a rapid character reconstruction method for single and double-layer license plate recognition, addressing issues such as character misalignment, repetition, and inaccuracy in practical applications of single and double-layer license plate recognition. The aim is to efficiently and accurately separate the upper and lower layers of characters in single and double-layer license plates, thereby improving the overall recognition speed and accuracy.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A method for fast reconstruction of single and double-layer characters in license plate recognition includes the following steps: S1. Acquire vehicle images, detect and output license plate area images, license plate type, license plate confidence score, and the position information of the four corner points of the license plate through a license plate location detection network model; S2. If the confidence level of the license plate is higher than the first preset threshold, then perform affine transformation correction on the license plate area image based on the four corner point positions, and output a corrected license plate image with normalized size. S3. Input the corrected license plate image into the license plate character detection network model, output the bounding rectangle position, recognition confidence, character content and character type of each character, and construct a character information linked list; S4. Sort the characters in the character information chain list horizontally, and determine whether the license plate is a single-layer structure or a double-layer structure based on the vertical distribution statistical characteristics of all characters. S5. If it is determined to be a two-layer structure, then according to the vertical coordinate of the center point of each character, it is dynamically divided into an upper-layer character subset and a lower-layer character subset. After horizontally sorting the characters in the two subsets, they are merged in the order of top to bottom, and the reconstructed complete character sequence is output. S6. If it is determined to be a single-layer structure, extract the province Chinese characters from the character information chain, merge the remaining characters in horizontal order, and output the final recognition result.

[0009] Furthermore, the license plate location detection network model and the license plate character detection network model are deep learning models trained on different training tasks, including but not limited to Faster R-CNN, YOLOv5, and YOLOv8; the input size of the license plate location detection network model is 736×416 pixels, and the input size of the license plate character detection network model is 224×224 pixels.

[0010] Furthermore, the size normalization specifically includes: setting the width of the corrected license plate image to a uniform 224 pixels, while the height is set according to the license plate type, with a height of 70 pixels for a single-layer license plate and 112 pixels for a double-layer license plate.

[0011] Furthermore, the determination of the license plate structure based on the vertical distribution statistical features of all characters specifically includes the following steps: S41. Calculate the average height of the bounding rectangles of all characters in the character information linked list, and denot it as the average character height; S42. Obtain the minimum and maximum values ​​of the vertical coordinates of the center points of all characters, and calculate their vertical span; S43. If the vertical span is greater than the average character height, and the vertical coordinates of the center points of at least two characters are less than a dynamic threshold, then it is determined to be a double-layer license plate; otherwise, it is determined to be a single-layer license plate.

[0012] Furthermore, the dynamic threshold is determined as follows: a horizontal center line is determined based on the minimum and maximum values ​​of the character's horizontal coordinates; a straight line parallel to the line and offset by half the average character height is calculated based on the slope of the line containing the lower layer of characters on the license plate; the vertical coordinate of this straight line at the character's horizontal center line is the dynamic threshold.

[0013] Furthermore, the slope of the row containing the lower-level characters of the license plate is estimated in the following way: select the leftmost and rightmost characters in the horizontal position of the character information chain, and calculate the slope by connecting their center points.

[0014] Furthermore, the dynamic division of characters into upper-level character subsets and lower-level character subsets specifically includes: dividing according to the relative position of the vertical coordinate of the character center point and a baseline, wherein the baseline is parallel to the row of lower-level characters of the license plate and is located in the gap area between the upper and lower-level characters.

[0015] Furthermore, before step S4, the method further includes: traversing the character information linked list and deleting characters whose character recognition confidence is lower than a second preset threshold.

[0016] Furthermore, the extraction of province Chinese characters specifically includes: filtering characters of the type of province Chinese characters from the character information chain, and selecting the character with the highest recognition confidence as the province abbreviation in the output result.

[0017] As can be seen from the above technical solutions, the present invention has the following technical advantages compared with the prior art: 1. This invention uses a unified character detection network to detect all characters on a license plate at once, avoiding the redundant computation overhead and additional processing delay caused by image segmentation and two recognition inferences for double-layer license plates in traditional methods, thus significantly improving processing efficiency. It is especially suitable for embedded application scenarios with high real-time requirements. 2. This invention does not require physical image segmentation, but instead performs single / double layer determination and character layering based on the geometric spatial distribution information of all detected characters; it completely preserves the relative positional relationship between characters, fundamentally avoiding the loss of character context association caused by segmentation, and effectively solving the problems of character repetition, omission, and misorder; and combined with affine transformation correction, it can effectively resist the interference caused by license plate tilt. 3. The judgment and layering logic of this invention dynamically calculates the threshold based on the character statistical features of the current license plate instance, rather than using fixed parameters. This enables it to adapt to license plate images of different sizes and tilt levels, improving the generalization ability and stability of the method in different scenarios. At the same time, the introduction of a confidence filtering mechanism further ensures the overall reliability of the system when the detection quality of some characters is poor. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the steps in the single- or double-layer character rapid reconstruction method during the license plate recognition process of the present invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 This paper presents a method for fast reconstruction of single and double-layer characters in license plate recognition. This method integrates deep learning object detection with geometric feature-based logical judgment to achieve end-to-end fast and accurate recognition and reconstruction of single-layer and double-layer license plate characters. The following section combines... Figure 1 The specific implementation steps of this method are explained in detail. Example 1

[0021] This embodiment describes a typical license plate recognition and character reconstruction process, applicable to fixed scenarios such as road checkpoints or parking lot entrances and exits.

[0022] S1. License Plate Detection and Corner Location: Acquire vehicle images captured by the front-end camera. Input the images into a pre-trained license plate location detection network model (Net_plate). The model output includes: a label indicating the presence or absence of a license plate, a license plate region image (plate_image_raw), a license plate type (plate_type, such as "single-layer" or "double-layer"), a license plate detection confidence score (plate_conf), and the pixel coordinates of the four vertex corners of the license plate region: P1(x1,y1), P2(x2,y2), P3(x3,y3), and P4(x4,y4). If no license plate is detected, the process terminates.

[0023] The license plate location detection network model and license plate character detection network model described in this invention include, but are not limited to, Faster R-CNN, YOLOv5, and YOLOv8. In the implementation of this invention, YOLOv8 is specifically selected as an example of the network inference framework for license plate location detection and license plate character detection: Traditional machine vision-based target detection models are prone to false positives or false negatives when dealing with targets of different sizes, especially small targets. This invention addresses this issue by breaking away from traditional version iteration patterns with the YOLOv8 network model design philosophy. The YOLOv8 model is termed a state-of-the-art (SOTA) model (details can be found in the model information released by Ultralytics). This model builds upon previous YOLO versions and introduces new features and improvements, further enhancing performance and flexibility. Its C2f module, through cross-stage feature fusion, can better handle targets of different scales. By improving the feature fusion method, it excels particularly in small target detection, enhancing the detection capability for multi-scale targets. In license plate recognition, single and double-layer license plates, as well as license plate characters (including province abbreviations, letters, and numbers), all fall under the category of small target detection. Therefore, this invention places great emphasis on the accuracy of small target detection in images. The YOLOv8 model can well meet the requirements for license plate and character detection in the license plate recognition process.

[0024] Specifically, the captured image dataset is scaled proportionally to the same pixel size used for model inference, and then labeled using annotation software (such as LabelMe, LabelImg, etc.). Finally, the labeled image samples are divided into training, test, and validation sets in an 8:1:1 ratio, and Mosai data augmentation is enabled during training to improve the generalization ability of the image sample data. It is particularly important to note the following when training the deep learning network model in this example: (1) The size of the license plate location detection network model and the image input size are 736×416 pixels.

[0025] (2) The size of the license plate character detection network model and the image input size are 224×224 pixels.

[0026] S2. Confidence Filtering and Affine Transform Correction: Set the first preset threshold to 0.5. Determine if plate_conf is greater than or equal to 0.5. If not, the detection result is considered unreliable, and the process terminates; if yes, license plate image correction is performed.

[0027] The purpose of license plate correction is to eliminate the inclination of the license plate plane caused by the shooting angle. Specifically, according to the pixel coordinates of the four vertices obtained, a 3×2 affine transformation matrix T is calculated using the cv2.getAffineTransform() function in the OpenCV library:

[0028] Subsequently, the cv2.warpAffine() function and the matrix T are used to perform an affine transformation on the original license plate image plate_image_raw to obtain the corrected front view image plate_image_crt. The specific calculation process is as follows: plate_image_crt = cv2.warpAffine(plate_image_raw, T, (plate_width, plate_height)) The size of the corrected image is standardized as follows: the width (plate_width) is fixed at 224 pixels; the height (plate_height) is dynamically set according to the license plate type, 70 pixels for single-layer license plates and 112 pixels for double-layer license plates. This standardization process provides a uniformly sized input for subsequent character detection.

[0029] S3. Character Detection and Linked List Construction: The normalized corrected license plate image plate_image_crt is input into a pre-trained license plate character detection network model (Net_string). This model is preferably trained with an input size of 224×224 pixels and can detect and recognize each character in the license plate simultaneously. For each detected character, the model outputs a structure information, including: the position of the character bounding rectangle str_box (x, y, w, h), the character recognition confidence str_conf, the character content value str_val (such as "京", "A", "7"), and the character type str_type (such as "province Chinese character", "letter", "number"). Here, x, y, w, and h respectively represent the abscissa of the starting point, ordinate, character width, and character height of the character in the plate_image_crt image.

[0030] Then, the structure information of all characters is stored in a linked list, denoted as STR_LIST. If STR_LIST is empty, it means no characters are detected and the process terminates.

[0031] Step S4. Character Preprocessing and Single / Double-Layer Structure Judgment: S41. Sorting: Traverse STR_LIST and sort all characters in ascending order according to their horizontal position in the image (i.e., the abscissa of the center point str_xval) to obtain an initial sequence.

[0032] Since each character in the linked list STR_LIST has been reconstructed in ascending order, all characters are arranged from left to right according to the pixel abscissa of the coordinate positions. And since there are generally 2 characters (province abbreviation Chinese character and city letter) in the upper layer of the double-row license plate characters, the first and last characters of the lower layer characters are in the front and last positions of the license plate characters in the linked list STR_LIST. According to the pixel coordinates of the center points of the first and last characters in the linked list STR_LIST, the slope value of the line where the lower layer characters of the license plate are located can be calculated for the plate_ structure information. Given that the lines where the upper and lower layer characters of the double-layer characters in the license plate are parallel, the present invention uses the plate_k of the line where the lower layer characters are located to represent the slope of the line where the license plate characters are located. Subsequently, the single / double-layer determination logic is executed: S42. Statistic calculation: Calculate the average value str_avgh of the heights of the circumscribed rectangles of all characters; Find the minimum value str_ymin and the maximum value str_ymax of the ordinate str_yval = y + h / 2 of the center points of all characters, and calculate the vertical span ; Find the minimum value str_xmin and the maximum value str_xmax of the abscissa str_xval of the center points of all characters, and calculate the abscissa of the horizontal center line .

[0033] S43. Estimate the slope of the line where the lower layer characters are located: Select the first and last characters (i.e., the leftmost and rightmost characters) in the sorted STR_LIST, and calculate the slope of the line connecting their center points as the estimated value of the slope plate_k of the line where the lower layer characters are located: ; S44. Dynamic threshold calculation and determination: Calculate the dynamic threshold threshold_y. This threshold represents the ordinate of a theoretical demarcation line for distinguishing the upper and lower layers in the image. The calculation formula is: ; This formula describes the ordinate of a line with a slope of plate_k and located half a character height above the center of the vertical distribution of the characters at the horizontal center x_center.

[0034] Determination logic: If span_y > str_avgh and there are at least 2 characters with str_yval < threshold_y, then determine that the license plate is a double-layer license plate; otherwise, determine it as a single-layer license plate.

[0035] The core principle of this determination is that the upper and lower rows of characters of a double-layer license plate are significantly separated in the vertical direction, the vertical span is greater than the average height of the characters, and the upper layer characters are concentrated in the top area of the image.

[0036] Step S5, Double-layer license plate character reconstruction: If it is determined in step S4 that the license plate is double-layer, traverse the sorted STR_LIST, and according to the comparison of the vertical coordinate str_yval of the center point of each character with the dynamic threshold threshold_y, divide it into two subsets: If str_yval < threshold_y, then this character belongs to the upper character subset (UPPER_LIST).

[0037] If str_yval ≥ threshold_y, then this character belongs to the lower character subset (LOWER_LIST).

[0038] Sorting and merging within the layer: Sort the characters in UPPER_LIST and LOWER_LIST respectively in ascending order according to the horizontal coordinate str_xval of their center points. Take out all the characters in the sorted UPPER_LIST in order, and then take out all the characters in the sorted LOWER_LIST in order, and jointly form a new and complete character sequence, and use this new sequence to update STR_LIST.

[0039] S6, Generation and output of the final result: Whether the license plate is single-layer or double-layer, after the foregoing steps, the character sequence sorted according to the correct logic has been stored in STR_LIST. Next, perform the final arrangement: Extract the Chinese character of the province: Traverse the final STR_LIST, and move the characters with the character type str_type being "province Chinese character" into a temporary linked list PROV_STR_LIST.

[0040] Select the optimal Chinese character of the province: Select the character with the maximum recognition confidence (str_conf) from PROV_STR_LIST as the first character (i.e., the abbreviation of the province) of the license plate result.

[0041] Assemble the final string: Concatenate the selected Chinese character of the province with all the remaining characters in STR_LIST (keeping their order in STR_LIST) in sequence to form the final license plate recognition result string plate_output.

[0042] Validity verification and output: Check the number of characters in plate_output. For Chinese standard license plates, the valid result usually has no less than 7 characters. If the number of characters is insufficient or PROV_STR_LIST is empty, it is determined that this recognition is invalid and an empty result is output; otherwise, plate_output is output. Embodiment 2

[0043] This embodiment, based on Embodiment 1, further illustrates the implementation of the character confidence filtering enhancement step. After constructing the STR_LIST in step S3 and before sorting and judging in step S4, the following steps are inserted: traverse the STR_LIST and check the recognition confidence str_conf of each character. Set the second preset threshold to 0.3. If the str_conf of a certain character is lower than 0.3, the character recognition result is considered unreliable, and it is deleted from the STR_LIST. The purpose of this step is to remove low-confidence noise detection results before the subsequent judgment and reconstruction logic is executed. This can effectively improve the accuracy of the calculation of statistical quantities such as average height and coordinate extreme values, thereby making single and double layer judgment and character layering more robust, especially suitable for license plate scenes with poor image quality, partial occlusion, or dirt.

[0044] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for fast reconstruction of single and double-layer characters in license plate recognition process, characterized in that, Includes the following steps: S1. Acquire vehicle images, detect and output license plate area images, license plate type, license plate confidence score, and the position information of the four corner points of the license plate through a license plate location detection network model; S2. If the confidence level of the license plate is higher than the first preset threshold, then perform affine transformation correction on the license plate area image based on the four corner point positions, and output a corrected license plate image with normalized size. S3. Input the corrected license plate image into the license plate character detection network model, output the bounding rectangle position, recognition confidence, character content and character type of each character, and construct a character information linked list; S4. Sort the characters in the character information chain list horizontally, and determine whether the license plate is a single-layer structure or a double-layer structure based on the vertical distribution statistical characteristics of all characters. S5. If it is determined to be a two-layer structure, then according to the vertical coordinate of the center point of each character, it is dynamically divided into an upper-layer character subset and a lower-layer character subset. After horizontally sorting the characters in the two subsets, they are merged in the order of top to bottom, and the reconstructed complete character sequence is output. S6. If it is determined to be a single-layer structure, extract the province Chinese characters from the character information chain, merge the remaining characters in horizontal order, and output the final recognition result.

2. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 1, characterized in that, The license plate location detection network model and the license plate character detection network model are deep learning models trained on different training tasks, including but not limited to Faster R-CNN, YOLOv5, and YOLOv8; the input size of the license plate location detection network model is 736×416 pixels, and the input size of the license plate character detection network model is 224×224 pixels.

3. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 2, characterized in that, The size normalization specifically includes: setting the width of the corrected license plate image to a uniform 224 pixels, while the height is set according to the license plate type, with a height of 70 pixels for a single-layer license plate and 112 pixels for a double-layer license plate.

4. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 1, wherein determining the license plate structure based on the vertical distribution statistical features of all characters specifically includes the following steps: S41. Calculate the average height of the bounding rectangles of all characters in the character information linked list, and denot it as the average character height; S42. Obtain the minimum and maximum values ​​of the vertical coordinates of the center points of all characters, and calculate their vertical span; S43. If the vertical span is greater than the average character height, and the vertical coordinates of the center points of at least two characters are less than a dynamic threshold, then it is determined to be a double-layer license plate; otherwise, it is determined to be a single-layer license plate.

5. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 4, characterized in that, The dynamic threshold is determined as follows: a horizontal center line is determined based on the minimum and maximum values ​​of the horizontal coordinates of the characters; a straight line parallel to the line and offset by half the average character height is calculated according to the slope of the line containing the lower layer of characters on the license plate; the vertical coordinate of this straight line at the horizontal center line of the characters is the dynamic threshold.

6. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 5, characterized in that, The slope of the row containing the lower layer characters of the license plate is estimated in the following way: Select the leftmost and rightmost characters in the horizontal position of the character information chain, and calculate the slope by connecting their center points.

7. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 1, characterized in that, The specific steps of dynamically dividing the characters into an upper-level character subset and a lower-level character subset include: dividing the characters according to the relative position of the vertical coordinate of the character's center point and a baseline, wherein the baseline is parallel to the row containing the lower-level characters of the license plate and is located in the gap area between the upper and lower-level characters.

8. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 1, characterized in that, Before step S4, the method further includes: traversing the character information linked list and deleting characters whose character recognition confidence is lower than a second preset threshold.

9. The method for rapid reconstruction of single and double-layer characters in the license plate recognition process according to claim 1, characterized in that, The extraction of province characters specifically includes: filtering characters of province type from the character information chain, and selecting the character with the highest recognition confidence as the province abbreviation in the output result.

Images