License plate recognition system and method

The system enhances license plate recognition by using neural networks and grammar-based identification to improve accuracy and speed under varying environmental conditions.

CN114333338BActive Publication Date: 2024-12-06REALTEK SINGAPORE PTE LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011544026.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-09-30
Filing Date
2020-12-23
Publication Date
2024-12-06
Estimated Expiration
2040-12-23

AI Technical Summary

Technical Problem

Existing license plate recognition technologies suffer from low tolerance to environmental factors such as varying shooting angles, light interference, and plate deformations, leading to decreased accuracy and recognition speed.

Method used

A system comprising an image capture module, license plate detection, character segment extraction, classification, and recognition modules, utilizing neural networks for image processing and grammar-based character identification, with optional correction, to enhance accuracy and speed.

Benefits of technology

The system provides improved tolerance to environmental variations, maintaining high accuracy and fast recognition speed by accurately identifying characters on license plates despite deformations and lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114333338B_ABST
    Figure CN114333338B_ABST
Patent Text Reader

Abstract

A license plate recognition system includes an image capture module, a license plate detection module, a character segment extraction module, a character classification module and a character recognition module. The image capture module is used to capture an image. The license plate detection module is used to receive the image and detect the license plate in the image. The character segment extraction module is used to extract a plurality of character segments on the license plate. The character classification module is used to calculate the probability of each possible character in the character segments. The character recognition module is used to identify the selected character of the character segment based on the predetermined syntax of the character segments and the probabilities of the plurality of possible characters of the character segment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to image processing, and more particularly to a vehicle license plate recognition system and method. Background Art

[0002] In the application of image processing, license plate recognition technology is well known. However, in practical applications, due to different shooting angles or interference from light sources, day and night, sunny and rainy environments, the license plate image may have unclear license plate features, skewed license plates, deformed license plates, light noise, and broken, stained, and decorated license plates, which will cause the recognition accuracy to decrease. In addition, the existing license plate positioning technology usually finds the license plate image position based on the edge density value. If the license plate is stained or decorated, the characteristics of the edge density value may be destroyed, resulting in a significant decrease in the accuracy of license plate positioning. Furthermore, if the obtained license plate is too skewed or deformed, it will also make character cutting difficult to perform, and the license plate must be corrected through additional algorithms. The above factors show that the existing license plate recognition technology has a low tolerance for the environment, and various additional image processing technologies must be used to improve the recognition rate, but this will also reduce the speed of license plate recognition. Therefore, how to provide a better license plate recognition method to improve the tolerance of license plate recognition to the environment and maintain high accuracy and fast recognition speed under the condition of fixed hardware resources is a problem that must be solved at present. Summary of the invention

[0003] An embodiment provides a license plate recognition system, including an image capture module, a license plate detection module, a character segment extraction module, a character classification module and a character recognition module. The image capture module is used to capture an image. The license plate detection module is used to receive the image and detect the license plate in the image. The character segment extraction module is used to extract a plurality of character segments on the license plate. The character classification module is used to calculate the probability of each possible character of one of the character segments. The character recognition module is used to identify a selected character of the character segment based on a predetermined grammar of the character segments and the probabilities of a plurality of possible characters of the character segment.

[0004] The embodiment further provides a method for license plate recognition, including capturing an image, identifying a license plate in the image, extracting a plurality of character segments from the license plate, calculating the probability of each possible character of one of the character segments, and identifying a selected character of the character segment based on a predetermined grammar of the character segments and the probabilities of the plurality of possible characters of the character segment.

[0005] The embodiment further provides a method for extracting multiple character segments in a license plate, including detecting the license plate from an image, converting the license plate into a grayscale image, binarizing the grayscale image to produce a binary image, performing binary segmentation on the binary image to identify multiple character segments in the license plate, if the grayscale values ​​of the multiple characters are greater than the grayscale values ​​of the background of the license plate, calculating the number of the character segments, and if the number of the character segments is not substantially equal to an expected value, binary inverting the binarized image. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 Schematic diagram of a license plate recognition system in an embodiment.

[0007] Figure 2 yes Figure 1 A flow chart of a method in which a character segment extraction module is used to extract character segments from a license plate.

[0008] Figure 3 yes Figure 1 Schematic diagram of the character segment extraction module extracting character segments from a license plate.

[0009] Figure 4A and Figure 4B yes Figure 2 Schematic diagram of performing image binarization and debinarization in the method.

[0010] Figure 5 Schematic diagram of a binary image of a license plate in an embodiment.

[0011] Figure 6 It is a schematic diagram of characters and backgrounds marked with different labels in an embodiment.

[0012] Figure 7 yes Figure 1 Schematic diagram of the character classification module using model matching to obtain similarity.

[0013] Figure 8 yes Figure 1 Schematic diagram of the character classification module calculating character probabilities.

[0014] Fig. 9 yes Figure 1 A schematic diagram of a character recognition module identifying an identifier according to a predetermined grammatical rule.

[0015] Fig.10 yes Figure 1 Schematic diagram of the correction processing performed by the correction module.

[0016] Fig.11 yes Figure 1 Flow chart of a license plate recognition method executed by a license plate recognition system. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] In the following detailed description, reference may be made to the various specification drawings that are part of the present application and are used to illustrate specific embodiments of the present application. In the accompanying drawings, similar reference numerals describe substantially similar components in different figures. The various specific embodiments of the present application are described below in sufficient detail so that a person of ordinary skill in the art with relevant knowledge and skills in the art can implement the technical solutions of the present application. It should be understood that other embodiments may also be used or structural, logical or electrical changes may be made to the embodiments of the present application.

[0019] Figure 1 1 is a schematic diagram of a license plate recognition system 100 in an embodiment. The license plate recognition system 100 includes an image capture module 105, a license plate detection module 110, a character segment extraction module 120, a character classification module 130 and a character recognition module 140. The license plate recognition system 100 may also optionally include a correction module 150.

[0020] The image capture module 105 is used to capture images. The license plate detection module 110 is used to receive images and detect license plates in the images. The character segment extraction module 120 is used to extract multiple character segments on the license plate. The character classification module 130 is used to calculate the probability of each possible character of one of the multiple character segments. The character recognition module 140 is used to identify the selected character of the character segment based on the predetermined syntax of the multiple character segments and the probability of the multiple possible characters of the character segment. For example, the multiple possible characters of the character segment can be the multiple allowed characters of the character segment according to the predetermined syntax, and the character recognition module 140 can identify the character with the highest probability among the allowed characters as the selected character of the character segment. The correction module 150 is used to correct the selected character according to the maximum number of times the character appears in the same position of the multiple license plates.

[0021] In an embodiment, the image capture module 105 can be a camera or any device capable of capturing images. The license plate detection module 110, the character segment extraction module 120, the character classification module 130, the character recognition module 140 and the correction module 150 can all include neural network models trained for the above functions.

[0022] Figure 22 is a flow chart of a method 200 for extracting a character segment from a license plate by the character segment extraction module 120. The method 200 comprises the following steps:

[0023] S202: Detecting a license plate from an image;

[0024] S204: converting the license plate into a grayscale image;

[0025] S206: Binarizing the grayscale image to generate a binary image;

[0026] S208: performing binary segmentation on the binary image to identify multiple character segments in the license plate;

[0027] S210: Assuming that the grayscale value of the plurality of characters is greater than the grayscale value of the background of the license plate, calculate the number of the plurality of character segments; and

[0028] S212: Is the number of the plurality of character segments equal to the expected value? If yes, execute step S216; if no, execute step S214;

[0029] S214: perform binary inversion on the binary image, and return to step S208;

[0030] S216: End the process.

[0031] First, the license plate detection module 110 extracts the license plate according to the position information in the image. The character segment extraction module 120 converts the license plate color image into a grayscale image. The grayscale image is converted into a binary image by the character segment extraction module 120 using a default binarization threshold. Then, binary segmentation is performed to generate segmentations of each character and its bounding box. Next, the character segment extraction module 120 identifies possible character segments based on the size of the character segment, the aspect ratio of its bounding box, and the aspect ratio of the license plate. Other parts, such as plate frames, bolts, and nuts, are then deleted. If the number of character segments obtained matches the expected value of the number of character segments obtained from the rule grammar, character classification processing is then performed by the character classification module 130. Otherwise, the reverse binarization threshold is applied to the grayscale image to generate an inverted binary image. Then, binary segmentation is performed on the inverted binary image.

[0032] If the number of possible character segments is substantially not equal to the expected value of the number of character segments, it can be confirmed that the grayscale value of the license plate background is higher than the grayscale value of the characters; if the number of possible character segments is substantially equal to the expected value of the number of character segments, it can be confirmed that the grayscale value of the license plate background is lower than the grayscale value of the characters. This ensures that the character segment extraction module 120 can correctly identify any type of license plate.

[0033] Figure 33 is a schematic diagram of the character segment extraction module 120 extracting character segments from a license plate. Grayscale image 310 is a grayscale image of the license plate. Binarized image 320 can be generated by binarizing grayscale image 310. Character segment extraction module 120 then extracts character segments from binarized image 320. Image segment 330 includes each extracted character segment, and each character is surrounded by its bounding box.

[0034] FIG. 4A and FIG. 4B are schematic diagrams of performing image binarization and inverse binarization in method 200. As shown, low pixel values ​​represent low grayscale values, while high pixel values ​​represent high grayscale values. FIG. 4A shows a binarization threshold applied to convert grayscale image 310 into a binarized image 320. If the grayscale value is above the threshold, for example, the grayscale value is above the threshold 120, it is converted into a high pixel value in the binarized image 320. If the grayscale value is below the threshold, it is converted into a low pixel value in the binarized image 320. FIG. 4B shows an inverse binarization threshold applied to convert grayscale image 310 into a binarized image 320. If the grayscale value is below the threshold, it is converted into a high pixel value in the binarized image; conversely, if the grayscale value is above the threshold, it is converted into a low pixel value in the binarized image.

[0035] Figure 5 is a schematic diagram of a binary image of a license plate in an embodiment. In an embodiment, the characters have high grayscale values, while the background has low grayscale values. When the binary pixel values ​​of the characters are known, a binary segmentation method can be applied to generate character segments corresponding to each character. The embodiment can apply the method described in U.S. Patent Application No. 16 / 655,212 to generate character segments.

[0036] Figure 6 Schematic diagram of characters and backgrounds marked with different labels in the embodiment. In the fast binary segmentation process, each pixel will be marked with a label. Pixels with the same label are identified as belonging to the same character. For example, the pixel with label 10 belongs to the character "5", while the pixel with label 0 belongs to the background.

[0037] Figure 77 is a schematic diagram of the character classification module 130 using model matching to obtain similarity. The character segment 730 is transmitted to the similarity calculation module 720 for comparison with the characters in the reference character database 710 to obtain similarity. For example, when comparing with all the characters in the reference character database 710, the similarity calculation module 720 can calculate the similarity of each character in the reference character database 710 and the characters in the character segment 730 to generate the probability that each character in the reference character database 710 matches the characters in the character segment 730, and then select the character with the highest probability as the selected character of the character segment 730. The reference character database 710 has a variety of commonly used character models, for example: these character models can be compared with a variety of commonly used fonts in the region that governs license plates. The implementation of the similarity calculation module 720 is as follows: scaling each character model and one of the character segments 730 in the reference character database 710 so that they have the same number of pixels (i.e., the size of each character model and the character segment 730 have the same number of pixels), and then calculating the number of pixels with the same pixel value at the same position, and dividing the value obtained by the total number of pixels as the similarity of the characters. In addition, in order to increase the classification accuracy of the character classification module 130, the more characters that are more likely to be misclassified (e.g., 5, 6, 8, 9), the more commonly used character models are in the reference character database 710; conversely, the less likely characters are to be misclassified (e.g., 0, 1, 4, 7), the fewer character models are in the reference character database 710 to save memory space of the reference character database 710. For another example, the reference character database 710 has eight character models of the character "9", so eight similarities of the character "9" are calculated respectively, and the largest similarity among the eight similarities is used as the similarity of the character "9".

[0038] In some embodiments, the characters in the reference character database 710 include character models of the same character at different angles. For example, a character model of a character rotated from -90° to +90° along the center; or a character model of the character observed after squinting at certain angles. Such character models obtained in a similar manner are all within the scope of the disclosure of this application. This angle-transformed character model can not only assist, for example, in the recognition of license plates in a picture captured by a camera that is higher than the driving height of the car; it can also enhance the recognition effect of license plates in situations where, for example, the license plates are skewed or deformed. In some embodiments, the reference character database 710 can further add damaged or deformed character models. The recognition effect of license plates can be enhanced in situations where, for example, the license plates are damaged or deformed.

[0039] Figure 83 is a schematic diagram of the character classification module 130 calculating the character probability. In an embodiment, the character classification module 130 includes a neural network model for calculating the character probability, including a neural network and weight parameters. The neural network model can learn the weights after training the model using exemplary data, and the image segment 330 is adjusted to an image with a width of 32 pixels and a height of 24 pixels. For example, inputting 32x24x1 pixel values ​​into the neural network model can generate the probability of each possible character.

[0040] Fig. 9 It is a schematic diagram of the character recognition module 140 according to a predetermined grammatical rule identifier. The predetermined grammar may be related to the laws and regulations of the area governing the license plate. For example, according to the license plate regulations of Singapore, the number of characters ranges from 4 to 8 characters. In this figure, "A" means that the character at this position must be a letter. "D" means that the character at this position must be a number. "A / D" means that the character at this position can be a letter or a number. The license plate number begins with a letter. The letters "I" and "O" are not allowed in the license plate. The next few characters can be numbers between 1 and 9999, and the numbers cannot start with 0. The last character is a letter as a check, which can be "F", "I", "N", "O", "Q" or "V". Therefore, the grammatical rules described above can help the character recognition module 140 to correctly identify characters in the appropriate position.

[0041] Fig.10 is a schematic diagram of the correction process performed by the correction module 150. The correction module 150 is a selective module that can improve the accuracy of the recognition result. The image captured by the image capture module 105 can be a current frame of a dynamic image. There may be previous frames in the dynamic image that have undergone the license plate recognition process and are saved in the database of the correction module 150. Therefore, the license plate recognition result of the current frame can be compared with the recognition result of the previous frame, and correction can be performed on each character position. The character that appears the most times can be recognized as the correct character. This can improve the accuracy of recognition.

[0042] Fig.11 is a flow chart of a license plate recognition method 1100 performed by the license plate recognition system 100. The method 1100 includes the following steps:

[0043] S1102: Capturing images;

[0044] S1104: recognizing a license plate in an image;

[0045] S1106: extracting a plurality of character segments from the license plate;

[0046] S1108: Calculate the probability of each possible character in a character segment among the plurality of character segments;

[0047] S1110: Identify a plurality of allowed characters for the character segment according to a predetermined grammar of the plurality of character segments;

[0048] S1112: Selecting the character with the highest probability among the plurality of allowed characters as the selected character of the character segment; and

[0049] S1114: Correct the selected character based on the character that appears most times at the same position.

[0050] The calibration step S1114 is optional, but performing the calibration can improve the accuracy of the recognition result.

[0051] In summary, the license plate recognition system and method provided in the embodiment can quickly and accurately recognize the characters on the license plate. Therefore, compared with the prior art, the system and method provide better license plate recognition results and improve the tolerance of license plate recognition to the environment while maintaining high accuracy and fast recognition speed.

[0052] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

[0053]

Explanation of symbols

[0054] 100: License Plate Recognition System

[0055] 105: Image capture module

[0056] 110: License plate detection module

[0057] 120: Character segment extraction module

[0058] 130: Character classification module

[0059] 140: Character recognition module

[0060] 150: Correction module

[0061] 310: Grayscale image

[0062] 320: Binarized image

[0063] 330: Video Clip

[0064] 710: Reference character database

[0065] 720: Similarity calculation module

[0066] 730: character segment

[0067] 200,1100:Method

[0068] S202~S216, S1102~S1114: Steps

Claims

1. A license plate recognition system, include: An image capture module, used to capture an image; a license plate detection module, for receiving the image and detecting a license plate in the image; a character segment extraction module for detecting the license plate from an image; Converting the license plate into a grayscale image; Binarizing the grayscale image to generate a binary image, wherein the binary image has a plurality of pixels, and each pixel is marked with a label; The binary image is subjected to binary segmentation to extract multiple character segments on the license plate, and pixels with the same label are identified as belonging to the same character; Calculating a number of the character segments according to a comparison between the grayscale values ​​of the plurality of characters and the grayscale value of the background of the license plate; and If the number of the character segments is substantially not equal to an expected value, performing binary inversion on the binarized image; a character classification module for calculating the probability of each possible character of a character segment among the character segments; and A character recognition module is used to identify a selected character of the character segment according to a predetermined grammar of the character segments and the probabilities of the possible characters.

2. The system according to claim 1, further comprising: include: A correction module is used to correct the selected character according to the maximum number of times the character appears at the same position.

3. The system as claimed in claim 1, wherein the background of the license plate has one of a high pixel value or a low pixel value, and the plurality of characters of the license plate have the other of a high pixel value or a low pixel value.

4. The system as claimed in claim 1, wherein the license plate detection module comprises a neural network model.

5. The system of claim 1, wherein the character recognition module calculates the probability of each possible character in the character segment by comparing each possible character in the character segment with reference characters in a database.

6. The system of claim 1, in, The predetermined grammar corresponds to a license plate regulation.

7. A method for license plate recognition, include: capturing an image; recognizing a license plate in the image; Extracting a plurality of character segments from the license plate includes: converting the license plate into a grayscale image; Binarizing the grayscale image to generate a binary image, wherein the binary image has a plurality of pixels, and each pixel is marked with a label; The binary image is subjected to binary segmentation to extract multiple character segments on the license plate, and pixels with the same label are identified as belonging to the same character; Calculate the number of these character segments; If the number of the character segments is substantially not equal to an expected value, performing binary inversion on the binarized image; Calculating the probability of each possible character in a character segment among the character segments; A selected character is selected as the character segment according to a predetermined grammar of the character segments and the probability of possible characters.

8. The method according to claim 7, further comprising: include: The selected character is corrected based on a character that appears the most times at the same position.

9. The method according to claim 7, in: The background of the license plate has one of a high pixel value and a low pixel value, and the plurality of characters of the license plate have the other of a high pixel value and a low pixel value.

10. The method according to claim 7, in, The predetermined grammar corresponds to a license plate regulation.

11. The method of claim 7, wherein identifying the license plate in the image is performed using a neural network model to identify the license plate in the image.

12. The method of claim 7, wherein the probability of each possible character in the character segment in the character segments is calculated by comparing each possible character in the character segment with reference characters in a database.

13. A method for extracting multiple character segments from a license plate. include: detecting the license plate from an image; Converting the license plate into a grayscale image; Binarizing the grayscale image to generate a binary image, wherein the binary image has a plurality of pixels, and each pixel is marked with a label; Perform binary segmentation on the binary image to identify multiple character segments in the license plate, and pixels with the same label are identified as belonging to the same character; Calculating a number of the character segments according to a comparison between the grayscale values ​​of the plurality of characters and the grayscale value of the background of the license plate; and If the number of the character segments is substantially not equal to an expected value, the binary image is binary inverted.

14. The method according to claim 13, further comprising: include: The sizes of the character segments are compared with the size of the license plate to confirm that the character segments are valid character segments.

15. The method according to claim 13, further comprising: include: If the number of the character segments is substantially not equal to the expected value, then confirming that the grayscale value of the background of the license plate is higher than the grayscale values ​​of the characters; If the number of the character segments is substantially equal to the expected value, it is determined that the grayscale values ​​of the characters are higher than the grayscale value of the background of the license plate.