System, method and device for license plate recognition

Abstract

A method, device and system for license plate recognition are proposed to provide an LPR method that is effective with low requirements on image resolution. A system (100) for license plate recognition includes a first device (10) configured to capture an image of a vehicle (50), and a second device (20), wherein the second device (20) is configured to obtain an image of a license plate (60) from the image of the vehicle (50), and recognize the license plate (60) using a model, wherein the model is trained using an annotated image of the license plate, and the annotated image is generated based on an image of characters and an image of a scene. Because the model is trained using the generated annotated image of the license plate, time-consuming manual annotation can be omitted, which saves time and improves efficiency. With a well-trained and accurately annotated model, license plates can be recognized even based on images with lower resolution.

Description

Technical Field

[0001] The present invention relates to object recognition technology, and more particularly to a system, method and device for license plate recognition (LPR). Background Art

[0002] LPR is a technology that recognizes license plates, partitions and characters from images. Traditional LPR technology requires high image resolution and the correction recognition rate is not high. Summary of the invention

[0003] In a solution to the license plate recognition problem, an effective LPR method with lower image resolution requirements is provided.

[0004] According to a first aspect of the present disclosure, there is provided a system for license plate recognition, comprising:

[0005] - a first device configured to: capture an image of a vehicle, obtain an image of a license plate of the vehicle from the image of the vehicle, and send the image of the license plate to a second device;

[0006] - A second device configured to recognize a vehicle's license plate from the image of the license plate using a model, wherein the model is trained using annotated images of the license plate and the annotated images are generated based on an image of the characters and an image of the scene.

[0007] According to a second aspect of the present disclosure, a method for license plate recognition is provided, which can be performed by a second device of the present disclosure and includes:

[0008] - receiving an image of a vehicle's license plate;

[0009] - Recognizing a license plate from the image of the license plate using a model, wherein the model is trained using annotated images of the license plate and the annotated images are generated based on an image of the characters and an image of the scene.

[0010] According to a third aspect of the present disclosure, there is provided a device for license plate recognition, comprising:

[0011] - a receiving module configured to receive an image of a vehicle's license plate;

[0012] - A recognition module configured to recognize a license plate from the image of the license plate using a model, wherein the model is trained using annotated images of the license plate and the annotated images are generated based on an image of the characters and an image of the scene.

[0013] According to a fourth aspect of the present disclosure, there is provided a device for license plate recognition, comprising:

[0014] - at least one processor; and

[0015] - At least one memory, connected to the at least one processor, for storing computer executable instructions, which, when executed, cause the at least one processor to perform the method of the second aspect above.

[0016] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, which stores executable instructions, and when the executable instructions are executed by a computer, the computer can execute the method of the second aspect.

[0017] With the provided solution, time-consuming manual annotation can be omitted because the model is trained using the generated annotated images of license plates, which saves time and improves efficiency. With a well-trained and accurately annotated model, license plates can be recognized even based on lower resolution images.

[0018] In an embodiment of the present disclosure, the image of the scene is from the same or similar type of scene as the scene from which the image of the vehicle is from, which will improve the accuracy of the model when using the model to identify objects in images of similar / same type of scenes, and the recognition success rate can be increased.

[0019] In an embodiment of the present disclosure, when recognizing a vehicle's license plate from an image of the license plate, the procedure may include the following steps:

[0020] - Recognize characters on a license plate from an image of the license plate;

[0021] - Recognize the layout of characters on license plates;

[0022] -Get the position of each character on the license plate;

[0023] - Arranging the identified characters in the identified arrangement format and according to the obtained position of each character.

[0024] When each character on the license plate is recognized, and when the arrangement of the characters is post-processed, the license plate can be successfully recognized.

[0025] In an embodiment of the present disclosure, each character on the license plate is recognized from the image of the license plate via a bounding box; and if the ratio of the height of the bounding box of any character on one side of the license plate to the average height of the bounding boxes of all characters is higher than a predefined threshold, the recognition result is determined to be invalid. Therefore, the result of the license plate recognition can be double-checked to avoid erroneous results and improve the accuracy of the LPR.

[0026] According to a sixth aspect of the present disclosure, a method for model training is provided, comprising:

[0027] - obtaining first images, wherein each first image contains a first object;

[0028] -determining an arrangement format of the first object;

[0029] - generating a second image of the combination of the first object based on the first image and the arrangement format;

[0030] - placing said second image in a third image having a background with a contrast ratio with said second image above a predefined threshold.

[0031] - determining a first position of the second image in a third image;

[0032] - replacing the portion of the background in the third image with a fourth image of the scene to produce a fifth image;

[0033] - correlating the fifth image to the first position;

[0034] -Use the fifth image to train the model.

[0035] According to a seventh aspect of the present disclosure, a device for model training is provided, which may include:

[0036] - an image acquisition module, which is configured to acquire first images, wherein each first image contains a first object;

[0037] - an image generation module configured to generate a second image of a combination of the first object based on the first image and the arrangement format; place the second image in a third image of a background having a contrast with the second image above a predefined threshold; determine a first position of the second image in the third image; replace a portion of the background in the third image with a fourth image of the scene to generate a fifth image; and relate the fifth image to the first position;

[0038] - A training module configured to train a model using information of the first image and the first position.

[0039] According to an eighth aspect of the present disclosure, a device for model training is provided, which may include:

[0040] - at least one processor; and

[0041] - At least one memory, coupled to the at least one processor, for storing computer executable instructions, which when executed cause the at least one processor to perform the method of the sixth aspect.

[0042] According to a ninth aspect of the present disclosure, a computer-readable medium is provided, which stores executable instructions, and when the executable instructions are executed by a computer, the computer is enabled to perform the method of the sixth aspect.

[0043] By using the provided solution, images for training models are generated rather than collected, and the combination of objects and layout formats can be flexibly set and cover as many scenarios as possible, which will make the training model applicable to a variety of scenarios and images, so the recognition results through the training model can be more accurate and effective. In the case of high-contrast backgrounds, bounding boxes can be easily found, which will shorten the training time and increase the accuracy of the training model.

[0044] In an embodiment of the present disclosure, for each first object in the second image, the second position of the first object in the second image can be detected, and information of the second position can be generated, and the fifth image can be annotated with the identifier of each first object in the second image and the corresponding second position information.

[0045] With the provided solution, time-consuming manual annotation can be omitted, which saves time and improves efficiency. With a well-trained and accurately annotated model, license plates can be recognized even based on lower resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above-mentioned attributes and other features and advantages of the technology of the present invention and the implementation thereof will become more apparent, and the technology of the present invention itself will be better understood by referring to the following description of embodiments of the technology of the present invention in conjunction with the accompanying drawings, in which:

[0047] Figure 1 A system for license plate recognition of the present disclosure is described.

[0048] Figure 2A and Figure 2B A flow chart depicting license plate recognition of the present disclosure.

[0049] Figure 2C Recognition of an image of a license plate in an image showing a vehicle.

[0050] Figure 2D Recognition of characters in images showing license plates.

[0051] Figure 3 A flowchart depicting model training.

[0052] Figure 4 Image generation process showing the model trained for license plates.

[0053] Figures 5A to 5F Shows the images generated for training the model.

[0054] Fig. 6A Display an image of a row of license plates.

[0055] Figure 6B Display an image of two rows of license plates.

[0056] Fig. 7A Displays the original image of the character.

[0057] Figure 7B The detected contours of the characters shown in FIG7A are shown.

[0058] Figure 8 Shows the structure of a .txt file containing annotated information.

[0059] Fig. 9 Flowchart depicting license plate model training and license plate recognition.

[0060] Fig. 10A and Fig. 10B A block diagram showing an exemplary embodiment of an apparatus for license plate recognition of the present disclosure is depicted.

[0061] Fig.11A and Fig. 11B A block diagram showing an exemplary embodiment of an apparatus for model training of the present disclosure is depicted.

[0062] Reference Number:

[0063] 100,System for license plate recognition

[0064] 10. A first device for capturing an image of a vehicle

[0065] 20. Communication Server

[0066] 30. Second device for license plate recognition

[0067] 40, the third device for model training

[0068] 50. Vehicles

[0069] 60, license plate of vehicle 50

[0070] 200, Method for license plate recognition

[0071] S201 to S206, S2051 to S2054, steps of method 200

[0072] 300: Methods used for model training

[0073] S301 to S312, steps of method 300

[0074] 41: Character Image

[0075] 42: Image of license plate

[0076] 43: Background image

[0077] 44: Image of license plate in background

[0078] 45: Images from the scene

[0079] 46: Image of the license plate in the scene image

[0080] 47: Image of a license plate in a scene image with added noise

[0081] 900: Methods for model training and license plate recognition

[0082] 901: Images from the scene

[0083] 902: Image of character

[0084] 903: Generate license plate images and label the generated images

[0085] 904: Annotated images of license plates

[0086] 905: Pre-trained Model

[0087] 906: Training a model for license plate recognition

[0088] 907: Trained Model for License Plate Recognition

[0089] 908: Captured image of vehicle 50

[0090] 909: Detect license plate 60 from captured image, and detect numbers and letters from image of license plate 90

[0091] 910: Detected character

[0092] 911: Post-processing

[0093] 912: Recognized license plate

[0094] 301: Receiving module

[0095] 302: Identification module

[0096] 303: At least one storage

[0097] 304: At least one processor

[0098] 305: Communication module

[0099] 401: Image acquisition module

[0100] 402: Image generation module

[0101] 403: Training module

[0102] 404: At least one storage

[0103] 405: At least one processor

[0104] 406: Communication module DETAILED DESCRIPTION

[0105] Hereinafter, the above-mentioned features and other features of the present invention technology are described in detail. Various embodiments are described with reference to the drawings, wherein the same reference numerals are used throughout to refer to the same elements. In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It can be noted that the illustrated embodiments are intended to explain rather than limit the present invention. Obviously, such embodiments can be practiced without these specific details.

[0106] When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.

[0107] Referenced below Figure 1 7 describe the technology of the present invention in detail.

[0108] By introduction, Figure 1 A system 100 for license plate recognition of the present disclosure is depicted. The system 100 may include a first device 10 that captures an image of a vehicle within a location, such as a parked vehicle on a street. Figure 1 The vehicle 50 is shown, the first device 10 captures an image of the vehicle 50, and then the second device 20 in the system 100 attempts to recognize the license plate 60 from the image captured by the first device. The first device 10 may be installed in Figure 1 , at a certain height (e.g., 9 meters). The first device 10 may also be mounted on a building, etc., depending on the location to be monitored. In an embodiment of the present disclosure, the license plate 60 may be recognized at an angle α as small as 40 degrees and a distance d from the camera of the first device 10 to the license plate 60 as long as 20 meters.

[0109] The images described in this article can also be described as pictures or datasets. A dataset is a set of images that have been annotated.

[0110] In an embodiment of the present disclosure, the image of the vehicle may be sent to the second device 30 via the communication server 20 in the system 100. Optionally, the communication server 20 and the second device 30 may be deployed on the cloud. In contrast, the first device 10 is deployed on the edge side. The deployment of the second device 30 on the cloud can ensure that the second device 30 has sufficient computing resources to perform license plate recognition.

[0111] In an embodiment of the present disclosure, after capturing an image of the vehicle 50 (recorded as a "first image"), the first device 10 may detect whether there is a license plate in the first image. If so, the first device 10 may obtain an image of the license plate (i.e., the license plate 60 of the vehicle 60) (recorded as a "second image") and send it to the second device 30. After receiving the second image, the second device 30 may recognize the license plate 60 of the vehicle 50 from the second image using a model trained using annotated images of the license plate.

[0112] In an embodiment of the present disclosure, the system 100 may include a third device 40, which may also be deployed on the cloud. The third device 40 trains a model for recognizing a license plate of a vehicle. Optionally, the third device 40 and the second device 30 may be implemented in the same device or in separate devices.

[0113] Referring now to FIG. 2 , a flowchart of license plate recognition of the present disclosure is depicted. The method 200 may include the following steps:

[0114] S201 : Capturing an image of the vehicle 50 .

[0115] In this step, the first device 10 captures an image of the vehicle 50 . Figure 2C An image of vehicle 50 is shown.

[0116] S202: Obtain an image of the license plate 60 from the image of the vehicle 50. Figure 2C , the image within the bounding box around the license plate 60 is the image of the license plate 60 (recorded as “second image”).

[0117] In this step, the first device 10 attempts to detect the license plate 60 from the captured image of the vehicle 50. If detected, the first device 10 obtains an image of the license plate 60 from the image of the vehicle 50.

[0118] Instead of directly sending an image of the vehicle, the first device 10 first determines whether a license plate exists in the image of the vehicle 50, and if so, further obtains an image of the detected license plate 60 from the image of the vehicle 50, which can reduce the size of the image file to be transmitted, and by utilizing pre-processing on the edge side, the processing load on the cloud can be significantly reduced.

[0119] S203 : Send the image of the license plate 60 to the communication server 20 .

[0120] In this step, the first device 10 sends an image of the license plate. In the embodiment of the present disclosure shown in FIG. 2 , the first device 10 sends the image to the second device 30 via a communication server 20 such as an MQTT (Message Queuing Telemetry Transport) receiving server. In another embodiment of the present disclosure, the first device 10 may send the image directly to the second device 30.

[0121] S204 : Sending the image of the license plate 60 to the second device 30 .

[0122] In this step, the communication server 20 transmits the image of the license plate 60 to the second device 30 .

[0123] S205 : Recognize the license plate 60 of the vehicle 50 .

[0124] In this step, the second device recognizes the license plate 60. It can recognize the characters (including but not limited to letters and / or numbers) on the license plate 60 and organize the characters in the correct order. The details of this step can be referred to Figure 3 and related descriptions.

[0125] S206: Determine whether the recognition result is valid.

[0126] After the license plate 60 of the vehicle 50 is recognized in step S205, optionally, post-processing may be performed to determine whether the recognition result is valid, and the details of this step will be described later.

[0127] Reference Figure 3 , depicting a detailed flow chart of the above step S205. Step S205 may include the following sub-steps:

[0128] S2051 : Recognize characters on the license plate 60 of the vehicle 50 from the second image.

[0129] In this step, the second device 30 recognizes each character on the license plate 60, such as Figure 2D As shown, there is a bounding box around each recognized character. The second device 30 may use a model of the license plate for license plate recognition.

[0130] S2052: Identify the arrangement format of the characters on the license plate 60.

[0131] After recognizing each character, all characters should be arranged in the correct format. Thus, in this step, the second device 30 recognizes the arrangement format of the characters. Take the license plate of Hong Kong as an example, there is a row of ( Fig. 6A Shown in) and two rows ( Figure 6B For the license plate 60 used in Hong Kong, it must be determined whether the license plate is a single row or a double row license plate.

[0132] Feature extraction such as Hough transform may be performed to determine the arrangement format of the characters on the license plate 60. And if the license plate 60 is divided into two parts, each part is in a row.

[0133] S2053: Obtain the position of each character on the license plate 60.

[0134] In this step, the second device 30 can calculate the position of each character. For example, if a Hough transform is performed, the x coordinate can be used to obtain the position of each character in a row.

[0135] S2054: Arrange the recognized characters in the recognized arrangement format and according to the obtained position of each character.

[0136] In this step, the second device 30 may arrange the recognized characters according to the recognized arrangement format and character positions.

[0137] In step S206, post-processing may be performed to determine whether the recognition result is valid. To avoid random sequencing of numbers and characters and other erroneous recognition results, the following processing may be performed. Post-processing may include a) sequencing b) suppression and c) cross-checking.

[0138] a) Sequencing

[0139] If a Hough transform is performed to determine the layout format of the characters on the license plate 60, a local maximum search such as NMS (non-maximum suppression) can be performed for each row. A radius threshold can be set for NMS execution. Taking the license plate of Hong Kong as an example, if the license plate 60 is a one-row license plate and the number of characters exceeds 7 or is less than 3, the recognition result will be determined to be invalid and can be ignored; if the license plate 60 is a two-row license plate and the number of characters in each row is small (should not be less than 1), the recognition result will be determined to be invalid, or this row will be abandoned. The setting of the threshold value for the number of characters can be changed as needed. In addition, if the license plate 60 is a two-row license plate, it can also be determined whether the upper row is shorter than the lower row. If not, the recognition result can be determined to be invalid.

[0140] b) Inhibition:

[0141] Because the model for license plate recognition cannot be perfect, mismatch suppression can be performed to suppress mismatches on the edges of the license plate.

[0142] Optionally, if the ratio of the height of the bounding box of any character on one side of the license plate 60 to the average height of the bounding boxes of all characters is higher than a predefined value, the recognition result can be determined to be invalid. The ratio can be set to 30% or other values ​​and can be changed as needed.

[0143] Optionally, the slope of the characters can be checked. Take out a character in the license plate 60 and form a row with the remaining characters. If the taken character is on the row, the recognition result can be determined to be valid.

[0144] c) Cross-check:

[0145] A cross-check with character recognition can be performed. A score for each character can be calculated, and a threshold sc can be set. If the average of the scores of all characters plus sc is greater than the maximum score of each character, the recognition result can be determined to be valid.

[0146] In this way, the mismatch of the license plate can be reduced to less than 10% but there are still some situations where it does not work (for example, if there is no license plate in the second image, or the character score is too low due to obstacles or blur, or if there is more than one car in the second image).

[0147] In the above step S205, the second device 20 uses the model to recognize the license plate 60. It is often a huge challenge to obtain enough data to train an AI (artificial intelligence) model such as a model for license plate recognition. Data collection usually takes a lot of work. Even if the data is collected, it still takes a lot of work to properly label each data item so that the model can be properly trained. Current AI model training relies on manual data collection and labeling techniques. Parts of this manual process may have been automated, but the overall method is still very cumbersome and time-consuming.

[0148] In the embodiments of the present disclosure, the model can be trained using an annotated image of a license plate, and an annotated image can be generated based on an image of a character and an image of a scene. The advantage of correctly annotating a large amount of data can be easily obtained, and when sufficient data is correctly annotated, the trained model can be more accurate, so the accuracy of license plate recognition can be higher.

[0149] It is worth mentioning that the model training described can be used not only for license plate recognition, but also for other AI model training. Figure 3 A flow chart depicting model training that can be performed by the third device 40 mentioned above. Figure 3 The method 300 for model training may include the following steps, and the steps may be performed under the control of a computer program such as a python script to generate images and automatically add annotations.

[0150] S301: Obtain a first image.

[0151] In this step, the third device 40 may obtain first images, each of which contains a first object, such as characters on a license plate.

[0152] Different countries or regions have different templates and fonts for characters. The sub-portion of the image may be obtained according to the country or region in which the object recognition is performed.

[0153] S302: Determine the arrangement format of the first object, such as one row or two rows of license plates.

[0154] S303: Generate a second image of a combination of the first objects based on the first image and the arrangement format of the first objects.

[0155] S304: placing the second image in a third image having a background whose contrast with the second image is higher than a predefined threshold.

[0156] This requirement for a third image of the background is a bound for recognizing objects, which must be learned during training. If, for example, a black background is used, this must be learned to recognize the location of the target based on non-blackness, which will obviously not work for real images of objects on real backgrounds.

[0157] Some kind of transformation, such as rotation, shrinking, etc., may be performed before placing the second image 42 in the third image.

[0158] S305: Determine the position of the second image in the third image.

[0159] S306: Replace a portion of the background in the third image with a fourth image of the scene to generate a fifth image.

[0160] The sizes of the third and fourth images of the scene may be the same for position determination and image replacement.

[0161] S306 ′: Add noise to the fifth image to generate a sixth image.

[0162] This is an optional step.

[0163] S307: Correlate the fifth image / sixth image with the position.

[0164] The correlation with location can be used for model training to know the location of the object to be recognized.

[0165] S308: Use the fifth image / sixth image to train the model.

[0166] It is worth mentioning that the above procedure only describes the generation process of one image. In order to train the model, hundreds / thousands of images are usually generated. And the number of images of the scene should be large enough to avoid being memorized during training. Optionally, a fourth image of the scene can be collected from an open database or purchased from a commercial database or captured at the scene where the object to be identified is located. A limited number of images can also be processed via cropping, synthesis and other types of editing to generate a large number of images.

[0167] In addition to generating a large number of images, annotations can also be added to the images for model training. In a traditional way, it may take about 1 minute to manually label each image, which is time-consuming. Here, automatic annotation can be performed in a short time and is more accurate than manual annotation. Therefore, method 300 may further include the following steps of adding annotations:

[0168] For each first object in the second image, a second position of the first object in the second image is detected, and information of the second position is generated, and then the fifth image is annotated with an identifier of each first object in the second image and the information of the corresponding second position. Fig. 7A , Figure 7B and Figure 8 Detailed presentation of examples of adding annotations.

[0169] Figure 4 The image generation process of the model (an example of the second model mentioned above) for training license plates is shown. The generation process can be performed by a computer program such as a script. In this article, we name the script gen.py. The script gen.gy can draw 1000+ license plates using randomly organized characters. (In the form of the script's code, the number of generated images can be changed).

[0170] Figures 5A to 5F An image showing a model generated for training license plates. Figure 5A The expected output of the trained model for the image shown in is HH412RFP, for Figure 5B For FB78PFD, Figure 5C For JW01GAI, Figure 5D For AM46KVG, Figure 5E For XG86KI0, Fig. 5F However, for Figure 5C to Figure 5F , the trained model cannot recognize the license plate. Figure 5C The reason is that the license plate shown in the image is partially truncated. Figure 5D The reason is that the license plate shown in the image is too small. Figure 5E The reason is that the license plate shown in the image is too large for Fig. 5F The reason is the complete absence of license plates.

[0171] Reference Figure 4 First, generate the characters on the license plate. If you are going to generate a Hong Kong license plate, you can use the Hong Kong font to ensure the accuracy of license plate recognition. The characters can be arranged randomly. For a row of license plates, you can use the API named "generate_code()"; for two rows of license plates, you can use the API named "generate_code_two_line()", and use a fixed format for the 2 letters in the upper row and the 4 numbers in the lower row.

[0172] Next, an image 42 of the license plate containing the generated characters is generated. For one-row license plates, the API "generate_plate(font_height, char_ims)" can be used, which accepts two parameters: "font_height", indicating the height of the font (all characters on the license plate should have the same height); and "char_ims", indicating the font image that can be converted to numpy by PIL. For two-row license plates, the API "generate_two_line_plate(font_height, char_ims)" can be used, and the two parameters have the same meaning as the parameters in the API "generate_plate(font_height, char_ims)", and the width of the characters on the two-row license plates is the same, while the width on the one-row license plate may be different.

[0173] Next, an image 45 of the scene may be generated, and the image 42 of the license plate may be merged into the image 45 of the scene. The image 42 of the license plate may be rotated about a random axis to obtain a different appearance of the license plate, and then an image 46 of the license plate in the image 45 of the scene may be generated.

[0174] The API "generate_bg(num_bg_images)" can be used to generate images 45 of the scene and receives the number of images of the scene (which is also the total number of license plates to be generated) as a parameter.

[0175] API

[0176] "make_affine_transform(from_shape, to_shape, min_scale, max_scale, scale_variation=1.0, rotation_variation=1.0, translation_variation=1.0):" can be used to make a transformation and obtain a bounding box of the license plate in the image of the scene. It receives 5 parameters and returns an image 46 of the license plate in the image 45 of the scene, and a bounding box of the license plate in the image 45 of the scene. The parameter "from_shape" indicates the shape of the license plate, and the "to_shape" indicates the shape of the scene, and these two parameters are used to generate a bounding box. The parameters "min_scale", "max_scale" and "scale_variation" indicate the scale range of the image 46 of the license plate in the image 45 of the scene. The parameters "Rotation_variation" and "translation_variation" indicate the rotation and translation ranges of the image 45 of the license plate in the image 46 of the scene.

[0177] The API numpy.random, normal(scale, size) can be used to generate noise. The parameter "scale" indicates the density of the noise. After the noise is added, an image 47 of the license plate in the image of the scene with the added noise can be generated.

[0178] Introducing license plate image generation, Fig. 6A Show an image of a row of license plates, and Figure 6B Display an image of two rows of license plates.

[0179] We will now introduce automatic labeling of license plates. Compared to manual labeling, automatic labeling saves time and labor. According to tests using artificial images to train neural networks (such as Mask-RCNN), very good results can be obtained. 50,000 images can be generated and labeled in less than 5 minutes. The following is an example of labeling and training Mask R-CNN.

[0180] Mask R-CNN is an example segmentation model that can not only detect objects but also identify the location of any object pixel by pixel. Mask R-CNN is simpler to train and adds only small overhead relative to Faster R-CNN, and it outperforms all existing single-model entries on every task.

[0181] To train Mask R-CNN, first annotations should be added to the image. In the traditional way, it may take about 1 minute to manually label each image, which is time-consuming. Here, automatic annotation can be done in a short time and is more accurate than manual annotation. The following shows how automatic annotation works.

[0182] Each image generated consists of a scene and an object. Before merging the scene and the object, the position of the object in the image of the scene can be obtained. Then, the position of the object can be saved in relation to the object. Here, the format of the saved position can be a binary image and used to extract the contour.

[0183] The API "skimage.measure.find_contours()" can be used to detect the location of an object. Here, we take the contour of an object as an example, and the outermost contour can accurately locate the object, which is exactly what we need in Mask-RCNN, such as Fig. 7A and Figure 7B as shown in .

[0184] Based on the outermost contour, a label can be generated for each object. The label can be recorded in a .txt file (label_data.txt, such as Fig. 9 As shown in the figure, it is the input of Mask R-CNN. And the following is the structure of label_data.txt (one row for each image):

[0185] "image_name number_of_object name_of_first_object number_of_point x1,x2,x3,...,y1,y2,y3,...name_of_second_object number_of_point x1,x2,x3,...,y1,y2,y3,..."

[0186] The generated images may be stored into a folder named “test_number_plate” and another folder named “mask_Bigscale_number_plate.” The first folder may contain a data set of generated images, and the second folder may contain the locations of the characters on the image of the license plate.

[0187] Reference Fig. 9 , depicting an exemplary process 900 of model training and license plate recognition.

[0188] The image 901 of the scene and the image 902 of the characters are two inputs of the frame 903. At the frame 903, an image of the license plate is generated and annotated, and then the frame 903 outputs the annotated image 904 of the license plate.

[0189] The annotated image 904 of the license plate and the pre-trained model 905 are two inputs to box 906. At box 906, the model for license plate recognition is trained and the trained model 907 is output.

[0190] The trained model 907 and the captured image 908 of the vehicle 50 are two inputs to block 909. At block 909, the license plate 60 is recognized from the captured image 908, and numbers and letters are detected from the image of the license plate 90.

[0191] As output of block 909, detected characters 910 are post-processed by block 911 and a recognized license plate 912 is output.

[0192] Fig. 10A A block diagram showing an exemplary embodiment of a second device 30 of the present disclosure is depicted. Fig. 10A , the second device 30 may include:

[0193] A receiving module 301 configured to receive an image of a license plate 60 of a vehicle 50;

[0194] A recognition module 302 configured to recognize the license plate 60 from an image of the license plate 60 using a model, wherein the model is trained using annotated images of the license plate and the annotated images are generated based on an image of the characters and an image of the scene.

[0195] Optionally, the image of the scene is from the same or a similar type of scene as the scene from which the first image is from.

[0196] Optionally, the identification module 302 is further configured to:

[0197] - recognizing characters on the license plate 60 from the image of the license plate 60;

[0198] - Recognize the arrangement format of the characters on the license plate 60;

[0199] - Get the position of each character on the license plate 60;

[0200] - Arranging the identified characters in the identified arrangement format and according to the obtained position of each character.

[0201] Optionally, when identifying characters on the license plate 60 from the image of the license plate 60, the recognition module 303 is further configured to: identify each character on the license plate 60 from the image of the license plate 60 via a bounding box; the recognition module 302 is further configured to: determine that the recognition result is invalid if the ratio of the height of the bounding box of any character on one side of the license plate 60 to the average height of the bounding boxes of all characters is higher than a predefined threshold.

[0202] Fig. 10B Another block diagram showing an exemplary embodiment of a second device 30 of the present disclosure is depicted. Fig. 10B , the second device 30 may include:

[0203] - at least one memory 204 configured to store instructions;

[0204] - at least one processor 205 coupled to said at least one memory 204 and configured to perform the steps performed by the second device 30 when executing executable instructions.

[0205] Optionally, the second device 30 may further include a communication module 305 configured to transmit / receive images, data, etc. from / to the communication server 20 / first device 10. The at least one processor 304, the at least one memory 303 and the communication module 305 may be connected via a bus or directly connected to each other.

[0206] It is worth mentioning that the modules 301 and 302 may be software modules including instructions, which are stored in the at least one memory 303 and perform the method 200 when executed by the at least one processor 304 .

[0207] Fig.11A A block diagram showing an exemplary embodiment of a third device 40 of the present disclosure is depicted. Fig.11A , the third device 40 may include:

[0208] An image acquisition module 401 is configured to acquire first images, wherein each first image contains a first object;

[0209] - Image generation module 402, which is configured to generate a second image of a combination of the first object based on the first image and the arrangement format; place the second image in a third image of a background having a contrast with the second image above a predefined threshold; determine a first position of the second image in the third image; replace a portion of the background in the third image with a fourth image of the scene to generate a fifth image; relate the fifth image to the first position;

[0210] A training module 403 is configured to train a model using information of the first image and the first position.

[0211] Optionally, the image generation module 402 is further configured to:

[0212] - for each first object in the second image, detecting a second position of the first object in the second image; generating information of the second position;

[0213] When annotating the fifth image, annotating the fifth image with the identifier of each first object in the second image and the information of the corresponding second position.

[0214] Fig. 11B A block diagram showing another exemplary embodiment of a third device 40 of the present disclosure is depicted. Fig. 11B , the third device 40 may include:

[0215] - at least one memory 404 configured to store executable instructions;

[0216] - at least one processor 405 coupled to said at least one memory 404 and configured to perform the method 300 when executing said executable instructions.

[0217] Optionally, the third device 40 may also include an I / O port 406 configured to receive input and send output. The at least one processor 405, the at least one memory 404 and the I / O port 406 may be connected via a bus or directly connected to each other.

[0218] It is worth mentioning that the above modules 401 to 403 may be software modules including instructions, which are stored in the at least one memory 404 and execute the method 300 when executed by the at least one processor 405 .

[0219] The present disclosure also provides a computer-readable medium storing executable instructions, which, when executed by a computer, enable the computer to perform any one of the methods provided in the present disclosure.

[0220] The computer program is being executed by at least one processor and performs any one of the methods provided in the present disclosure.

[0221] Although the technology of the present invention has been described in detail with reference to certain embodiments, it should be understood that the technology of the present invention is not limited to those precise embodiments. In fact, in view of the disclosure describing the exemplary modes for practicing the present invention, many modifications and variations can be made by those skilled in the art without departing from the scope and spirit of the present invention. Therefore, the scope of the present invention is indicated by the appended claims, rather than by the foregoing description. All changes, modifications and variations that fall within the equivalent meaning and scope of the claims will be considered to be within the scope of the claims.

Claims

1. A system (100) for license plate recognition, comprising: A first device (10) is configured to: capturing an image of a vehicle (50), obtaining an image of a license plate (60) of the vehicle (50) from the image of the vehicle (50), and sending the image of the license plate (60) to a second device (30); the second device (30) being configured to recognize the license plate (60) of the vehicle (50) from the image of the license plate (60) using a model, wherein the model is trained using annotated images of license plates and the annotated images are generated based on images of characters and images of a scene; Wherein, when the second device (30) recognizes characters on the license plate (60) from the image of the license plate (60), the second device (30) is further configured to: recognize each character on the license plate (60) from the image of the license plate (60) via a bounding box; a bounding box exists around each character; Wherein when the second device (30) recognizes the license plate (60) of the vehicle (50) from the image of the license plate (60), it is further configured to: recognizing characters on the license plate (60) from the image of the license plate (60); Identifying the arrangement format of the characters on the license plate (60); wherein a Hough transform is performed to determine the arrangement format of the characters on the license plate (60), a local maximum search for non-maximum suppression is performed for each row, and a radius threshold is set for the execution of non-maximum suppression; Obtaining the position of each character on the license plate (60); The identified characters are arranged in the identified arrangement format and according to the obtained position of each character.

2. The system (100) of claim 1, wherein the image of the scene is from a scene of the same or similar type as the scene from which the image of the vehicle (50) is from.

3. The system (100) of claim 1, wherein The second device (30) is further configured to determine that the recognition result is invalid if the ratio of the height of the bounding box of any character on one side of the license plate (60) to the average height of the bounding boxes of all characters is higher than a predefined threshold.

4. A method (200) for license plate recognition, comprising: receiving ( S204 ) an image of a license plate ( 60 ) of a vehicle ( 50 ); recognizing (S205) the license plate (60) from the image of the license plate (60) using a model, wherein the model is trained using an annotated image of the license plate and the annotated image is generated based on an image of characters and an image of the scene; Recognizing (S2051) characters on the license plate (60) from the image of the license plate (60) further comprises: recognizing each character on the license plate (60) from the image of the license plate (60) via a bounding box; a bounding box exists around each character; Wherein identifying (S205) the license plate (60) from the image of the license plate (60) further comprises: recognizing (S2051) characters on the license plate (60) from the image of the license plate (60); Identifying (S2052) the arrangement format of the characters on the license plate (60); wherein a Hough transform is performed to determine the arrangement format of the characters on the license plate (60), a local maximum search for non-maximum suppression is performed for each row, and a radius threshold is set for the execution of non-maximum suppression; Obtaining (S2053) the position of each character on the license plate (60); The recognized characters are arranged (S2054) in the recognized arrangement format and according to the obtained position of each character.

5. The method (200) of claim 4, wherein the image of the scene is from a scene of the same or similar type as the scene from which the first image is from; wherein the first image includes an image of the vehicle.

6. The method (200) of claim 4, wherein The method (200) further comprises: If the ratio of the height of the bounding box of any character on one side of the license plate (60) to the average height of the bounding boxes of all characters is higher than a predefined threshold, it is determined (S206) that the recognition result is invalid.

7. A second device (30) for license plate recognition, comprising: a receiving module (301) configured to receive an image of a license plate (60) of a vehicle (50); a recognition module (302) configured to recognize the license plate (60) from the image of the license plate (60) using a model, wherein the model is trained using annotated images of license plates and the annotated images are generated based on images of characters and images of a scene; When recognizing characters on the license plate (60) from the image of the license plate (60), the recognition module (303) is further configured to: recognize each character on the license plate (60) from the image of the license plate (60) via a bounding box; a bounding box exists around each character; The identification module (302) is further configured to: recognizing characters on the license plate (60) from the image of the license plate (60); Identifying the arrangement format of the characters on the license plate (60); wherein a Hough transform is performed to determine the arrangement format of the characters on the license plate (60), a local maximum search for non-maximum suppression is performed for each row, and a radius threshold is set for the execution of non-maximum suppression; Obtaining the position of each character on the license plate (60); The identified characters are arranged in the identified arrangement format and according to the obtained position of each character.

8. The second device (30) of claim 7, wherein the image of the scene is from a scene of the same or similar type as the scene from which the first image is from, wherein the first image includes an image of the vehicle.

9. The second device (30) according to claim 7, wherein The recognition module (302) is further configured to determine that the recognition result is invalid if the ratio of the height of the bounding box of any character on one side of the license plate (60) to the average height of the bounding boxes of all characters is higher than a predefined threshold.

10. A second device (30) for license plate recognition, comprising: at least one processor (304); as well as At least one memory (303) coupled to the at least one processor (304) for storing computer executable instructions which, when executed, cause the at least one processor (304) to perform the method according to any one of claims 4 to 6.

11. A method (300) for model training, comprising: Obtaining (S301) first images, wherein each first image contains a first object; wherein the first image comprises an image of a vehicle; Determine (S302) the layout format of the first object; wherein, a Hough transform is performed to determine the layout format of the first object, a local maximum search of non-maximum suppression is performed for each row, and a radius threshold is set for the execution of non-maximum suppression; generating (S303) a second image of a combination of the first objects based on the first image and the arrangement format; placing (S304) the second image in a third image having a background whose contrast with the second image is above a predefined threshold; determining (S305) a first position of the second image in the third image; replacing (S306) a portion of the background in the third image with a fourth image of the scene to generate a fifth image; Associating the fifth image with the first position (S307); Using (S308) the fifth image to train a model; It further includes: For each outermost contour of the first object in the second image, detecting a second position of the first object in the second image; generating information of the second position; The fifth image is annotated with the identifier of each first object in the second image and the corresponding second location information.

12. A third device (40) for model training, comprising: An image acquisition module (401) configured to acquire first images, wherein each first image contains a first object; An image generation module (402) is configured to: Determine the layout format of the first object; wherein, a Hough transform is performed to determine the layout format of the first object, a local maximum search of non-maximum suppression is performed for each row, and a radius threshold is set for the execution of non-maximum suppression; generating a second image of the combination of the first objects based on the first image and the arrangement format; placing the second image in a third image having a background with a contrast ratio with the second image above a predefined threshold; determining a first position of the second image in the third image; replacing a portion of the background in the third image with a fourth image of the scene to produce a fifth image; associating the fifth image with the first location; A training module (403) configured to train a model using the fifth image and information of the first position; The image generation module (402) is further configured to: For each outermost contour of the first object in the second image, detecting a second position of the first object in the second image; generating information of the second position; When annotating the fifth image, the fifth image is annotated with the identifier of each first object in the second image and the information of the corresponding second position.

13. A third device (40) for model training, comprising: at least one processor (405); as well as At least one memory (404) coupled to the at least one processor (405) for storing computer executable instructions which, when executed, cause the at least one processor (405) to perform the method according to claim 11.

14. A computer-readable medium storing executable instructions, which, when executed by a computer, enable the computer to perform the method according to any one of claims 4 to 6 and 11.

Images