Algorithm Pipeline and General Principles of a Cost-Effective Automatic Plate Recognition System for Resource-Constrained Embedded Devices

Abstract

To provide a method, device, and machine-readable medium for recognizing a license plate number on an image, in which all necessary processing and related calculations are performed by a low-resource IoT device (with a mobile processor).SOLUTION: A method includes steps of: detecting a license plate on a vehicle in an image; and recognizing characters on the license plate and coordinates of the characters with a neural network. The neural network includes, as loss function, classification loss, confidence loss, and Complete Intersection over Union (CIoU) loss. The method further includes a step of organizing the recognized characters based on the coordinates to form the recognized license plate number.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Non-limiting, exemplary embodiments of the present disclosure relate generally to the technical field of image recognition, and more specifically to methods and devices for recognizing license plate numbers in an image. [Background technology]

[0002] This section introduces aspects that may facilitate a better understanding of the present disclosure. Accordingly, the statements in this section are to be read in this light and are not to be understood as admissions about material in the prior art or material that is not in the prior art.

[0003] To reduce manpower efforts, it would be quite useful to automatically recognize license plate numbers of vehicles in images, and the license plate number data could be used, for example, to determine which vehicles have entered closed areas or to enforce traffic code rules.

[0004] For example, there are several existing approaches in the art for recognizing license plate numbers in images.

[0005] The document US9911055 discloses a method, system, and processor-readable medium for detecting and classifying license plates. In an exemplary embodiment, a photo of a vehicle can be captured by a photo capturing unit. The license plate region can then be located in the captured photo of the vehicle by extracting a set of candidate regions from the photo using a weak classifier. The set of candidate regions can be ranked using a secondary strong classifier. The captured photo can then be classified according to a confidence-based classification based on the classification criteria determined by the weak classifier and the secondary strong classifier.

[0006] In the document of US Patent Application Publication No. 20170017854, a method for Arabic optical character recognition using a hidden Markov model and a decision tree is disclosed, which includes: receiving an input image containing Arabic text; removing all diacritics from the input image by detecting the bounding box of each diacritic and comparing the coordinates of the diacritic with the bounding box of the text body; dividing the input image into four layers; performing feature extraction in the divided four layers; generating an HMM model for representing each Arabic character by inputting the result of the feature extraction into a hidden Markov model; performing iterative training of the HMM model until an overall likelihood criterion is satisfied; and predicting the location and class of the diacritic by inputting the result of the iterative training into a decision tree to generate a final recognition result. The present invention can facilitate easy recognition of Arabic characters by utilizing the character features of Arabic characters, and at the same time, can exhibit the characteristics of relatively high recognition accuracy.

[0007] In the document US10068146, a detection-based segmentation-free method and system for license plate recognition is disclosed. A photo capturing unit is used to first capture a photo of a vehicle. A license plate area is located in the vehicle photo. Then, a set of characters can be detected in the license plate area, and a geometry correction is performed based on the location of the set of characters detected in the license plate area. An operation is performed to sweep the OCR over the license plate area, and a hidden Markov model is used to utilize anchored digit / character locations to infer the characters with respect to the set of characters and the character locations. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] U.S. Patent No. 9,911,055 [Patent Document 2] US Patent Publication No. 20170017854 [Patent Document 3] U.S. Pat. No. 10,068,146 [Non-patent literature]

[0009] [Non-Patent Document 1] YOLOv3 paper[J.Redmon and A.Farhadi,“YOLOv3:An incremental improvement,”CoRR,vol.abs / 1804.02767,2018] Summary of the Invention [Problem to be solved by the invention]

[0010] In the existing approach, the vehicle's photo is sent to a back-end server for license plate recognition, which may result in a high occupancy of network traffic. This is because the on-site device that receives the photo does not have enough processing power to process the license plate recognition. Furthermore, if the photo containing the image of the vehicle is taken while the vehicle is moving, and thus the photo is blurred, the existing approach may need to install a higher quality camera with a faster shutter speed to solve this problem, which makes the hardware deployment expensive.

[0011] One of the objectives of the present disclosure is to solve or mitigate the above problems by proposing a solution that is capable of obtaining license plate numbers (which may include digits and / or letters) in a real-time setting from photos taken by a typical camera, where all necessary processing and related calculations are performed in a low-resource IoT device (having a mobile processor) to which the camera is connected. In contrast to the common approach of building license plate recognition systems, the proposed solution considers on-board calculations, does not utilize transmission of video frames from the capturing device to either a local or cloud server for further processing, and does not require a higher quality camera with a faster shutter speed. Thus, the proposed solution is paramount in existing approaches in terms of network bandwidth and hardware costs, reducing them by an order of magnitude.

[0012] The solution of the present disclosure includes the following aspects. [Means for solving the problem]

[0013] A first aspect of the solution of the present disclosure provides a method for recognizing a license plate number in an image, the method including: detecting a license plate for a vehicle in the image; recognizing characters and coordinates of the characters in the license plate by a neural network, where a loss function of the neural network includes a classification loss, a confidence loss, and a complete intersection over union (CIoU) loss; and organizing the recognized characters based on their coordinates to form a recognized license plate number.

[0014] A second aspect of the solution of the present disclosure provides a device for recognizing a license plate number in an image, the device comprising: a detection unit for detecting a license plate associated with a vehicle in the image; a recognition unit for recognizing characters and coordinates of the characters in the license plate by a neural network, where a loss function of the neural network includes a classification loss, a confidence loss, and a complete intersection over union (CIoU) loss; and a sorting unit for sorting the recognized characters based on the coordinates to form a recognized license plate number.

[0015] A third aspect of the solution of the present disclosure provides a device for recognising a license plate number in an image, the device comprising a processor and a memory storing instructions that, when executed by the processor, cause the device to perform a method according to the first aspect.

[0016] A fourth aspect of the solution of the present disclosure provides a machine-readable medium storing instructions which, when executed on a device, cause the device to perform a method according to the first aspect.

[0017] The above and other aspects, features, and advantages of the present disclosure will become more fully apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like numerals or characters are used to designate similar or equivalent elements. The drawings are illustrated to facilitate a better understanding of the embodiments of the present disclosure and are not necessarily drawn to scale. [Brief description of the drawings]

[0018] [Figure 1] 1 shows a flowchart of a method for recognizing license plate numbers in an image according to the present disclosure. [Diagram 2] 1 illustrates an exemplary architecture of a neural network for detecting vehicles. [Diagram 3] 1 illustrates an exemplary architecture of a neural network for detecting license plates. [Figure 4] 1 illustrates an exemplary architecture of a neural network for recognizing characters on license plates. [Diagram 5] FIG. 1 is a schematic block diagram of a device according to the present disclosure. [Figure 6] FIG. 2 is another schematic block diagram of a device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The embodiments herein are more fully described below with reference to the accompanying drawings. However, the embodiments herein may be embodied in many different forms and should not be construed as limiting the scope of the appended claims.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It is further understood that the terms "comprises," "comprising," "includes," and / or "including," as used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0021] Additionally, the use of ordinal terms such as "first," "second," "third," and the like, herein to modify elements does not by themselves imply any priority, precedence, or order of one element relative to another element, or a chronological order for performing actions of a method, but is merely used as a marker to distinguish one element having a certain name from another element having the same name (but for the purposes of using ordinal terms) to distinguish between the elements.

[0022] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood. It is further understood that the terms used herein should be interpreted as having a meaning consistent with their meaning in the context of this specification and related art, and should not be interpreted in an idealized or overly formal sense unless so expressly defined herein.

[0023] A flow chart of a method 100 for recognizing license plate numbers in an image according to the present disclosure is shown in Figure 1. The method 100 includes step 101 of detecting license plates for vehicles in the image, step 102 of recognizing characters and coordinates of the characters in the license plate by a neural network, where loss functions of the neural network include classification loss, confidence loss, and complete intersection over union (CIoU) loss, and step 103 of organizing the recognized characters based on their coordinates to form a recognized license plate number.

[0024] The method according to the present disclosure may utilize a state-of-the-art object detector based on a Convolutional Neural Network (CNN). The network may be trained using images from several open source datasets, as well as images manually collected and classified by UrbanChain Group Limited. The computational efficiency of the method allows it to run on a variety of resource-constrained embedded devices (also known as IoT devices), including system-on-chip (SoCs) based on ARM or Intel mobile processors, with an integrated camera connected to the SoC via a MIPI-CSI interface.

[0025] The amount of shared CPU-GPU RAM consumed by the proposed method is estimated to be as low as 1.5Gb. Using an innovative approach, the method does not rely on video transmission over the network and does not utilize any image processing on cloud servers, resulting in minimal data usage, bandwidth requirements, and overall cost of ownership.

[0026] An embodiment of a method according to the present disclosure is described below, which may involve some pre-processing and / or pre-conditions, such as taking photographs each containing an image of a moving vehicle, selecting one or more relatively clear photographs from the taken photographs, etc. However, it will be understood that the embodiments may also be applied without those pre-processing and pre-conditions, e.g., the image of the vehicle originates directly from a database (instead of simply taking a photograph), the vehicle is stationary, and / or the image itself is sufficiently clear.

[0027] As explained above, step 101 is performed on an image of the vehicle. The image may originate from, for example, a photograph. For example, multiple photographs of a moving vehicle may be taken with at least one camera, for example in a parking lot, with each photograph of the multiple photographs taken with the same camera being taken consecutively at a different time. In one example, the multiple photographs include 5-10 photographs. In another example, the multiple photographs may also be adjacent frames of a video relating to the vehicle. In another example, the photographs are taken with two cameras connected to the same collection device, each camera having a different angle of view and altitude relative to the ground plane, capturing possible variations in the angle of the vehicle's headlights, affecting the quality of the photograph taken.

[0028] If some of the captured photographs are blurred, it may be necessary to reduce the motion blur to allow accurate recognition of the symbol on the license plate. One solution to this problem may be a camera with a fast shutter speed, but such hardware may be quite expensive for scaled deployments. For example, it is proposed to solve the problem algorithmically by selecting one or more relatively clear photographs from the plurality of photographs for further processing, the selection being based on comparing at least one calculated first parameter associated with the blurriness of the photograph to a threshold value. Preferably, the first parameter is the variance value of the photograph. Preferably, edge detection is performed on the plurality of photographs before calculating the first parameter. Preferably, the Laplacian kernel (D 2 xy ) is used for edge detection.

[0029] For example, multiple (e.g., 5-10) consecutively taken photos can be set in a buffer and then its measure of blur calculated by sliding each photo with a Laplacian kernel and then calculating the variance of the resulting matrix. The Laplacian is a measure of the 2D isotropy of the second spatial derivative of a photo. The Laplacian of a photo highlights areas of rapid intensity change and is thus often used for edge detection. The Laplacian L(x,y) of a photo with pixel intensity values ​​I(x,y) is given by:

[0030]

number

[0031] Since the input photo is represented as a set of discrete pixels, a separate convolution kernel can be found that can approximate the second derivative in (1).

[0032]

number

[0033] Using kernel (2), the Laplacian can be calculated using standard convolution methods. The variation of the Laplacian can serve as a reliable feature of the "blurriness" of a photo, and with an appropriate threshold, can be used to select the sharper photo(s) for further processing.

[0034] Then, in one example, in each of the selected photo(s), vehicles may be detected and corresponding image regions containing the vehicles may be cropped as images to which the methods of the present disclosure are applied, and vehicles are detected, for example, by a trained neural network with a custom architecture optimized for efficient inference on low resource devices.

[0035] An exemplary architecture of a neural network for vehicle detection is shown in FIG. 2, where the parameters and dimensions of the individual layers are described. The input size of the photos fed into the network is adapted to the non-square frame size (e.g., 448×288 pixels) that can be obtained from a typical camera. The network was trained on a private dataset of about 5000 photos collected and classified by UrbanChain Group Limited. The photos were taken in unconstrained road traffic scenarios and in parking lots. Various data augmentation measures such as inversion, rescaling, blurring, and application of a cut-mix approach were used to train the network, resulting in the creation of composite photos with different features from the single classified photo, thereby preventing overfitting.

[0036] In an example, after the image region containing the vehicle is cropped from the photo, the cropped image region may serve as the image processed by the method according to the present disclosure. The license plate in each cropped image region of the selected photo may be detected by a trained neural network dedicated for efficient inference on a mobile processor, for example, with a carefully designed architecture. For example, the neural network for detecting license plates may utilize the ideas of the YOLOv3 paper [J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” CoRR, vol. abs / 1804.02767, 2018], trained on thousands of real photos manually collected and classified, and adapted to run in near real time on low-resource embedded devices, for example based on ARM chips. An exemplary architecture of the neural network for detecting license plates is shown in FIG. 3. The resolution of the resized input image is set to, for example, 608×608 during model training and set to, for example, 416×416 during the inference phase to accelerate the computation on the embedded platform.

[0037] Once the license plate is detected from the image, the characters and the coordinates of the characters on the license plate can be recognized, for example, by a neural network with a custom architecture designed for efficient computation on low-resource devices. For fast and resource-efficient character detection in real time, the layer configuration and order and the number of corresponding parameters (including convolution filters and their sizes) in each layer of the neural network are carefully selected with the aim of minimizing the total number of multiplication and addition operations involved. The network architecture can be considered as a critical part of the proposed solution. For example, the network architecture has shown a license plate recognition accuracy of 99.7% during field tests, which are conducted on up to 2000 license plate numbers recognized from photos taken by a collection device in unconstrained real-world scenarios, both during daytime and nighttime.

[0038] An example of a layer-by-layer description of the proposed neural network architecture for recognizing characters on license plates is shown in Figure 4. The associated computational cost in terms of the number of multiplication-add operations at each stage (i.e., layers of the neural network) is also listed in Figure 4.

[0039] The neural network is trained to predict, for example, 35 classes ('0'-'9', 'A'-'Z', letter 'O' is detected / recognized together with digit '0') and outputs the class and pixel coordinates of each recognized character. In one embodiment, the sum of loss functions used as optimization criteria during the neural network training process includes classification loss, confidence loss, and complete cross over union (CIoU) loss, and has the following formula:

[0040]

number

[0041] The confidence scores mentioned above are the output of the neural network. The M-dimensional vector of confidence scores (where M is the total number of classes, letters and digits available in this particular example) is a slice of the initial vector of predicted parameters of each detection box, such as the detection box coordinates. This initial vector is obtained following a series of algebraic operations (e.g., matrix addition, multiplication, concatenation pooling, etc.) commonly used while building neural networks, using a set of weights resulting from a process of non-convex optimization of an objective loss function by a type of stochastic gradient descent (SGD) algorithm, e.g., SGD with momentum. In other words, the confidence loss is the loss in making class prediction. In one embodiment, for a positive match prediction of a character, the confidence loss may be penalized, for example, according to the confidence score of the character's class. For a negative match prediction of a character, the confidence loss may be penalized, for example, according to the confidence score of a special class. In the special class, the undetected objects are classified and may be calculated as a softmax loss on the confidence c of multiple classes (class scores) shown in (3) below.

[0042]

number

[0043] where N is the number of matched default boxes (in the single shot object detector paradigm used to build the object detector).

[0044] Additionally, various techniques may be utilized to minimize errors introduced during the license plate character recognition stage, for example, at both the character and string level (the string representation of a given license plate number).

[0045] For example, non-maximum suppression can be an important process for character recognition. It is possible to generate multiple detection boxes for the same single character, where a detection box is the area of ​​the image that contains all the pixels of the character. First, all detection boxes are sorted based on their confidence score (second parameter) calculated for each box. The detection box with the maximum confidence score is selected, and all other remaining detection boxes that have a significant overlap with the detection box as determined by the third parameter are suppressed (i.e., discarded). The third parameter is equal to the area of ​​the intersection of the pair of detection boxes divided by the total area covered by the pair of detection boxes. For each selected detection box, the third parameter of each pair of detection boxes including the selected detection box and another remaining detection box is compared to a predefined threshold. If the third parameter is higher than the predefined threshold, the other remaining detection box is discarded. This process is applied recursively on all of the remaining detection boxes. Finally, all of the selected detection boxes can be used as the final detection box for the character in the license plate.

[0046] In one embodiment, the detecting and recognizing steps in the method according to the present disclosure are performed for each of a plurality of images from adjacent frames in a video of the vehicle, and the method further includes, for each character position, comparing characters from the positions on the plurality of images to each other and selecting the character that occurs most often as the recognized character at that position.

[0047] Furthermore, to address the number of incorrectly recognized license plates, well-known string processing algorithms such as the Levenshtein edit distance may be utilized to remove errors resulting from, for example, motion blur, various occlusions, and lighting of the target vehicle. The Levenshtein algorithm calculates the edit distance, i.e., the minimum number of edit operations required to modify one string to obtain another string.

[0048] Thus, in another embodiment, the detecting, recognizing and sorting steps in the method according to the present disclosure are performed on each of a plurality of images from adjacent frames in a video of a vehicle to form a plurality of license plate numbers, and the method further includes: assigning all license plate numbers having a short edit distance (e.g., 2) from the plurality of license plate numbers to a cluster; and choosing the license plate number that occurs most often in the cluster as the recognized license plate number. The resulting license plate number can then be sent to a backend server along with some additional metadata payload. This step can reduce the risk of retrieving the wrong character from the image, and it allows the missing data to be retrieved from the adjacent images.

[0049] The recognized characters may be set, for example, from a one-line or two-line license plate number. In one embodiment, the recognized license plate number is a two-line license plate number. In this embodiment, initially, each character may be assigned to a position on the second line by default. To perform a reliable distribution between the two lines, each character of the license plate number is compared with two previous characters and two subsequent characters, and the difference between multiple values ​​of the Y coordinate of the center of the character's detection box for each pair of corresponding characters is calculated. If the difference exceeds 80% of the average height (in pixels) of the character's box (calculated by taking the average of all heights of the detected character boxes), the character with the smallest value of the Y coordinate is set to the first line.

[0050] In a further embodiment, a system is proposed that includes at least one collection device, including at least one camera, a processor, memory, and a power source, and at least one server. The collection device is configured to perform the method according to the present disclosure and to transfer at least the license plate number to the server. The high computational efficiency allows the method to be performed in the device with relatively low power, and thus most of the computations can be performed in the collection device. By doing this, it allows the transmission of smaller data packages (license plate number, metadata, etc., and unprocessed raw photos). It will also be much cheaper because of the low requirements on available computational power and memory.

[0051] 5 shows a schematic block diagram of a device 500 according to the present disclosure. The device 500 is used for recognizing a license plate number in an image, and may include a detection unit 501 for detecting a license plate related to a vehicle in the image, a recognition unit 502 for recognizing characters in the license plate and the coordinates of the characters by a neural network, where the loss function of the neural network includes a classification loss, a confidence loss, and a complete intersection over union (CIoU) loss, and a sorting unit 503 for sorting the recognized characters based on the coordinates to form a recognized license plate number.

[0052] It can be appreciated that the device 500 described herein may be implemented by various units, such that a device 500 implementing one or more functions described in the embodiments may include not only the units shown in the corresponding figures, but also other units for implementing those one or more functions. Furthermore, the device 500 may include a single unit configured to perform two or more functions, or separate units for each separate function. Furthermore, the units may be implemented in hardware, firmware, software, or any combination thereof.

[0053] It will be understood that blocks of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that can be provided to a processor of a general purpose computer, special purpose computer, and / or other programmable data processing apparatus to create a machine, such that the instructions, when executed by the processor of the computer and / or other programmable data processing apparatus, cause means for performing the function / action specified in the block or blocks of the block diagrams and / or flowchart illustrations.

[0054] It is also understood that the functions / actions noted in the flowchart blocks may occur out of the order noted in the operational diagrams. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order depending on the functions / actions involved. Although some of the figures include communication path arrows indicating a primary direction of communication, it is understood that communication may occur in the opposite direction to that shown arrow.

[0055] Furthermore, the solutions of the present disclosure may take the form of a computer program in a memory having computer usable or computer readable program code embodied in a medium for use by or in association with an instruction execution system. In the context of this document, a memory may be any medium that can contain, store, or be adapted to convey a program for use by or in association with an instruction execution system, instruction execution apparatus, or instruction execution device.

[0056] Thus, as shown in Fig. 6, the present disclosure also provides a device 600 including a processor 601 and a memory 602. In the device 600, the memory 602 stores instructions that, when executed by the processor 601, cause the device 600 to perform the methods described above with the aid of the embodiments.

[0057] The present disclosure also provides a machine-readable medium (not shown) storing instructions that, when executed on a device, cause the device to perform the methods described with the above embodiments.

[0058] While this specification contains many details of specific embodiments, these should not be construed as limiting the scope of any embodiment or the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular embodiment. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as operating in a particular combination and may even initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be directed to a subcombination, or a variation of the subcombination.

[0059] It is obvious to those skilled in the art that as the technology advances, the concept of the present invention can be implemented in various ways. It should be understood that the above-described embodiments are given to illustrate rather than limit the present disclosure, and modifications and variations can be used without departing from the spirit and scope of the present disclosure as those skilled in the art will easily understand. Such modifications and variations are considered to be within the scope of the present disclosure and the appended claims. The scope of protection of the present disclosure is defined by the appended claims.

Claims

1. A method (100) for recognizing license plate numbers in an image, comprising: Detecting (101) a license plate for a vehicle in an image; a step (102) of recognizing a character on the license plate and the coordinates of the character by a neural network, the loss function of the neural network including a classification loss, a confidence loss, and a complete intersection over union (CIoU) loss; organizing the recognized characters based on the coordinates to form the recognized license plate number (103); A method (100).

2. The loss function is [0010] 2. The method of claim 1, wherein:

3. For positive match predictions of the character, the confidence loss is penalized according to the confidence score of the character's class, and for negative match predictions of the character, the confidence loss is penalized according to a softmax loss on the confidence of multiple classes in the following equation: [0025] 2. The method (100) of claim 1, wherein N is the number of matched default boxes.

4. The detecting and recognizing steps are performed for each of a plurality of images from adjacent frames in the video of the vehicle, and the method further comprises: for each position of the character: comparing the characters from the locations on a plurality of images to one another; selecting the character that occurs most often as the recognized character at the location; The method of claim 1 , comprising:

5. The detecting, recognizing, and organizing steps are performed for each of a plurality of images from adjacent frames in a video relating to the vehicle to form a plurality of the license plate numbers, the method further comprising: assigning all said license plate numbers having a short edit distance from a plurality of said license plate numbers to a cluster; selecting the license plate number that occurs most often in the cluster as the recognized license plate number; The method of claim 1 , comprising:

6. 2. The method (100) of claim 1, wherein the recognized license plate number is a two line license plate number.

7. 2. The method (100) of claim 1, wherein the neural network predicts 35 classes for the characters, including the numbers "0" through "9" and the letters "A" through "Z," with the letter "O" and the number "0" being considered as one class.

8. Prior to the detecting step, detecting said vehicle in a photograph or a video frame relating to said vehicle; cropping an area including the detected vehicle from the photograph or the video frame as the image; The method of claim 1 , further comprising:

9. A device (600) for recognizing the license plate number in the image, comprising: A processor (601); A memory (602) storing instructions which, when executed by the processor, cause the device to perform the method of any one of claims 1 to 8; A device (600).

10. A machine-readable medium storing instructions that, when executed on a device, cause the device to perform a method according to any one of claims 1 to 8.

Images