A two-stage method for accurate extraction of unconstrained license plate regions from coarse to fine

Abstract

A two-stage unconstrained license plate area extraction method from coarse to fine, firstly, the YOLOv3 network is used to coarsely locate the license plate area in the license plate image, then the improved network based on MobileNetV3 is used to locate the vertex position of the license plate in the coarsely located area, and finally the perspective transformation is used to correct the precisely located license plate area to the rectangular area. The present invention uses a two-stage license plate location method, which has a more obvious advantage in the accuracy of license plate location compared to the one-stage network that directly locates the license plate, while also maintaining the real-time nature of license plate location.

Description

Technical Field

[0001] The present invention relates to the technical field of license plate information recognition, and in particular to a two-stage method for accurately extracting a non-constrained license plate area from coarse to fine. Background Art

[0002] The license plate recognition system is a vital part of the intelligent transportation system, and its application scenarios are wide-ranging, such as highway toll stations, parking lot entrances and exits, traffic violation monitoring, traffic law enforcement, etc. In road traffic management, the license plate is the only basis for obtaining vehicle owner information. Therefore, accurate and effective recognition of license plate information is of great significance during vehicle driving. As a key part of the automatic license plate recognition system, license plate detection has always been a research topic that has attracted much attention. Existing license plate detection technologies can be roughly divided into two types, namely traditional methods and methods based on deep learning.

[0003] Traditional license plate detection algorithms can be divided into two types: 1) Algorithms based on image grayscale features. This type of algorithm mainly converts the image into grayscale and then uses the texture, edge and other features of the license plate to extract the license plate area. References [1] Lalimi, MA, Ghofrani, S., & McLernon, D. (2013). A vehicle license plate detection method using region and edge based methods. Computers & Electrical Engineering, 39 (3), 834–845. doi: 10.1016 / j.compeleceng.2012.09.015. A license plate detection method based on region and edge is proposed. The non-license plate area is smoothed using an improved region-based algorithm. The edge detection operator and morphological filtering are used to locate the license plate area in combination with geometric features such as license plate area, aspect ratio, and edge density. Reference [2] Jun, F., & Shuguang, D. (2012). A vehicle license plate location and correction method based on the characteristics of license plate. Proceedings of the 10th World Congress on Intelligent Control and Automation. doi: 10.1109 / wcica.2012.6357836. For different license plate tilt angles, a license plate location method based on texture feature information is proposed. The license plate area is determined according to the character information on the license plate, and the tilted license plate is corrected in combination with Hough transform. The proposed license plate location method is more suitable for processing images with simple backgrounds. The biggest advantage of the grayscale feature-based method is its fast processing speed, but its detection effect is greatly affected by the external environment, such as complex background and uneven lighting.

[0004] 2) Algorithms based on image color features. This type of method directly uses the color features of the license plate to obtain the license plate area. Reference [3] Tian, ​​Y., Song, J., Zhang, X., Shen, P., Zhang, L., Gong, W., … Zhu, G. (2016). An algorithm combined with color differential models for license-plate location. Neurocomputing, 212, 22–35. doi: 10.1016 / j.neucom.2016.02.081. An Adaboost algorithm combined with a color differential model is proposed. The algorithm includes two parts: coarse positioning and fine positioning. Coarse positioning uses the color differential model to obtain the license plate candidate area, and then uses the Adaboost algorithm to train the feature classifier to achieve fine positioning of the license plate. The color of license plates is usually fixed, so the color feature-based method can easily locate the license plate area. However, when the color of the vehicle is similar to the license plate or there is an area similar to the license plate in the background, such as traffic signs, billboards, etc., the license plate detection effect will be affected.

[0005] In recent years, deep learning technology has developed rapidly, and convolutional neural networks (CNNs) have been widely used in license plate detection. Reference

[16] Li H, Yang R, Chen X. License plate detection using convolutional neural network [C] / / 2017 3rd IEEE International Conference on Computer and Communications (ICCC). IEEE, 2017. A CNN model is proposed to locate the license plate in combination with a sliding window strategy. This method can detect various types of license plates with high accuracy and fast running speed. However, its data comes from images taken by a fixed camera, and the inclination angle of the license plate does not change much. The generalization ability of the model needs to be evaluated.

[0006] Reference [9] Silva, SM, & Jung, CR (2017). Real-Time Brazilian License Plate Detection and Recognition Using Deep Convolutional Neural Networks. 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). doi: 10.1109 / sibgrapi.2017.14. A CNN network (FV / LPD-NET) was proposed to detect the front view of the car and the license plate in cascade mode, which greatly improved the license plate detection accuracy. However, the network's generalization ability for license plate detection with large tilt angles needs to be improved. When the license plate is tilted at a large angle, the license plate area obtained by the above method will contain a large number of redundant areas, which is not conducive to subsequent character recognition. For this reason, the literature [7] Xie, L., Ahmad, T., Jin, L., Liu, Y., & Zhang, S. (2018). A New CNN-Based Method for Multi-Directional Car License Plate Detection. IEEE Transactions on Intelligent Transportation Systems, 19(2), 507–517. doi: 10.1109 / tits.2017.2784093. proposed a CNN-based multi-directional license plate detection framework, which uses the rotation angle prediction mechanism to determine the precise rotation rectangular area of ​​the license plate. This method has advantages in detection accuracy and computational complexity, but it is only suitable for processing rotation problems within the plane and still cannot accurately locate tilted license plates in three-dimensional space.

[0007] The Chinese patent "License plate detection method based on end-to-end multi-task deep learning" [Application number: 201910615131.3] proposes a license plate detection method based on end-to-end multi-task deep learning. It extracts features of different levels of the input image through a fully convolutional neural network, and uses two feature merging branches to regress the license plate and its corresponding vehicle bounding box respectively, thereby realizing license plate detection and vehicle detection at the same time.

[0008] Chinese patent "License plate detection method, storage medium and detection equipment based on convolutional network" [Application number: 201910210152.7] A license plate detection method, storage medium and detection equipment based on convolutional network, which extracts image features through convolutional network, and then obtains the front and rear positions of the vehicle by the front and rear detection module, and passes the obtained local feature map of the front and rear areas to the license plate detection module to obtain the position of the license plate.

[0009] Existing methods and patents based on convolutional neural networks usually use rectangular boxes to locate license plates. Among them, YOLOv3, as a deep learning framework, has a very wide range of applications in the field of target detection. Reference [9] Silva, SM, & Jung, CR (2017). Real-Time Brazilian License Plate Detection and Recognition Using Deep Convolutional Neural Networks. 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). doi: 10.1109 / sibgrapi.2017.14. A YOLO-based CNN network is proposed to simultaneously detect the front view of the car and the license plate.

[0010] The Chinese patent "A license plate detection method based on deep learning" [application number: 201811037178.8] proposes a license plate detection method based on deep learning. The license plate detection model is constructed by adding the inherent color attributes of the license plate to the YOLO model, and the license plate area is corrected using Radon transform, and the license plate area is obtained by fine-tuning using color and edge information. YOLOv3 uses rectangular frame detection, which is generally suitable for processing license plate images with small tilt angles. License plate images obtained in real scenes usually have different degrees of tilt. For license plates with large tilt angles, the license plate area located using a rectangular frame often contains more redundant information, thereby affecting the accuracy of license plate character recognition. As a lightweight framework, the MobileNet network is mainly used in the field of target classification. The effect of target detection in complex scenes is slightly inferior to other detection networks, but it runs fast and is usually used in combination with other networks.

[0011] Chinese patent [application number: 201810633017.9] uses SSD-MobileNet to detect the license plate area and classify the license plate types. This is to improve the accuracy of license plate detection and reduce false detection and missed detection of license plates. Summary of the invention

[0012] The invention provides a two-stage method for accurately extracting non-constrained license plate areas from coarse to fine, which can accurately locate the vertex of the license plate in a non-constrained environment and has a fast detection speed.

[0013] The technical solution adopted by the present invention is:

[0014] A two-stage unconstrained license plate area extraction method from coarse to fine. First, the YOLOv3 network is used to coarsely locate the license plate area in the license plate image, and then the improved network based on MobileNetV3 is used to locate the vertex position of the license plate in the coarsely located area. Finally, the perspective transformation is used to correct the finely located license plate area into a rectangular area.

[0015] A two-stage method for accurately extracting unconstrained license plate regions from coarse to fine, comprising the following steps:

[0016] Step 1: Rough positioning of the license plate area based on YOLOv3:

[0017] Use YOLOv3 to train the license plate detection model, extend the outer rectangular box to all sides to obtain a new rectangular box as the target label. The outer rectangular box extension strategy is as shown in formula (1) to formula (4):

[0018]

[0019]

[0020]

[0021]

[0022] Where: (x tl ,y tl )、(x dr ,y dr ) are the coordinates of the upper left and lower right vertices of the minimum bounding rectangle, (nx tl ,ny tl )、(nx dr ,ny dr ) are the coordinates of the upper left and lower right vertices of the extended bounding box, w0 and h0 are the width and height of the minimum circumscribed rectangle of the license plate, and W and H are the width and height of the image.

[0023] Step 2: Precise positioning of license plate vertices based on lightweight network MobileNetV3:

[0024] Based on the lightweight network MobileNetV3-Small, the last convolutional layer at the end of the network uses "sigmoid" as the activation function; at the same time, the convolutional layer Conv is used to replace the fully connected layer to improve the AL-MobileNetV3 network;

[0025] After the AL-MobileNetV3 network performs a series of feature extraction on the input image, it finally outputs 8 real numbers between 0 and 1, px0, py0, px1, py1, px2, py2, px3, and py3, which are the relative positions of the four vertices in the image. Finally, the output is further transformed to obtain the specific coordinates of the license plate vertices: lower right (x0, y0), lower left (x1, y1), upper left (x2, y2), and upper right (x3, y3). AL-MobileNetV3 uses mean square error (MSE) as the network's loss function, see formula (5):

[0026]

[0027] Where: (px i ,py i ), (px i ′,py i ′), i = 0, 1, 2, 3 are the vertex positions of the license plate predicted by the network and actually marked respectively.

[0028] Step 3: License plate correction and extraction:

[0029] License plate correction mainly involves geometric transformation of the license plate area, that is, establishing a mapping relationship between coordinates through some known reference points in the image, solving the unknown parameters in the mapping relationship, and then correcting the coordinates of other pixels in the image according to the mapping relationship;

[0030] The geometric transformation is a perspective transformation. The principle of perspective transformation is shown in formula (6) to formula (8):

[0031]

[0032] u=x′ / w′ (7);

[0033] v = y′ / w′ (8);

[0034] in, is the transformation matrix, (x, y) is the original image pixel coordinate, and the corresponding pixel coordinate after transformation is (u, v). Since the perspective transformation in this stage is a plane-to-plane mapping, w=1 is set to obtain formula (9) and formula (10):

[0035]

[0036]

[0037] Among them, k mn =a mn / a 33 , (m=1,2,3, n=1,2,3);

[0038] The four vertex coordinates obtained in the license plate precise positioning stage are lower right (x0, y0), lower left (x1, y1), upper left (x2, y2), and upper right (x3, y3). After correction, the corresponding vertex coordinates (u i ,v i ), the transformation matrix can be obtained by formula (9) and formula (10), and then the license plate area is mapped into the rectangular area to obtain the corrected license plate.

[0039] The present invention provides a two-stage method for accurately extracting non-constrained license plate areas from coarse to fine, and the technical effects are as follows:

[0040] 1) Aiming at the problem of license plate positioning in unconstrained scenarios, the present invention proposes a two-stage license plate positioning method from coarse to fine. In the coarse positioning stage, a bounding box extension strategy is used to expand the target area, which is beneficial to improving the detection accuracy of small targets. In the fine positioning stage, a regression network is used to achieve accurate positioning of the license plate.

[0041] 2) The present invention proposes a method for accurately locating the vertices of a license plate. First, the YOLOv3 network is used to roughly locate the license plate area in the image, and then a network improved based on MobileNetV3 is used to locate the position of the license plate vertex in the roughly located area. Finally, the perspective transformation is used to correct the precisely located license plate area to the rectangular area. This scheme can effectively solve the problem that the rectangular frame positioning contains redundant information.

[0042] 3) The present invention first detects the image area containing the license plate, and then performs vertex detection in the obtained area. The area detection is based on a rectangular frame, which limits the effective range of detection, so the regression method can be used to effectively predict the license plate vertices. Since the four vertices are fixed output, it is also applicable to images of poor quality. After the license plate area and vertex detection are realized, the license plate is finally corrected, which can more accurately realize the license plate positioning and license plate area extraction, and can provide good input for subsequent recognition.

[0043] 4) The present invention proposes a precise license plate extraction technology based on YOLOv3 and MobileNetV3. The method first uses YOLOv3 to roughly locate the license plate area in the image, then uses the improved MobileNetV3 network to locate the license plate vertices in the obtained license plate area, and finally uses perspective transformation to perform correction. The present invention uses a two-stage license plate positioning method. Compared with the one-stage network that directly performs license plate positioning, it has a more obvious advantage in the accuracy of license plate positioning, while also maintaining the real-time nature of license plate positioning. By improving the network structure of MobileNet, the present invention realizes precise license plate positioning using the classification function of MobileNet, effectively expanding the application scope of MobileNet. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 (1) is a rectangular frame positioning and vertex positioning effect diagram 1;

[0045] Figure 1(2) shows the effect of rectangle frame positioning and vertex positioning Figure 2 ;

[0046] Figure 1(3) shows the effect of rectangle frame positioning and vertex positioning Figure 3 ;

[0047] Figure 1(4) shows the effect of rectangle frame positioning and vertex positioning Figure 4 ;

[0048] Figure 1 (5) is a fifth effect diagram of rectangular frame positioning and vertex positioning;

[0049] Figure 1 (6) is a sixth effect diagram of rectangular frame positioning and vertex positioning;

[0050] Figure 1 (7) is a seventh effect diagram of rectangular frame positioning and vertex positioning;

[0051] Figure 1(8) is the eighth effect diagram of rectangular frame positioning and vertex positioning.

[0052] Figure 2 It is the relationship diagram between the target bounding box and the minimum enclosing rectangular box.

[0053] Figure 3 This is a diagram of the perspective transformation principle.

[0054] Figure 4 This is the flow chart for license plate positioning and extraction. DETAILED DESCRIPTION

[0055] The present invention changes the MobileNet network structure, fully utilizes its advantage of fast running speed and combines the powerful target detection ability of YOLOv3 to propose a two-stage non-constrained license plate area accurate extraction method from coarse to fine. YOLOv3 is used for coarse positioning of the license plate area, and MobileNet extracts the license plate vertices in the coarse positioning area. Since the processed image size is small, the use of the MobileNet network can still achieve good results. The method proposed by the present invention can accurately locate the license plate vertices in an unconstrained environment, with fast detection speed. The rectangular frame positioning and vertex positioning effects are shown in Figure 2. Figure 1(1) to Figure 1(8) As shown, the green rectangular boxes in Figure 1 (1) to Figure (4) are the license plate positioning effects based on the rectangular box, the red dots in Figure (5) to Figure (8) are the vertex positions of the license plate, and the yellow rectangular boxes are the license plate positioning effects based on the vertex. It can be seen from the figure that the license plate area obtained by the vertex-based method is more accurate and does not contain non-license plate information, while the area obtained by the rectangular box-based method contains more redundant information. Therefore, the vertex-based license plate positioning method is better.

[0056] The license plate images obtained in real scenes usually have different degrees of inclination. The license plate positioning method based on the rectangular frame is generally suitable for processing license plate images with small inclination angles. For license plates with large inclination angles, the license plate area positioned by the rectangular frame often contains more redundant information, thereby affecting the accuracy of license plate character recognition. To this end, the present invention proposes a two-stage non-constrained license plate area precision extraction method from coarse to fine. In the first stage, the YOLOv3 network is used to roughly locate the license plate area. In the second stage, the improved MobileNetV3 network is used to accurately locate the license plate vertices in the coarse positioning area. Finally, the perspective transformation is used to correct the license plate to the rectangular area.

[0057] YOLOv3 is a widely used target detection network. It has fast detection speed and high accuracy, especially improving the detection effect of small targets. The license plate usually accounts for a small proportion of an image. Using YOLOv3 to roughly locate the license plate area can effectively reduce the number of license plate missed detections. Therefore, YOLOv3 is used in the first stage to roughly locate the license plate area in the image.

[0058] The improved MobileNetV3 network is used to extract license plate vertices in the license plate precision positioning stage. The main reasons are as follows:

[0059] 1) MobileNetV3 is a lightweight network, and its biggest advantage is its high speed. Precise positioning is to extract the vertices of the license plate in the rough positioning area. The input image size at this stage is small and easy to process. The method proposed in the present invention includes two networks. The lightweight network can accurately locate the license plate while maintaining a fast detection speed;

[0060] 2) Unlike common classification networks, the output layer of MobileNetV3 does not use a fully connected layer, but is implemented using a convolutional layer. In addition, the present invention uses a convolutional layer to replace the fully connected layer in the MobileNetV3 network. The advantage of this processing is that the convolutional layer will not destroy the spatial structure of the image. The vertices of the license plate actually have a certain spatial structure in the image. Therefore, the improved MobileNetV3 is suitable for processing precise positioning problems.

[0061] 3) The MobileNetV3 network itself cannot directly handle precise positioning tasks. After adjusting the output layer of the network, the precise positioning function is achieved.

[0062] The present invention provides a two-stage method for accurately extracting non-constrained license plate areas from coarse to fine. The method mainly includes three parts: coarse positioning of the license plate area, precise positioning of the license plate vertices, and license plate extraction. The license plate training set used in the present invention only needs to contain the label information of the four vertices of the license plate. The specific content is as follows:

[0063] (I) Rough positioning of license plate area based on YOLOv3:

[0064] The main purpose of the coarse positioning of the license plate area is to find all the candidate areas of the license plate in the image and reduce the interference of other factors in the picture, so as to facilitate the subsequent accurate positioning of the license plate vertex, so the target detection accuracy and recall rate are required to be high. The size of the license plate in the license plate image obtained from the real scene will be affected by the shooting distance, resulting in inconsistent license plate sizes in the picture samples, and the existing target detection network generally has poor detection effect on small targets at a distance, so how to accurately detect small targets is particularly important. The present invention adopts YOLOv3 to achieve coarse positioning of the license plate area. YOLOv3 adopts a multi-scale strategy to detect targets of different sizes. It divides the target detection network into three scales, namely 13*13, 26*26, and 52*52, and obtains feature maps of targets of different sizes through different scales, thereby realizing large-scale span target detection. In addition, YOLOv3 also adopts a feature fusion strategy to fuse shallow information containing coarse-grained features and deep information containing fine-grained features, further improving the precision of precise detection of targets with less texture, color, and edge information. More importantly, YOLOv3 is a classic one-stage framework that directly treats target detection as a regression problem, performs classification and detection at the same time, locates the bounding box in space, and predicts the box category. The entire process uses only one convolutional neural network and is completed in one go, so it has good performance in real-time target detection. In this stage, when using YOLOv3 to train the license plate coarse positioning model, the network structure is not changed. Only the bounding box annotation strategy in the model training is fine-tuned for the problem itself to reduce the false detection rate and missed detection rate of license plates.

[0065] When using YOLO to train a license plate detection model, the license plate circumscribed rectangular frame is generally used as the training set label. The training set label in the coarse positioning stage of this patent does not directly use the license plate circumscribed rectangular frame, but extends the circumscribed rectangular frame to all sides to obtain a new rectangular frame as the target label. The reasons for adopting this strategy are: ① It expands the target size, indirectly improving the network's detection accuracy for small license plates; ② It makes the detected target include the complete license plate area. The second stage of license plate fine positioning requires determining the license plate vertex position from the coarse positioning area. If the license plate is not completely contained in the coarse positioning area, the fine positioning loses its meaning. Specific strategies such as Figure 2 As shown, Figure 2 The blue box in the middle is the rectangular box surrounding the license plate, and the black box is the target bounding box. The bounding box extension strategy is shown in formula (1) to formula (4).

[0066]

[0067]

[0068]

[0069]

[0070] Where: (x tl ,y tl )、(x dr ,y dr ) are the coordinates of the upper left and lower right vertices of the minimum bounding rectangle, (nx tl ,ny tl )、(nx dr ,ny dr ) are the coordinates of the upper left and lower right vertices of the extended bounding box, w0 and h0 are the width and height of the minimum circumscribed rectangle of the license plate, and W and H are the width and height of the image.

[0071] (II) Precise positioning of license plate vertices based on lightweight network MobileNetV3:

[0072] The precise positioning of the license plate requires locating the four vertices of the license plate in the candidate area obtained by coarse positioning. In this stage, the lightweight network model MobileNetV3-Small is selected to complete the task of precise positioning of the license plate. The main reasons are as follows: ① The image processed in the precise positioning stage of the license plate is only a part of the original image, and the image size is small; ② The license plate positioning uses two convolutional neural network models, which take a long time to run, so a lightweight model is used for precise positioning to reduce time consumption.

[0073] The MobileNetV3 network is usually used to process target classification problems. In order to make it suitable for the task of accurately locating the vertex of the license plate, this paper uses the lightweight network MobileNetV3-Small as the basis and improves it to obtain the AL-MobileNetV3 (Accurate Location based on MobileNetV3) network. The main improvements to the original network are as follows:

[0074] ①. Change the activation function of the original network output layer. The last convolutional layer at the end of the original network uses "softmax" as the activation function, while the present invention uses "sigmoid" as the activation function. The reason is that the value of "softmax" is distributed between 0 and 1 and the sum is 1, and the values ​​are interrelated. "Sigmoid" maps each value to between 0 and 1, and the values ​​are independent of each other. There is actually no specific constraint between the coordinate values ​​of the vertices of the license plate in the real scene, so it is not suitable to use "softmax" activation. In addition, the relative coordinates of the license plate vertices must be between 0 and 1, and the coordinate transformation is continuous without mutation, so the "sigmoid" function is selected;

[0075] ②. Change the structure of the SE (Squeeze-and-Excitation) network sub-block. The original SE sub-block uses a fully connected layer (Dense). The present invention uses a convolutional layer (Conv) to replace the fully connected layer. This is mainly because this stage is to achieve the task of accurately locating the license plate. The spatial position relationship of the pixels in the image plays an important role. Using a fully connected layer will cause the spatial position information feature to be lost. Using a convolutional layer instead of a fully connected layer can effectively retain and learn this position feature, thereby improving the positioning accuracy.

[0076] After the AL-MobileNetV3 network performs a series of feature extraction on the input image, the network finally outputs 8 real numbers between 0 and 1, px0, py0, px1, py1, px2, py2, px3, and py3, which are the relative positions of the four vertices in the image. Finally, the output is further transformed to obtain the specific coordinates of the license plate vertices: lower right (x0, y0), lower left (x1, y1), upper left (x2, y2), and upper right (x3, y3). AL-MobileNetV3 uses mean square error (MSE) as the network's loss function, see formula (5):

[0077]

[0078] Where: (px i ,py i ), (px i ′,py i ′), i = 0, 1, 2, 3 are the vertex positions of the license plate predicted by the network and actually marked respectively.

[0079] (III) License plate correction and extraction:

[0080] In the images obtained from real scenes, most license plates have a certain tilt angle, so the license plate area is generally a regular quadrilateral. The main task of the license plate extraction stage is to intercept the license plate area according to the vertex coordinates obtained by precise positioning, and use image transformation to correct the license plate into a rectangular area.

[0081] The license plate correction of the present invention mainly performs a geometric transformation on the license plate area, that is, through certain known reference points in the image, a mapping relationship between coordinates is established, and the unknown parameters in the mapping relationship are solved, and then other pixel coordinates in the image are corrected according to the mapping relationship. Commonly used image geometric transformation methods include affine transformation and perspective transformation. Affine transformation is a two-dimensional linear transformation, which is mainly used to process image transformations within a single plane. Three pairs of non-collinear corresponding points can uniquely determine the affine transformation mapping relationship; while perspective transformation is a projection transformation in space. Its essence is to project the image into a new viewing plane through a certain mapping relationship. Four pairs of corresponding points are required to determine the perspective transformation. The camera acquires an image by actually projecting one plane onto another plane. This is a typical perspective transformation. The principle of perspective transformation is as follows Figure 3 , see formula (6) to formula (8) for specific implementation:

[0082]

[0083] u=x′ / w′ (7);

[0084] v = y′ / w′ (8);

[0085] in, is the transformation matrix, (x, y) is the original image pixel coordinate, and the corresponding pixel coordinate after transformation is (u, v). Since the perspective transformation in this stage is a plane-to-plane mapping, w=1 is set to obtain formula (9) and formula (10):

[0086]

[0087]

[0088] Among them, k mn =a mn / a 33 , (m=1,2,3, n=1,2,3);

[0089] The four vertex coordinates obtained in the license plate precise positioning stage are lower right (x0, y0), lower left (x1, y1), upper left (x2, y2), and upper right (x3, y3). The data set used in the present invention is mainly small car license plates with a size of 440*140. The corresponding vertex coordinates after correction (u i ,v i ) is (440,140), (0,140), (0,0), (440,0). The transformation matrix can be obtained by formula (9) and formula (10), and then the license plate area can be mapped into the rectangular area to obtain the corrected license plate.

[0090] The specific model training and model detection of the present invention are shown in Figure 4As shown in the figure, the coarse positioning and fine positioning of the license plate use different standards to judge the accuracy of the model detection: 1) In the coarse positioning stage, it is judged whether the real vertex position is within the detection area. If so, the detection is correct, otherwise the detection is wrong; 2) In the fine positioning stage, it is judged whether the IOU between the real vertex and the area surrounded by the detection vertex is greater than a certain threshold. If it is greater than the threshold, the positioning is accurate, otherwise it is inaccurate. In the two stages of license plate positioning, the precision and recall rates are used to evaluate the effects of the two models respectively, so as to select the model with good performance. The calculation of the precision rate (Precision) and the recall rate (Recall) are shown in formula (11) and formula (12). When evaluating the model, if the precision rate and the recall rate are low, the model can be retrained by changing the number of training times, learning rate, etc.

[0091] Precision=TP / (TP+FP) (11);

[0092] Recall = TP / (TP+FN) (12);

[0093] Among them: TP is true positive, indicating the number of correctly detected license plates; FP is false positive, indicating the number of non-license plates detected as license plates; FN is false negative, indicating the number of license plates detected as non-license plates.

Claims

1. A two-stage method for extracting unconstrained license plate areas from coarse to fine, characterized in that: First, the YOLOv3 network is used to roughly locate the license plate area in the license plate image, and then the improved network based on MobileNetV3 is used to locate the vertex position of the license plate in the roughly located area. Finally, the perspective transformation is used to correct the precisely located license plate area into the rectangular area. The specific steps include: Step 1: Rough positioning of the license plate area based on YOLOv3: Use YOLOv3 to train the license plate detection model, extend the outer rectangular box to all sides to obtain a new rectangular box as the target label. The outer rectangular box extension strategy is as shown in formula (1) to formula (4): Where: (x tl ,y tl )、(x dr ,y dr ) are the coordinates of the upper left and lower right vertices of the minimum bounding rectangle, (nx tl ,ny tl )、(nx dr ,ny dr ) are the coordinates of the upper left and lower right vertices of the extended bounding box, w0 and h0 are the width and height of the minimum circumscribed rectangle of the license plate, and W and H are the width and height of the image; Step 2: Precise positioning of license plate vertices based on lightweight network MobileNetV3: Based on the lightweight network MobileNetV3-Small, the last convolutional layer at the end of the network uses "sigmoid" as the activation function; at the same time, the convolutional layer Conv is used to replace the fully connected layer to improve the AL-MobileNetV3 network; After AL-MobileNetV3 performs a series of feature extraction on the input image, it finally outputs 8 real numbers between 0 and 1, px0, py0, px1, py1, px2, py2, px3, and py3, which are the relative positions of the four vertices in the image. Finally, the output is further transformed to obtain the specific coordinates of the license plate vertices: lower right (x0, y0), lower left (x1, y1), upper left (x2, y2), and upper right (x3, y3). AL-MobileNetV3 uses mean square error (MSE) as the network loss function, see formula (5): Where: (px i ,py i ), (px i ′,py i ′), i = 0, 1, 2, 3 are the vertex positions of the license plate predicted by the network and actually marked respectively; Step 3: License plate correction and extraction: License plate correction mainly involves geometric transformation of the license plate area, that is, establishing a mapping relationship between coordinates through some known reference points in the image, solving the unknown parameters in the mapping relationship, and then correcting the coordinates of other pixels in the image according to the mapping relationship; The geometric transformation is a perspective transformation. The principle of perspective transformation is shown in formula (6) to formula (8): u = x′ / w′ (7); v = y′ / w′ (8); in, is the transformation matrix, (x, y) is the original image pixel coordinate, and the corresponding pixel coordinate after transformation is (u, v). Since the perspective transformation in this stage is a plane-to-plane mapping, w=1 is set to obtain formula (9) and formula (10): Among them, k mn =a mn / a 33 , (m=1,2,3, n=1,2,3); The four vertex coordinates obtained in the license plate precise positioning stage are lower right (x0, y0), lower left (x1, y1), upper left (x2, y2), and upper right (x3, y3). After correction, the corresponding vertex coordinates (u i ,v i ), the transformation matrix can be obtained by formula (9) and formula (10), and then the license plate area is mapped into the rectangular area to obtain the corrected license plate.

2. According to claim 1, a two-stage method for accurately extracting non-constrained license plate areas from coarse to fine, characterized in that: Different standards are used to judge the accuracy of model detection for coarse license plate positioning and precise license plate positioning: 1) In the coarse positioning stage, it is determined whether the actual vertex position is within the detection area. If so, the detection is correct, otherwise the detection is incorrect; 2) In the precise positioning stage, it is determined whether the IOU between the real vertex and the area enclosed by the detected vertex is greater than a certain threshold. If it is greater than the threshold, the positioning is accurate, otherwise it is inaccurate; The accuracy and recall rates were used to evaluate the effects of the two models in the two stages of license plate localization.

Images