License plate recognition method, system, device and storage medium

Abstract

This application provides a license plate recognition method, system, device, and storage medium. The method includes: acquiring a license plate region image of a license plate to be recognized; inputting the license plate region image into a key point prediction model to obtain the predicted key point positions of the license plate to be recognized, wherein the predicted key points include predicted corner key points and predicted non-corner key points; calculating a perspective transformation matrix based on preset key point positions of a standard license plate and the predicted key point positions of the license plate to be recognized; correcting the license plate region image according to the perspective transformation matrix to obtain a corrected license plate image; and inputting the corrected license plate image into a license plate character recognition model to obtain license plate characters. By adopting this application, the problem of difficult recognition of low-quality license plates at large angles can be solved more robustly.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a license plate recognition method, system, device and storage medium. Background Technology

[0002] License plate recognition is a challenging yet crucial task in urban traffic management, video surveillance, vehicle identification, and parking management. Factors such as image clarity, lighting conditions, weather conditions, image distortion, and the variability of license plate characters complicate the problem. A robust license plate recognition system should be able to adapt to various environmental changes without sacrificing accuracy.

[0003] License plate images in natural scenes can undergo various deformations due to changes in shooting angle, ground tilt, or non-standard license plate design. These deformations include perspective transformation, bending, tilting, and distortion. Tilt and distortion of license plate images can easily affect the accuracy of subsequent license plate number recognition. Therefore, accurate license plate correction is of significant practical importance. However, traditional correction methods, such as Hough transform and Randon transform, can usually only solve the problem of license plate tilt and are powerless to correct other geometric deformations. Moreover, after license plate correction, a series of precise license plate position localization processes are required, which consumes a lot of computation time and lacks real-time capability. Existing license plate correction methods based on the regression of the four corner points of the license plate are prone to inaccurate corner point localization for distorted or occluded license plates, thus affecting the correction results.

[0004] Existing license plate recognition methods generally include three main steps: license plate detection, license plate correction, and license plate character recognition. License plate detection is used to locate the approximate area of ​​the license plate; license plate correction is used to standardize tilted license plates, which helps improve the accuracy of the next step, license plate character recognition; and license plate character recognition is used to sequentially identify the numbers on the corrected license plate image as characters.

[0005] The existing technical solutions have the following main drawbacks:

[0006] When the acquired license plate image has low quality, partial occlusion, or large angle tilt, the existing license plate correction method based on four corner points does not detect the corner points accurately enough, which often leads to poor correction. Summary of the Invention

[0007] In view of the problems in the prior art, the purpose of this application is to provide a license plate recognition method, system, device and storage medium to more robustly solve the problem of difficult recognition of low-quality license plates at large angles.

[0008] This application provides a license plate recognition method, including the following steps:

[0009] Obtain an image of the license plate area of ​​the vehicle to be identified;

[0010] The license plate area image is input into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points.

[0011] The perspective transformation matrix is ​​calculated based on the preset key point positions of the standard license plate and the predicted key point positions of the license plate to be identified. The preset key points include preset corner key points and preset non-corner key points.

[0012] The license plate area image is corrected according to the perspective transformation matrix to obtain a corrected license plate image;

[0013] The corrected license plate image is input into the license plate character recognition model to obtain the license plate characters.

[0014] This application employs a license plate recognition method that first obtains the predicted key point positions of the license plate region image based on a key point prediction model. Then, it calculates a perspective transformation matrix based on the preset key point positions of a standard license plate and the predicted key point positions of the license plate region image. Finally, it corrects the image based on this perspective transformation matrix. Since the key points include not only corner key points but also other non-corner key points selected as needed, it effectively avoids the problem of inaccurate corner detection affecting the correction effect, thus improving the correction effect of the license plate region image. When performing license plate character recognition, images with better correction effects can obtain more accurate character recognition results.

[0015] In some embodiments, the keypoint prediction model is further trained using the following steps:

[0016] Obtain the sample license plate area image of the sample license plate, and determine the first position of the marked corner key point of the sample license plate;

[0017] The first position of the non-corner key point of the sample license plate is determined based on the preset key point position of the standard license plate and the first position of the marked corner key point of the sample license plate.

[0018] The sample license plate area image is input into the initial key point prediction model to obtain the second position of the labeled key points of the sample license plate output by the initial key point prediction model. The labeled key points include labeled corner key points and labeled non-corner key points.

[0019] A loss function is constructed based on the first and second positions of the labeled key points of the sample license plate, and the initial key point prediction model is optimized and trained based on the loss function to obtain the key point prediction model.

[0020] In some embodiments, determining the first position of the non-corner key point of the sample license plate based on the preset key point position of the standard license plate and the first position of the marked corner key point of the sample license plate includes the following steps:

[0021] Based on the first position of the marked corner key points of the sample license plate and the preset corner key point positions of the standard license plate, a position transformation matrix is ​​constructed;

[0022] Based on the preset non-corner key point positions of the standard license plate and the position transformation matrix, the first position of the labeled non-corner key point of the sample license plate is calculated.

[0023] In some embodiments, for the standard license plate, the area enclosed by the four corner points of the standard license plate is divided into (m-1)×(n-1) grids to obtain m×n grid points. Each grid point is used as a preset key point of the standard license plate, where m≥2, n≥2, and m×n≥4.

[0024] In some embodiments, the keypoint prediction model includes a first feature extraction layer, a keypoint prediction branch, and a local offset prediction branch; the step of inputting the license plate region image into the keypoint prediction model to obtain the predicted keypoint positions of the license plate to be identified includes the following steps:

[0025] The license plate area image is input into the key point prediction model. The image features of the license plate area image are extracted through the first feature extraction layer. The heat map is obtained through the key point prediction branch. The local offset value is obtained through the local offset prediction branch.

[0026] From the heatmap, select peak points whose response values ​​are greater than those of their neighboring points, and determine the location of the peak points;

[0027] Obtain the local offset value at the position corresponding to the peak point, and obtain the corrected position of the peak point based on the position of the peak point and the local offset value;

[0028] The peak point is paired with the preset key point of the standard license plate. The peak point corresponding to the preset key point of the standard license plate is used as the predicted key point. The correction position corresponding to the peak point used as the predicted key point is used as the predicted key point position.

[0029] In some embodiments, the license plate character recognition model includes a second feature extraction layer, a globally separable convolutional layer, a convolutional layer, a reshape layer, and a classification layer connected in series.

[0030] In some embodiments, the corrected license plate image is input into a license plate character recognition model, which is configured to perform the following operations:

[0031] The second feature extraction layer extracts features from the corrected license plate image to obtain image features of dimension C×H×W, where C, H, and W represent the initial dimension, initial height, and initial width of the extracted image features, respectively.

[0032] The globally separable convolutional layer transforms the C×H×W dimension image features into C×1×1 dimension image features;

[0033] The convolutional layer transforms the C×1×1 dimensional image features into A×1×1 dimensional image features, where A = B1*B2, B1 is the preset number of each character category, and B2 is the preset maximum number of characters in a license plate.

[0034] The reshape layer transforms the A×1×1 dimensional image features into B1×B2×1 dimensional image features;

[0035] The classification layer performs classification prediction based on the image features of the B1×B2×1 dimension, and obtains the predicted values ​​of B2 license plate characters.

[0036] This application embodiment also provides a license plate recognition system applied to the aforementioned license plate recognition method, the system comprising:

[0037] The image acquisition module is used to acquire images of the license plate area of ​​the license plate to be identified;

[0038] The key point prediction module is used to input the license plate area image into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points.

[0039] The image correction module is used to calculate a perspective transformation matrix based on the preset key point positions of a standard license plate and the predicted key point positions of the license plate to be identified, wherein the preset key points include preset corner key points and preset non-corner key points; and to correct the license plate area image according to the perspective transformation matrix to obtain a corrected license plate image.

[0040] The license plate recognition module is used to input the corrected license plate image into the license plate character recognition model to obtain the license plate characters.

[0041] This application employs a license plate recognition system. First, through an image acquisition module and a key point prediction module, the predicted key point positions of the license plate region image to be recognized are obtained based on a key point prediction model. Then, through an image correction module, a perspective transformation matrix is ​​calculated based on the preset key point positions of a standard license plate and the predicted key point positions of the license plate region image to be recognized. The image is then corrected based on this perspective transformation matrix. Since the key points include not only corner key points but also other non-corner key points selected as needed, the problem of inaccurate corner point detection in the license plate image affecting the correction effect is effectively avoided, thus improving the correction effect of the license plate region image. When the license plate recognition module performs license plate character recognition, images with better correction effects can obtain more accurate character recognition results.

[0042] This application also provides a license plate recognition device, including:

[0043] processor;

[0044] A memory in which executable instructions of the processor are stored;

[0045] The processor is configured to perform the steps of the license plate recognition method by executing the executable instructions.

[0046] By employing the license plate recognition device provided in this application, the processor executes the license plate recognition method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned license plate recognition method.

[0047] This application also provides a computer-readable storage medium for storing a program, which, when executed by a processor, implements the steps of the license plate recognition method.

[0048] By employing the computer-readable storage medium provided in this application, the program stored therein implements the steps of the license plate recognition method when executed, thereby obtaining the beneficial effects of the above-described license plate recognition method. Attached Figure Description

[0049] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0050] Figure 1 This is a flowchart of a license plate recognition method according to an embodiment of this application;

[0051] Figure 2 This is a schematic diagram of a standard license plate grid according to an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of a grid point prediction model according to an embodiment of this application;

[0053] Figure 4 This is an overall flowchart of license plate correction according to an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of the structure of a license plate character recognition model according to an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of a license plate recognition system according to an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of the structure of a license plate recognition device according to an embodiment of this application;

[0057] Figure 8 This is a schematic diagram of the structure of a computer storage medium according to an embodiment of this application. Detailed Implementation

[0058] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore repeated descriptions of them will be omitted.

[0059] like Figure 1 As shown, in one embodiment, this application provides a license plate recognition method, including the following steps:

[0060] S100: Obtain the license plate area image of the vehicle to be identified;

[0061] The process of obtaining the license plate region image of the license plate to be identified here includes locating the license plate detection box in the original captured image using a license plate detector, and then expanding each side of the detection box outward by a certain size according to the result of the license plate detection box to crop out the license plate region; the license plate detector can be implemented using a machine learning model, for example, it can be trained using the YOLOX object detection algorithm, but this application is not limited to this.

[0062] S200: Input the license plate area image into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points.

[0063] S300: Calculate the perspective transformation matrix based on the preset key point positions of the standard license plate and the predicted key point positions of the license plate to be identified. The preset key points include preset corner key points and preset non-corner key points.

[0064] S400: Correct the license plate area image according to the perspective transformation matrix to obtain a corrected license plate image;

[0065] S500: Input the corrected license plate image into the license plate character recognition model to obtain the license plate characters.

[0066] This application employs a license plate recognition method. First, in steps S100 and S200, the predicted key point positions of the license plate region image to be recognized are obtained based on a key point prediction model. Then, in steps S300 and S400, a perspective transformation matrix is ​​calculated based on the preset key point positions of a standard license plate and the predicted key point positions of the license plate region image to be recognized. The image is then corrected based on this perspective transformation matrix. Since the key points (predicted key points and preset key points of a standard license plate) include not only corner key points but also other non-corner key points selected as needed, the problem of inaccurate corner detection of the license plate image affecting the correction effect is effectively avoided, thus improving the correction effect of the license plate region image. When performing license plate character recognition in step S500, images with better correction effects can obtain more accurate character recognition results.

[0067] In this embodiment, the license plate recognition method further includes training the key point prediction model using the following steps:

[0068] Obtain the license plate area image of the sample license plate and determine the first position of the marked corner key point of the sample license plate; here, the first position of the marked corner key point of the sample license plate can be determined by the first position of the corner point marked manually; or it can be automatically detected by training a corner detector based on the obvious features of the corner point.

[0069] The first position of the marked non-corner key point of the sample license plate is determined based on the preset non-corner key point position of the standard license plate and the first position of the marked corner key point of the sample license plate.

[0070] The sample license plate area image is input into the initial key point prediction model to obtain the second position of the labeled key points of the sample license plate output by the initial key point prediction model. The labeled key points include labeled corner key points and labeled non-corner key points.

[0071] A loss function is constructed based on the first and second positions of the labeled key points of the sample license plates, and the initial key point prediction model is trained and optimized based on the loss function to obtain the key point prediction model. The key point prediction model is obtained by training the initial key point prediction model until the loss function converges.

[0072] The license plate recognition method of this application is based on more key points on the license plate, which can more robustly correct the license plate. The selected key points do not necessarily have to be the four corner points with obvious features on the license plate. They can be points at any proportional position on the license plate, such as the point at one-quarter of the horizontal coordinate and one-fifth of the vertical coordinate. Therefore, some key points can be evenly set according to the positions of the four corner points. In this embodiment, for a standard license plate, the area enclosed by the four corner points of the standard license plate is divided into (m-1)×(n-1) grids, resulting in m×n grid points. Each grid point is used as a preset key point of the standard license plate, where m≥2, n≥2, and m×n≥4. Thus, the positions of more key points of the sample license plate can be determined through the preset key point positions of the standard license plate, rather than just obtaining the positions of the corner key points. Figure 2 As shown, the rectangular area enclosed by the four corner points of a standard license plate is divided into 4×2 grids, resulting in 5×3 grid points, or a total of 15 preset key points, including four preset corner key points and 11 preset non-corner key points. For the standard license plate, the coordinates of these 15 preset key points are predetermined. By determining the first position of the marked non-corner key points of the sample license plate based on the positions of the preset non-corner key points of the standard license plate and the first position of the marked corner key points of the sample license plate, the coordinates of the 11 marked non-corner key points of the sample license plate can be determined. The values ​​of m and n here are merely examples. In other embodiments, other values ​​can be used, such as dividing the standard license plate into a 2×1 grid to determine 3×2 = 6 preset key points, or dividing the standard license plate into a 2×2 grid to determine 3×3 = 9 preset key points, etc. In other alternative implementations, the preset key points may not be grid points, but key points in other locations determined in advance in the standard license plate. For example, a preset key point may be selected in the center, upper left, or lower right area of ​​the standard license plate. Then, the position of the marked key points on the sample license plate according to the standard license plate will correspond to the preset key points of the standard license plate.

[0073] The following uses the preset key points as grid points as an example to illustrate the specific implementation method of license plate correction and recognition. The key point prediction model is the grid point prediction model. The labeled key points and predicted key points are the labeled grid points and predicted grid points, respectively. It is understood that the following method is also applicable to other types of key points.

[0074] In this embodiment, determining the first position of the marked non-corner key point of the sample license plate based on the preset non-corner key point position of the standard license plate and the first position of the marked corner key point of the sample license plate includes the following steps:

[0075] Based on the first position of the marked corner key points of the sample license plate and the preset corner key point positions of the standard license plate, a position transformation matrix is ​​constructed;

[0076] Based on the preset non-corner key point positions of the standard license plate and the position transformation matrix, the first position of the marked non-corner key points of the license plate to be identified is calculated.

[0077] Therefore, this application only needs to calculate the first position of all the marked grid points based on the first position coordinates of the four marked corner grid points of the manually marked sample license plate. Let the position coordinates of the four marked corner grid points of the sample license plate be (x1,y1), (x2,y2), (x3,y3), and (x4,y4), and the position coordinates of the four preset corner grid points of the standard license plate be (0,0), (440,0), (440,140), and (0,140). At this time, there are 4 source coordinates (position coordinates of the four marked corner grid points of the sample license plate) and 4 target coordinates (position coordinates of the four corner grid points of the standard license plate). Through these 4 pairs of corresponding coordinates, 8 equations can be constructed, and 8 unknowns can be calculated. These 8 solutions and 1 can form a 3×3 matrix, which is the position transformation matrix M1 from the position coordinates of the four marked corner grid points of the sample license plate to the position coordinates of the four preset corner grid points of the standard license plate. Conversely, for any preset key point on a standard license plate, the corresponding labeled key point on the sample license plate can be obtained by multiplying it by the inverse of the position transformation matrix M1. Therefore, multiplying the coordinates of the non-corner grid points on the standard license plate by the inverse of the position transformation matrix M1 yields the first position of the labeled non-corner grid points on the sample license plate.

[0078] Specifically, the formula for constructing the position transformation matrix M1 from the coordinates of the four marked corner grid points of the sample license plate to the coordinates of the four preset corner grid points of the standard license plate is as follows, where,

[0079]

[0080] Where xyz represents the coordinates of the labeled corner points of the sample license plate area image on the three coordinate axes, and x'y'z' represents the coordinates of the preset corner points of the standard license plate on the three coordinate axes. Since the license plate area image is a two-dimensional image, only two coordinate axes are actually used, so the coordinates of the third coordinate axis z and z' are 1; through 4 pairs of matched points (xyz→x'y'z'), these nine parameters a can be calculated. 11 ~a 33 In practice, multiplying by a non-zero factor is usually performed to make the last parameter a... 33 Since the value is 1, we only need to calculate the other 8 parameters to obtain the position transformation matrix M1.

[0081] In this embodiment, the grid point prediction model mainly performs two sub-tasks: grid point localization and grid point local offset correction. Specifically, the grid point prediction model includes a first feature extraction layer, a grid point prediction branch, and a local offset prediction branch. Figure 3 The diagram illustrates the structure of an example grid point prediction model. The first feature extraction layer comprises a backbone network and a head network. The backbone network uses a ResNet18 network with half the channels. The head network employs an upsampling operation to transform the feature map to a certain size, making the size of the predicted feature map one-quarter of the input image. To improve the grid point prediction performance, this embodiment uses a feature pyramid fusion method that fuses low-level and high-level features of the network, enhancing the distinguishability of the features.

[0082] Step S200: Input the license plate area image into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points, including the following steps:

[0083] The license plate area image is input into the grid point prediction model. The image features of the license plate area image are extracted through the first feature extraction layer. The heat map is obtained through the grid point prediction branch. The local offset value is obtained through the local offset prediction branch.

[0084] From the heatmap, select peak points whose response values ​​are greater than those of their neighboring points to determine the location of the peak points; specifically, on the grid point prediction branch, extract the peak points on the heatmap by comparing the response values ​​of all response points on the heatmap with the response values ​​of their 8 connected neighboring points, and take the response points whose response values ​​are greater than or equal to the response values ​​of their 8 connected neighboring points as peak points.

[0085] In this embodiment, when training the grid point prediction branch, for a labeled grid point c, whose position is p, the corresponding grid point position on the downsampled low-resolution image is calculated. R is the downsampling step size. Then, all grid points (x, y) of the labels are passed through a Gaussian kernel. Calculation to generate heatmap Where σ p C' is the standard deviation of the target scale adaptation, C′ is the number of channels, which is also the number of grid points, that is, each key point is predicted on different feature channels, and W' and H' are the width and height of the image features after downsampling, respectively.

[0086] The loss function for training the grid point heatmap is the focus loss function of pixel-level logistic regression, and the calculation formula is as follows:

[0087]

[0088] α and β are the hyperparameters of the loss function, which were set to 2 and 4 respectively in the experiment. N is the number of grid points in the image. Dividing by N is mainly to normalize all loss values. This is a heatmap of grid point prediction branch predictions, Y x,y,c′ This is a sample heatmap.

[0089] The local offset value at the position corresponding to the peak point is obtained, and the corrected position of the peak point is obtained based on the position of the peak point and the local offset value. Since the grid point position will be deviated due to the discreteness of the data during image downsampling, a local offset is predicted for each grid point. The local offset values ​​of all grid points are predicted on the same channel. This local offset prediction branch is trained using the following loss function:

[0090]

[0091] in, To predict local offset values.

[0092] Specifically, the accurate predicted location of a grid point is obtained by adding the local offset value of the offset correction to the location of the peak point. When outputting the prediction results, only grid points with a confidence level greater than 0.4 are output. Here, the confidence level refers to the response value of each grid point, and 0.4 is the set threshold, which can be selected as needed.

[0093] The peak point is paired with the preset grid point of the standard license plate. The peak point corresponding to the preset grid point of the standard license plate is used as the prediction grid point. The correction position corresponding to the peak point, which is used as the prediction key point, is used as the position of the prediction grid point.

[0094] In most cases, the grid point prediction model can obtain relatively accurate positions of each predicted grid point. However, in practical use, due to the complexity of the environment, the images captured by the camera are affected by factors such as lighting and motion, often resulting in images of poor quality. Feeding such low-quality images into the model may lead to the prediction of some incorrect grid points, missed detections, or even duplicate grid points. In other words, the predicted grid point positions obtained by the model and the preset grid point positions of the standard license plate may not correspond one-to-one in some cases, resulting in some incorrect pairings. To accommodate these situations, this embodiment uses RANSAC (Random Sample Consensus) to filter mismatched points when calculating the perspective transformation matrix. This method iteratively distinguishes correct and incorrect pairings from a set of observation datasets containing "outsiders," and the final iteration almost entirely uses the correct pairings to calculate the model parameters of the perspective transformation matrix. As can be seen from the method of calculating the perspective transformation matrix, compared to the existing method that only uses four corner points, the multiple keypoint calculation method of this application is more robust and has better compatibility.

[0095] In actual coding, the `findHomograpy` function in OpenCV can be used, with RANSAC parameters passed in (RANSAC parameters include, for example: n—minimum number of samples (e.g., 2 for a line); t—fault tolerance threshold (e.g., designed so that all points within 0.25m of a line are inliers); k—number of iterations (e.g., 1000 iterations); w—proportion of inliers in the dataset (e.g., 70% are inliers); p—success rate (e.g., the desired probability of success is 0.99); d—minimum number of consistency sets (the minimum number of inliers required to generate the model)). This allows for robust calculation of the optimal perspective transformation matrix between the predicted and labeled grid points. Figure 4 The flowchart illustrates the overall process of license plate correction. With the perspective transformation matrix M, the corrected image can be obtained by multiplying it with the input license plate region image. It's important to note that if a large number of mismatched points are calculated, the image may not contain the license plate or the image quality may be too poor, and it should be discarded.

[0096] Current license plate recognition methods are generally based on CTC (Connectionist Temporal Classification), but existing CTC-based methods are more suitable for single-layer license plate recognition and less effective for two-layer license plate recognition. Therefore, this embodiment designs a method for license plate number recognition that is compatible with both single-layer and two-layer license plates, based on global feature classification. This method, like CTC-based methods, is easy to implement and deploy on various platforms.

[0097] In this embodiment, the license plate character recognition model includes a second feature extraction layer, a globally separable convolutional layer (GDConv), a convolutional layer (Conv), a reshape layer, and a classification layer (such as a SoftMax classification layer) connected in sequence. Figure 5 The structure of the license plate character recognition model in this embodiment is shown. The second feature extraction layer can employ a CNN (Convolutional Neural Network) backbone network to extract features from the input corrected license plate image. This license plate character recognition model uses a ResNet18 with half the channels as the base network, and the details of the entire recognition model network structure are shown in Table 1.

[0098] Table 1. Network structure of the license plate recognition model in this application.

[0099]

[0100] In this embodiment, in step S500, the corrected license plate image is input into the license plate character recognition model, which is configured to perform the following operations:

[0101] The second feature extraction layer extracts features from the corrected license plate image to be identified, obtaining image features of dimension C×H×W, where C, H, and W represent the initial dimension, initial height, and initial width of the extracted image features, respectively.

[0102] The globally separable convolutional layer transforms the C×H×W dimension image features into C×1×1 dimension image features. In this embodiment, before connecting to the classifier, fully connected layers and global pooling layers are discarded, and globally separable convolutional layers are used instead. Using globally separable convolutional layers has two advantages: firstly, compared to fully connected layers, a large number of parameters can be reduced, alleviating the overfitting problem; secondly, compared to global pooling layers, information is not lost due to averaging operations, but like global pooling layers, the model can implicitly encode the license plate character positions into the features.

[0103] The convolutional layer transforms the C×1×1 dimensional image features into A×1×1 dimensional image features, where A = B1*B2, B1 is the preset number of each character category, and B2 is the preset maximum number of characters in a license plate. In this embodiment, B1 is 83, B2 is 9, and A is 747 as an example. Specifically, different dimensions can be set according to different recognition tasks. The 1×1 convolutional layer following the globally separable convolutional layer is to change the number of channels to 747. In the license plate recognition task, considering the 8-digit new energy vehicle license plate number, including the end character, the maximum number of characters to be recognized is 9. The license plate character categories include Chinese characters, letters, numbers, and end characters, totaling 83 types. Therefore, the channel dimension here is changed to 9×83=747.

[0104] The reshape layer transforms the A×1×1 dimensional image features into B1×B2×1 dimensional image features. In this embodiment, the 747-dimensional features output by the preceding convolutional layer are transformed into 83×9×1 dimensions, which corresponds exactly to the 83 categories of the 9 character positions of the license plate number. This makes it convenient to connect only one classifier to perform classification training for the 9 characters.

[0105] The classification layer performs classification prediction based on the image features of the B1×B2×1 dimension, and obtains the predicted values ​​of B2 license plate characters.

[0106] Due to the imbalance in license plate character data across provinces, this application employs a focus loss function instead of the SoftMax loss function to alleviate the problem of imbalanced Chinese character data. The focus loss function is an improvement upon the standard SoftMax loss function, which reduces the weights of easily classified samples, allowing the model to focus more on difficult-to-classify samples during training. The formula for the focus loss function is as follows:

[0107] FL(P t )=-α(1-P t ) γ log(P t )

[0108] Where P I This is the probability score output by the SoftMax function. α and γ are hyperparameters used in the experiment. According to the experimental results in the paper "Focal Loss for Dense Object Detection", setting α and γ to 0.25 and 2 respectively can achieve the optimal experimental results.

[0109] This application validates the performance of the license plate recognition method through experiments. In the experiments, the backbone network of the grid point prediction model was a ResNet18 with half the channels. The resolution of the license plate region image to be recognized was 128×128. Grid points and local offset values ​​were predicted on the generated 32×32 feature map. The objective function was optimized using the Adam optimization method and cosine learning rate decay. The training batch size was 32, the initial learning rate was 0.01, and the number of iterations was 50 epochs.

[0110] The backbone network of the license plate character recognition model is also a ResNet18 with half the number of channels, and the input resolution is 64×176. The objective function is optimized using SGD and cosine learning rate decay. The training batch size is 32, the initial learning rate is 0.005, and the number of iterations is 20 epochs.

[0111] Training was conducted on the Caffe deep learning framework, with a single NVIDIA GTX 1080Ti graphics card as the hardware platform.

[0112] The training dataset consists of images from the public CCPD set, license plates generated by OpenCV, and images from surveillance scenes captured by private cameras. To train the grid point prediction model and the license plate character recognition model, the coordinates of the four corner points and the character numbers of the license plates need to be labeled. The CCPD public set and the generated license plates already contain these labels; therefore, only the private data needs to be labeled.

[0113] Through testing, the license plate recognition method of this application not only robustly recognizes license plates with large angles and low quality, but also provides good support for double-layer license plates. During training, a rich collection of image samples covering various provinces can be used to alleviate the problem of recognition errors caused by character class imbalance.

[0114] like Figure 6 As shown, this application embodiment also provides a license plate recognition system applied to the aforementioned license plate recognition method, the system comprising:

[0115] The image acquisition module M100 is used to acquire the license plate area image of the license plate to be identified;

[0116] The key point prediction module M200 is used to input the license plate area image into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points.

[0117] The image correction module M300 is used to calculate a perspective transformation matrix based on the preset key point positions of the standard license plate and the predicted key point positions of the license plate to be identified. The preset key points include preset corner key points and preset non-corner key points. The module also corrects the license plate area image according to the perspective transformation matrix to obtain a corrected license plate image.

[0118] The license plate recognition module M400 is used to input the corrected license plate image into the license plate character recognition model to obtain the license plate characters.

[0119] This application employs a license plate recognition system. First, the image acquisition module M100 and the key point prediction module M200 obtain the predicted key point positions of the license plate area image based on the key point prediction model. Then, the image correction module M300 calculates the perspective transformation matrix based on the preset key point positions of the standard license plate and the predicted key point positions of the license plate area image, and corrects the image based on the perspective transformation matrix. Since the key points (predicted key points and preset key points of the standard license plate) include not only corner key points but also other non-corner key points selected as needed, the problem of inaccurate corner point detection of the license plate image affecting the correction effect is effectively avoided, thus improving the correction effect of the license plate area image. When the license plate recognition module M400 performs license plate character recognition, the image with better correction effect can obtain more accurate character recognition results.

[0120] This application also provides a license plate recognition device, including a processor; a memory storing executable instructions of the processor; wherein the processor is configured to perform the steps of the license plate recognition method by executing the executable instructions.

[0121] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0122] The following reference Figure 7 To describe an electronic device 600 according to this embodiment of the present application. Figure 7 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0123] like Figure 7As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0124] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the above-described section of the electronic prescription transfer processing method according to various exemplary embodiments of this application. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0125] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0126] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0127] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0128] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0129] By employing the license plate recognition device provided in this application, the processor executes the license plate recognition method when executing the executable instructions, thereby obtaining the beneficial effects of the above-mentioned license plate recognition method.

[0130] This application also provides a computer-readable storage medium for storing a program that, when executed by a processor, implements the steps of the license plate recognition method. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the above-described electronic prescription processing method section of this specification according to various exemplary embodiments of this application.

[0131] refer to Figure 8 As shown, a program product 800 for implementing the above-described method according to an embodiment of this application is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0132] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0133] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0134] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to external computing devices (e.g., via the Internet using an Internet service provider).

[0135] By employing the computer-readable storage medium provided in this application, the program stored therein implements the steps of the license plate recognition method when executed, thereby obtaining the beneficial effects of the above-described license plate recognition method.

[0136] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the scope of protection of this application.

Claims

1. A license plate recognition method, characterized by, Includes the following steps: Obtain an image of the license plate area of ​​the vehicle to be identified; The license plate area image is input into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points. The perspective transformation matrix is ​​calculated based on the preset key point positions of the standard license plate and the predicted key point positions of the license plate to be identified. The preset key points include preset corner key points and preset non-corner key points. The license plate area image is corrected according to the perspective transformation matrix to obtain a corrected license plate image; The corrected license plate image is input into the license plate character recognition model to obtain the license plate characters; The key point prediction model includes a first feature extraction layer and a key point prediction branch; the step of inputting the license plate area image into the key point prediction model to obtain the predicted key point positions of the license plate to be identified includes the following steps: The license plate area image is input into the key point prediction model, the image features of the license plate area image are extracted through the first feature extraction layer, and the heat map is obtained through the key point prediction branch. From the heatmap, select peak points whose response values ​​are greater than those of their neighboring points, and determine the location of the peak points; The peak point is paired with the preset key point of the standard license plate, and the peak point corresponding to the preset key point of the standard license plate is used as the predicted key point. The position of the predicted key point is determined according to the position of the peak point.

2. The license plate recognition method of claim 1, wherein, It also includes training the keypoint prediction model using the following steps: Obtain the sample license plate area image of the sample license plate, and determine the first position of the marked corner key point of the sample license plate; The first position of the non-corner key point of the sample license plate is determined based on the preset key point position of the standard license plate and the first position of the marked corner key point of the sample license plate. The sample license plate area image is input into the initial key point prediction model to obtain the second position of the labeled key points of the sample license plate output by the initial key point prediction model. The labeled key points include labeled corner key points and labeled non-corner key points. A loss function is constructed based on the first and second positions of the labeled key points of the sample license plate, and the initial key point prediction model is optimized and trained based on the loss function to obtain the key point prediction model.

3. The license plate recognition method of claim 2, wherein, The step of determining the first position of the non-corner key point of the sample license plate based on the preset key point position of the standard license plate and the first position of the marked corner key point of the sample license plate includes the following steps: Based on the first position of the marked corner key points of the sample license plate and the preset corner key point positions of the standard license plate, a position transformation matrix is ​​constructed; Based on the preset non-corner key point positions of the standard license plate and the position transformation matrix, the first position of the labeled non-corner key point of the sample license plate is calculated.

4. The license plate recognition method of claim 2, wherein, For the standard license plate, the area enclosed by the four corner points of the standard license plate is divided into (m-1)×(n-1) grids, resulting in m×n grid points. Each grid point is used as a preset key point of the standard license plate, where m≥2, n≥2, and m×n≥4.

5. The license plate recognition method of claim 1, wherein, The keypoint prediction model also includes a local offset prediction branch; After inputting the license plate area image into the key point prediction model, the local offset value is also obtained through the local offset prediction branch; The predicted key point location is determined based on the location of the peak point using the following steps: Obtain the local offset value at the position corresponding to the peak point, and obtain the corrected position of the peak point based on the position of the peak point and the local offset value; The correction position corresponding to the peak point of the predicted key point is taken as the position of the predicted key point.

6. The license plate recognition method of claim 1, wherein, The license plate character recognition model includes a second feature extraction layer, a globally separable convolutional layer, a convolutional layer, a reshape layer, and a classification layer, which are connected in sequence.

7. The license plate recognition method of claim 6, wherein, The corrected license plate image is input into a license plate character recognition model, which is configured to perform the following operations: The second feature extraction layer extracts features from the corrected license plate image to obtain image features of dimension C×H×W, where C, H, and W represent the initial dimension, initial height, and initial width of the extracted image features, respectively. The globally separable convolutional layer transforms the C×H×W dimension image features into C×1×1 dimension image features; The convolutional layer converts the image features of the C×1×1 dimension into image features of an A×1×1 dimension, where A=B1 B2, B1 are preset numbers of each character category, and B2 is a preset maximum number of characters of the license plate. The reshape layer transforms the A×1×1 dimensional image features into B1×B2×1 dimensional image features; The classification layer performs classification prediction based on the image features of the B1×B2×1 dimension, and obtains the predicted values ​​of B2 license plate characters.

8. A license plate recognition system characterized by, The system, applied to the license plate recognition method according to any one of claims 1 to 7, comprises: The image acquisition module is used to acquire images of the license plate area of ​​the license plate to be identified; The key point prediction module is used to input the license plate area image into the key point prediction model to obtain the predicted key point positions of the license plate to be identified. The predicted key points include predicted corner key points and predicted non-corner key points. The image correction module is used to calculate a perspective transformation matrix based on the preset key point positions of a standard license plate and the predicted key point positions of the license plate to be identified, wherein the preset key points include preset corner key points and preset non-corner key points; and to correct the license plate area image according to the perspective transformation matrix to obtain a corrected license plate image. The license plate recognition module is used to input the corrected license plate image into the license plate character recognition model to obtain the license plate characters.

9. A license plate recognition apparatus characterized by comprising: include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the license plate recognition method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the license plate recognition method according to any one of claims 1 to 7.

Images