License plate image automatic correction module driven by weak supervision license plate classification information and training strategy thereof

Abstract

According to a license plate image automatic correction module driven by weak supervision license plate classification information and a training strategy thereof, a license plate vertex regression sub-network extracts position information of license plate vertexes by using a convolutional network, and a final coordinate regression amount is obtained through integration of a full connection layer; the transformation matrix solving module calculates a perspective inverse transformation matrix through perspective inverse transformation according to the vertex coordinates of the license plate and the corrected output license plate image size; the grid generation module determines a corresponding position relationship between the input license plate image and the output license plate image, and obtains a corresponding relationship of all positions through mapping; and the interpolation sampling module acts the mapping relation TM of the grid generation module on the pixel points of the input license plate image, extracts the pixel values on the corresponding coordinates of the input license plate image, and fills the pixel values on the corresponding points of the output license plate image so as to generate a converted image. According to the method, the problem that the commonly used LPRNet recognition network cannot recognize the double-row license plate is solved, and the interpretability of the result of each part of the correction algorithm module is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of license plate recognition, and in particular to a license plate image automatic correction module driven by weakly supervised license plate classification information and a training strategy thereof. Background Art

[0002] Open license plate recognition technology has a wide range of application scenarios, such as parking lot management, road traffic monitoring, toll station automation, judicial evidence collection, etc. Its biggest advantage is that it does not require additional constraints and restrictions on moving vehicles, and can be applied even when the vehicle is moving and the shooting angle is greatly deviated. However, due to the influence of complex environmental light, weather conditions, license plate damage and other factors, especially the unpredictable shooting angle, open license plate recognition technology still has many problems that need to be overcome.

[0003] In the license plate recognition system, the license plate correction module is particularly important, because whether the license plate image area is at a straight-view angle has an important impact on the subsequent license plate character recognition accuracy, such as Figure 1 This is especially true for license plate recognition systems in non-constrained scenarios, where the system not only has to deal with the problems caused by the affine transformation of the license plate image, but also needs to correct the possible perspective transformation.

[0004] From the perspective of whether the vertex coordinates of the license plate need to be labeled, the currently widely used license plate correction algorithms can be divided into two categories: the first category is to obtain the four vertex coordinates of the license plate area through the license plate positioning module, and then calculate the perspective transformation matrix based on the relationship between these coordinates and the required correction image size.

[0005] For example, in the literature [1]: Xu Guangzhu, Kuang Wan, Li Xingwei, et al. License plate localization based on YOLOv3 and vertex offset estimation [J]. Journal of Computer-Aided Design and Graphics, 2021, 33(04): 569-579. In the disclosed technical solution, the extended model of YOLO is used to regress the vertex coordinates of the license plate, and then the perspective transformation is performed;

[0006] The Chinese patent "A method for correcting tilted license plates based on deep learning" (application number: 202110767061.0) builds a visualization tool for marking the corner points of license plates and performs corresponding perspective transformations according to the type of license plates. On the one hand, this type of algorithm requires effective annotation of the vertices of license plates in large-scale license plate datasets, and on the other hand, requires the positioning algorithm to accurately estimate the vertex coordinates. This limits its application in many cases.

[0007] Another type of license plate image correction algorithm uses a spatial transformation network to adjust parameters using the supervision information back-propagated from the subsequent license plate recognition network in a machine learning manner. Although it also uses a supervised learning method, this method does not require the information of the vertex coordinates of the license plate. Compared with the spatial transformation network, it is equivalent to a weak supervision method. For example, the LPRNet network uses STN to achieve automatic correction of the rotation transformation of the license plate area. Reference [2]: Li Wenjie, Zhang Zusheng, Zhou Kunxiao, et al. License plate recognition algorithm RD-LPRNet based on image correction and denoising [J]. Journal of Dongguan University of Technology, 2023, 30(05): 22-33. In the disclosed technical solution, by improving the STN network, the correction network RNet and the denoising network DNet are combined to construct the correction network RDNet. The network predicts the affine transformation parameters by creating a smaller image copy, and then maps these parameters to the original image to achieve correction of the target area. The Chinese patent "A method and handheld terminal for license plate detection and recognition" (application number: 202011079512.3) obtains a secondary correction network by performing an affine transformation on the initially corrected license plate image through STN. This type of correction algorithm uses learning to automatically regress the transformation parameters of the spatial transformation matrix.

[0008] Reference [3]: Jaderberg M, Simonyan K, Zisserman A. Spatial transformer networks [J]. Advances in neural information processing systems, 2015, 28. In the public technical solution, the six parameters required for the affine transformation are regressed.

[0009] Reference [4]: ​​Li H, Wang P, You M, et al. Reading car license plates using deep neural networks [J]. Image and Vision Computing, 2018, 7214-23. In the disclosed technical solution, the six parameters required for affine transformation are predicted in a weakly supervised manner, and then the bidirectional long short-term memory network and CTC algorithm are used to realize the recognition of license plate characters without segmentation. This method is suitable for the license plate correction problem corresponding to affine transformation, but it cannot achieve effective correction for the more widely existing perspective transformation situation.

[0010] Compared with affine transformation, license plate images that have undergone perspective transformation are more common in non-constrained scenarios. The biggest difference between perspective transformation and affine transformation is that the parallel relationship between parallel lines in the image before and after the transformation is no longer guaranteed, such as Figure 2As shown. From the perspective of the transformation matrix, it can be clearly seen that the perspective transformation is more complicated than the affine transformation. The perspective transformation formula is shown in formula (1):

[0011]

[0012] The parameter h in formula (1) 13 、h 23 and parameter h 11 、h 12 、h 21 、h 22 They correspond to the translation and in-plane rotation problems in affine transformation. The perspective transformation also contains an additional parameter h 31 、h 32 and h 33 , when the parameter h 33 is 1 and the parameter h 31 、h 32 When it is 0, the perspective transformation degenerates into an affine transformation, so the affine transformation is a special case of the perspective transformation.

[0013] If the STN network currently widely used in license plate correction is extended to regress the perspective transformation matrix, the problem of training convergence often occurs. This is mainly because there are mutual influences between the parameters in the perspective transformation. Due to the high coupling between the parameters of the perspective transformation matrix, it means that a small fluctuation of a single parameter can cause complex and nonlinear changes in the output image. Usually, another parameter needs to be changed to slow down this change. For example, increasing the perspective parameter h 31 and h 32 This may cause severe distortion of the image. In order to reduce this distortion, it may be necessary to adjust the scaling parameter h. 11 and h 12 This entanglement relationship between parameters makes it impossible for weak supervision information based on subsequent license plate recognition modules to effectively guide the adjustment of STN network parameters. Therefore, the STN currently used for license plate correction is to deal with affine transformation correction, which greatly limits its application in license plate correction in non-constrained environments.

[0014] In addition, most of the current license plate recognition research is still focused on single-line character license plates, such as references [5]: Hu Aiming, Zhou Xiaokuan. Car license plate image segmentation method using morphological features [J]. Journal of Computer-Aided Design and Graphics, 2003, (06): 716-719. and reference [6]: Ta TD, Le DA, Le MT. Automatic number plate recognition on electronic toll collection system for Vietnamese conditions [C]. New York: ACM Press, 2015: Article No. 29.

[0015] On the one hand, most of the currently widely used large-scale license plate image datasets only contain single-line character license plate images. Even if they contain double-line character license plates, their number accounts for a very small proportion of the entire dataset. For example, CCPDv1 and v2 only contain blue and green single-line character license plates; the CBLPRD-330k dataset contains only more than 300 double-line license plate images. On the other hand, license plate recognition algorithms mainly use OCR (optical character recognition) models with CRNN (a combination of CNN and RNN) architecture, such as paddleOCR, trOCR, etc., or pure CNN architecture models, such as LPRNet and its variants, and cooperate with CTC loss, but these algorithms only support the recognition of single-line characters. This makes it impossible for the STN correction network for double-line license plates to effectively obtain supervision information through the subsequent recognition network, because the subsequent recognition network cannot directly identify double-line license plates. Therefore, many current license plate recognition studies based on STN correction networks are still aimed at single-line character license plates. Summary of the invention

[0016] In view of the above problems, the present invention proposes a license plate image automatic correction module driven by weakly supervised license plate classification information and its training strategy. Through a perspective transformation correction network PTN (PerspectiveTransformation Network) designed by STN and combined with the weak supervision information provided by the license plate view type binary classification network, the automatic correction problem of double-line and single-line character license plates is uniformly solved. The problem that the commonly used LPRNet recognition network cannot recognize double-line license plates is solved, and the interpretability of the results of each part of the correction algorithm module is improved.

[0017] The technical solution adopted by the present invention is:

[0018] The license plate image automatic correction module driven by weakly supervised license plate classification information includes:

[0019] License plate vertex regression subnet, transformation matrix solution module, grid generation module, interpolation sampling module;

[0020] The license plate vertex regression subnet includes multiple convolutional layers and fully connected layers, which use the convolutional network to extract the position information of the license plate vertices and integrate them through the fully connected layer to obtain the final coordinate regression value;

[0021] The transformation matrix solving module calculates the perspective inverse transformation matrix through perspective inverse transformation according to the coordinates of the license plate vertices and the corrected output license plate image size;

[0022] The grid generation module determines the corresponding position relationship between the input license plate image and the output license plate image, and determines the mapping of the position relationship to obtain the corresponding relationship of all positions;

[0023] The interpolation sampling module converts the mapping relationship T between the input license plate image and the output license plate image obtained by the grid generation module M It acts on the pixel points of the input license plate image, then extracts the pixel values ​​at the corresponding coordinates of the input license plate image and fills them into the corresponding points of the output license plate image to generate the converted image.

[0024] In the input license plate image area, the license plate vertex regression subnet first extracts features from the input license plate image area, and then uses the fully connected layer to perform high-level feature extraction on the input license plate image; the network structure of the license plate vertex regression subnet includes 7 convolutional layers and 3 fully connected layers.

[0025] The transformation matrix solving module comprises the following steps:

[0026] (1): First input the source point coordinates M i (X mi ,Y mi ), and the target point coordinates N i (X ni ,Y ni ), i=1,2,3,4;

[0027] (2): Determine M i and N i Are the dimensions of equal and greater than or equal to 4? If so, initialize the P matrix and the Q matrix; if not, terminate the step;

[0028] (3): Solve the linear equation system P*A=Q by mathematical methods; the details are as follows:

[0029] 4 sets of coordinate vertices, where (U mi ,V mi )(i=1,2,3,4) represent the coordinates of the four target points in the upper left, upper right, lower left and lower right of the output license plate image, respectively.ni ,Y ni )(i=1,2,3,4) represent the coordinates of the four source points of the upper left, upper right, lower left and lower right of the input license plate image. Then, through these 8 points, we can get the linear equation system corresponding to Formula 1, where the matrix on the left is denoted as P and the matrix on the right is denoted as Q. Then, by solving the linear equation system, A=P -1 *Q can get 8 parameters (a11, a12, a13, a21, a22, a23, a31, a32). These 8 parameters are the first 8 parameters of the perspective inverse transformation matrix A. The 9th parameter is the scale factor, which is usually set to 1. Then these 9 parameters are integrated to get a perspective inverse transformation matrix A with a shape of 3*3.

[0030]

[0031] (4): Post-process the perspective inverse transformation matrix; specifically: set the scale factor to 1 and combine it with the 8 parameters obtained in the previous step, and then obtain a perspective inverse transformation matrix A with a shape of 3*3.

[0032] (5): Return the final perspective inverse transformation matrix A;

[0033] Among them, M i (X mi ,Y mi ) are the coordinates of the four vertices of the license plate in the input license plate image; N i (X ni ,Y ni ) are the coordinates of the four corner points in the output license plate image; A is the perspective inverse transformation matrix; P is the coordinate set of each point in the quadrilateral area surrounded by the coordinates of the four vertices of the license plate in the input license plate image; Q is the matrix composed of the coordinates of the corresponding pixels of the output license plate image.

[0034] By solving the transformation matrix, we can obtain a 3*3 perspective inverse transformation matrix A.

[0035] The grid generation module first receives the output perspective inverse transformation matrix A and the pixel coordinates of the output license plate image, and calculates the pixel coordinates of the input license plate image through the above information;

[0036] Next, the perspective inverse transformation matrix A is applied to each pixel point in the input license plate image to transform it to obtain the pixel point coordinates of the output license plate image;

[0037] Finally, through the grid generation module, the corresponding position relationship between the input license plate image and the output license plate image can be determined, and the value of each position is determined from which specific position in the input license plate image the value is mapped, thereby obtaining the corresponding relationship of all positions, and the mapping relationship between the input license plate image and the output license plate image is recorded as T M .

[0038] A license plate positive and tilt classification system based on a correction network (PTN), including a positive license plate image dataset and a tilted license plate image dataset, and then the images of these datasets are resized, normalized and data enhanced. During the training process, all parameters in the correction module are frozen.

[0039] A training strategy based on a correction network PTN includes the following steps:

[0040] Step 1: First, a separate classifier is trained to determine whether the license plate is correctly corrected. This classifier does not need to be trained jointly with the correction network PTN. After the classifier is trained, it can be used as an independent module to feed back its recognition results to the correction network PTN to help PTN improve the correction effect.

[0041] Step 2: After the initial training of the classifier described in step 1, its parameters are frozen to obtain a frozen classification network so that it is not updated in subsequent training;

[0042] Step 3: Randomly select the original license plate image, use the trained classification network to classify the license plate image data, and retain the license plate images classified as positive as new training samples A;

[0043] Step 4: Determine whether the image classified as tilted license plate is caused by image distortion, and retain the distorted license plate image as a new training sample B;

[0044] Step 5: Reorganize the extracted new training samples A and B according to a certain number to obtain a new data set for training the correction module;

[0045] Step 6: Finally, combine the correction module trained in step 5 with the classification network that has been frozen in step 2, and then fine-tune the entire network.

[0046] The present invention provides a weakly supervised license plate classification information driven license plate image automatic correction module and its training strategy, the technical effects are as follows:

[0047] 1) The present invention aims at the problem that the original STN spatial transformation network cannot effectively correct the license plate image after perspective transformation, and the fact that most of the current license plate recognition research only focuses on the single-line character problem and cannot directly recognize the license plate with two-line characters, and thus cannot provide effective supervision information for the correction network. A license plate correction method based on weakly supervised classification information with wide adaptability is proposed. The present invention first adopts the method of pasting, pastes the generated single-row and double-row license plates on the license plate area of ​​the real vehicle image of CCPD, and then removes the license plate to ensure that the license plate image has a real background, thus solving the problem of lack of double-row license plate data set; in addition, through the improved correction network PTN and the license plate view classification network, various license plates can be effectively corrected, and at the same time, it can also make a judgment on whether the subsequent recognition network needs to recognize. This scheme improves the interpretability and effectiveness of the results of each part of the entire automatic correction module, and does not affect the calculation speed.

[0048] 2) In order to further improve the convergence of the license plate correction algorithm module based on weakly supervised classification information with wide adaptability proposed in the present invention during training, the present invention proposes a training strategy. Through this training method, the input license plate image can be geometrically corrected adaptively based on the classification accuracy, and the learning of spatial transformation parameters centered on the classification accuracy is realized, thereby enhancing the accuracy of license plate correction.

[0049] 3) The present invention does not require complex annotations, and only requires two types of license plate images: license plate images at a frontal angle and license plate images at a non-frontal angle, to complete the training of the entire network. In order to solve the problem of the lack of double-line character license plate image datasets, the present invention spatially transforms the synthesized license plate images according to the actual scene of the real unconstrained environment, and maps them to the license plate area of ​​the real vehicle image, thereby constructing a double-line character license plate image dataset that is very close to the actual situation, providing good support for the effective training of the model.

[0050] 4) Aiming at the problem of correcting and classifying license plate images (including double lines) in open scenes, the present invention proposes a license plate correction module based on weakly supervised classification information with wide adaptability. Different from the widely used STN network, the correction network PTN obtained by improving the original STN network structure can better correct single and double-line license plates, and has good performance in perspective transformation correction. In addition, the present invention also adopts a classification network, which can provide feedback on the correction results of PTN.

[0051] 5) To further improve the convergence of the license plate image automatic correction module driven by weakly supervised classification information with wide adaptability during the training process. The present invention firstly pastes the generated single-row and double-row license plates on the original license plates of CCPD by pasting, and then removes the license plates, thereby solving the problem of lack of double-row license plate data sets. In terms of training strategy, the data sets required for the training correction module are extracted from the training image sets of positive and negative samples, and the learning of spatial transformation parameters centered on classification accuracy is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0053] Figure 1 Schematic diagram of recognition error caused by non-orthogonal viewing angle.

[0054] Figure 2 Schematic diagram of affine transformation and perspective transformation.

[0055] Figure 3 This is a structural diagram of the automatic license plate correction module based on weakly supervised classification information.

[0056] Figure 4 This is the subnetwork graph for the license plate vertex regression.

[0057] Figure 5 Maps the relationship diagram for the image.

[0058] Figure 6 This is the flow chart of the interpolation sampling module.

[0059] Figure 7 This is a classification network structure diagram.

[0060] Figure 8 This is the flow chart for classification network training.

[0061] Fig. 9 Flowchart for extracting the positive and negative training image sets used to train the auto-correction module.

[0062] Fig.10 This is a comparison chart of the license plate before and after correction;

[0063] Fig.10 In the figure, part a is the license plate image before correction, and part b is the correction effect image after improvement of this patent. DETAILED DESCRIPTION

[0064] A widely applicable license plate correction algorithm based on weak classification information is proposed. The core of the algorithm is the correction network PTN, which consists of four parts: the license plate vertex regression subnet Vertex-offset Regression Network (abbreviated as: Vertex-offset RN); the transformation matrix solution module; the grid generation module and the interpolation sampling module. The overall structure is as follows Figure 3 shown.

[0065] (I) Composition of the automatic license plate image correction algorithm:

[0066] 1. License plate vertex regression subnet:

[0067] The structure of the license plate vertex regression subnet is as follows Figure 4 As shown, where (X i ,Y i )(i=1,2,3,4) represents the coordinates of the four vertices of the license plate, and the entire subnet can regress 8 parameters.

[0068] The license plate vertex regression network obtains the input license plate image by license plate positioning. The present invention is not limited to license plate positioning or license plate detection algorithms, and is applicable to all license plate areas detected by license plate positioning algorithms. The input size set by the present invention is 36*94. If the output result of the license plate positioning algorithm used is different from this, a simple conversion can be performed, which will not be described in detail here. In the input license plate image area, the features of the license plate area are first extracted, and then the fully connected layer is used to extract high-level features of the license plate. The network structure includes 7 convolutional layers and 3 fully connected layers. The convolutional network is used to extract the position information of the license plate vertices, and the final coordinate regression value is obtained by integration through the last three fully connected layers.

[0069] 2. Transformation matrix solution module:

[0070] This module can calculate the perspective inverse transformation matrix based on the coordinates of the four vertices of the license plate (including double rows) and the image size of the corrected output through the perspective inverse transformation formula. The transformation matrix solution steps are as follows:

[0071] 1) First enter the source point coordinates M i (X mi ,Y mi )(i=1,2,3,4) and the target point coordinates N i (X ni ,Y ni )(i=1,2,3,4);

[0072] 2) Determine M i and N i Are the dimensions equal and greater than or equal to 4? If so, initialize the P matrix and Q matrix;

[0073] 3) Solve the linear equation system P*A=Q by mathematical methods;

[0074] 4) Post-processing the perspective inverse transformation matrix;

[0075] 5) Return the final perspective inverse transformation matrix A;

[0076] Among them, M i (X mi ,Y mi )(i=1,2,3,4) are the coordinates of the four vertices of the license plate in the input license plate image;

[0077] N i (X ni ,Y ni )(i=1,2,3,4) are the coordinates of the four corner points in the output license plate image;

[0078] A is the perspective inverse transformation matrix; P is the coordinate set of each point in the quadrilateral area surrounded by the coordinates of the four vertices of the license plate in the input license plate image; Q is the matrix composed of the coordinates of each corresponding pixel of the output license plate image.

[0079] 3. Mesh generation module:

[0080] By solving the above transformation matrix, a 3*3 perspective inverse transformation matrix A can be obtained. In order to produce a corrected image, there must be a module to receive the perspective inverse transformation matrix A output by the above modules and the pixel coordinates of the output license plate image, and use the above information to calculate the pixel coordinates of the input license plate image. Next, the perspective inverse transformation matrix A is applied to each pixel point in the input license plate image and transformed to obtain the pixel coordinates of the output license plate image; finally, the corresponding position relationship between the input license plate image and the output license plate image can be determined through this grid generation module, and it can be determined from which specific position in the input license plate image the value of each position is mapped, thereby obtaining the corresponding relationship of all positions, and the mapping relationship between the input license plate image and the output license plate image is recorded as T. M When the pixel coordinate point of the input license plate image is (X u1 , X u2 ), the pixel coordinates of the output license plate image are (X v1 , X v2 ), the mapping relationship between the two is as follows Figure 5 shown.

[0081] 4. Interpolation sampling module:

[0082] The working principle of the interpolation sampling module is to convert the mapping relationship T obtained by the grid generation module in the upper part MIt acts on the pixel points of the input license plate image, then extracts the pixel values ​​at the corresponding coordinates of the input license plate image and fills them into the corresponding points of the input license plate image to generate the converted image.

[0083] This process requires adjusting the position of each point in the input license plate image so that they can match the transformed image space. It uses an interpolation algorithm that calculates pixel values ​​based on position mapping. Because the position correspondence between the input license plate image and the input license plate image may be a decimal, the widely used bilinear interpolation algorithm is used to calculate the pixel value of the interpolation point, and then assign the value to the output image. The interpolation sampling module algorithm flow chart is shown below. Figure 6 As shown, in this way, the interpolation sampling module can effectively enhance the resolution of the image while maintaining the image details, and can maintain high computational efficiency and good locality, which can further improve the robustness of the model to geometric changes in the input data and help enhance the overall recognition or detection performance.

[0084] (II) License plate classification network algorithm module composition:

[0085] In order to build an effective license plate straight and oblique classification system, the present invention uses a custom convolutional neural network to perform binary classification of license plates (whether they are tilted or not). The main components and functions of this network are shown in Table 1:

[0086] Table 1 Classification of network modules and their functions

[0087]

[0088]

[0089] The network structure diagram is as follows Figure 7 shown.

[0090] (III) License plate classification network algorithm module composition:

[0091] In order to further improve the convergence of the license plate correction module based on weakly supervised classification information with wide adaptability during the training process, the present invention proposes a new training strategy. First, a classifier can be trained separately to determine whether the license plate is correctly corrected. This classifier does not need to be trained jointly with the correction network PTN. After the classifier training is completed, it can be used as an independent module to feed back its recognition results to the correction network PTN to help the correction network PTN improve the correction effect.

[0092] In the training of the classification network, the classification network is trained using the normal and tilted license plate training samples. In order to build an effective license plate normal and tilt classification system, it is first necessary to prepare an image dataset containing normal and tilted license plates, and then resize, normalize, and perform data augmentation on these images to enhance the robustness of the model. During the training process, all parameters in the correction network PTN need to be frozen. This ensures that the correction network PTN will not perform any form of correction on the license plate image. The license plate image is actually directly training the classification network. The specific flow chart is as follows: Figure 8 shown.

[0093] After the initial training of the classifier, its parameters are frozen so that it will not be updated in subsequent training, thereby ensuring that the learned feature extraction pattern will not change. On this basis, the training process of positive and negative samples is taken. The first step is to randomly select the original license plate images, use the trained classification network to classify these data, and retain the license plate images classified as "positive" as new training samples A. The second step is to determine whether the license plate images classified as "crooked" are caused by image distortion, and retain the distorted license plate images as new training samples B. The third step is to reorganize the extracted data samples A and B according to a certain number to obtain a new data set for training the correction model. Finally, the trained correction model is combined with the frozen classification network, and then the entire network is fine-tuned. The training process sets the learning rate very low, with the aim of improving the performance of the network while maintaining existing knowledge by fine-tuning the network parameters. The specific process is as follows Fig. 9 shown.

[0094] In order to verify the effectiveness of the license plate correction algorithm module based on weakly supervised classification information with wide adaptability proposed in the present invention, the present invention draws on the large dataset CCPD for license plate recognition and maps it on it to solve the problem of lack of double-row license plate dataset. By experimenting with the improved PTN network and the original STN network, different classification rates are obtained as shown in Table 2 below.

[0095] Table 2 Comparative experiment of CCPD map classification (unit: %)

[0096] Model Classification accuracy Original STN plus classification network 87.49％ The present invention's correction module plus classification network 98.52％

[0097] The experimental results show that compared with the original STN, the improved PTN has improved the classification accuracy of license plates by 11.03%. Fig.10As shown in the figure, it can be seen that the license plate correction effect is obvious, and the difficulty of license plate identification is significantly reduced. After the license plate correction, the classification effect is significantly improved. Such correction can greatly reduce the error rate and ensure that the license plate recognition system can correctly identify the license plate information in various scenarios.

[0098] The present invention proposes a license plate image automatic correction module driven by weakly supervised license plate classification information, which has the following characteristics:

[0099] (1) Aiming at the problem of correcting license plate images in open scenes, the present invention proposes a widely adaptable automatic license plate image correction module based on weakly supervised classification information. This module is applicable to both single-row and multi-row license plates, and does not require a license plate positioning algorithm to give the vertex position of the license plate, nor does it require additional labeled data. The training of the proposed perspective transformation correction network PTN can be achieved through the weak supervision signal provided by a license plate view classification network. In addition, the license plate view classification network used in the present invention can also judge the license plate image that is subsequently input into the recognition network. For images that are not corrected or are not license plate images, the classification network can be used to implement filtering or feedback. The entire algorithm solution not only solves the problem that the original STN cannot effectively correct the license plate due to perspective transformation, but also solves the problem of guiding the correction network training due to the lack of license plate recognition supervision information provided by the recognition network that can directly recognize double-row character license plates. In order to solve the problem of lack of double-row license plate data set, the present invention also designs a method of combining license plate generation with mapping technology, pasting the generated single-row and double-row license plates on the license plate area of ​​the real vehicle image of CCPD, and then taking off the license plates to ensure that the license plates have a real scene background, effectively solving the problem of lack of double-row license plate data set.

[0100] (2) In order to ensure the effective training of each module in the license plate correction algorithm based on weakly supervised classification information with wide adaptability proposed by the present invention, the present invention also proposes a training strategy. First, the classification network is initially trained through all training samples. At this time, the initial parameters of the correction module are frozen. The correction module will not perform any form of transformation or correction on the license plate image. In fact, all training samples first enter the classification network for training. Then, after the classification network is initially trained, its parameters are frozen. The present invention also adopts a positive and negative sample training image set extraction method, which uses the trained classification network for screening, wherein the data set A classified as "positive" and the data set B classified as "distorted" due to spatial distortion are respectively taken from the data sets A and B. The same amount of license plate images are remixed to obtain a new data set. Finally, the correction module is trained using the newly mixed data set to better adapt to the input format required by the classification network. This training strategy realizes the learning of spatial transformation parameters centered on classification accuracy, thereby enhancing the accuracy of license plate correction.

[0101] (3) In addition, the correction module and classification network proposed in the present invention can be trained synchronously, and the correction module can be adapted to any recognition network. Therefore, the license plate recognition network that the correction module proposed in the present invention can connect to is not limited to one, but is widely applicable to a variety of recognition networks and can be flexibly adjusted according to the usage scenario.

Claims

1. The license plate image automatic correction module driven by weakly supervised license plate classification information is characterized by include: License plate vertex regression subnet, transformation matrix solution module, grid generation module, interpolation sampling module; The license plate vertex regression subnet includes multiple convolutional layers and fully connected layers, which use the convolutional network to extract the position information of the license plate vertices and integrate them through the fully connected layer to obtain the final coordinate regression value; The transformation matrix solving module calculates the perspective inverse transformation matrix through perspective inverse transformation according to the coordinates of the license plate vertices and the corrected output license plate image size; The grid generation module determines the corresponding position relationship between the input license plate image and the output license plate image, and determines the mapping of the position relationship to obtain the corresponding relationship of all positions; The interpolation sampling module converts the mapping relationship T between the input license plate image and the output license plate image obtained by the grid generation module M It acts on the pixel points of the input license plate image, then extracts the pixel values ​​at the corresponding coordinates of the input license plate image and fills them into the corresponding points of the output license plate image to generate the converted image.

2. The license plate image automatic correction module driven by weakly supervised license plate classification information according to claim 1 is characterized by: In the input license plate image area, the license plate vertex regression subnet first extracts features from the input license plate image area, and then uses the fully connected layer to perform high-level feature extraction on the input license plate image; the network structure of the license plate vertex regression subnet includes 7 convolutional layers and 3 fully connected layers.

3. The license plate image automatic correction module driven by weakly supervised license plate classification information according to claim 1 is characterized by: The transformation matrix solving module comprises the following steps: (1): First input the source point coordinates M i (X mi ,Y mi ), and the target point coordinates N i (X ni ,Y ni ), i=1,2,3,4; (2): Determine M i and N i Are the dimensions of equal and greater than or equal to 4? If so, initialize the P matrix and the Q matrix; (3): Solve the linear equation system P*A=Q by mathematical methods; (4): Post-process the perspective inverse transformation matrix; (5): Return the final perspective inverse transformation matrix A; Among them, M i (X mi ,Y mi ) are the coordinates of the four vertices of the license plate in the input license plate image; N i (X ni ,Y ni ) are the coordinates of the four corner points in the output license plate image; A is the perspective inverse transformation matrix; P is the coordinate set of each point in the quadrilateral area surrounded by the coordinates of the four vertices of the license plate in the input license plate image; Q is the matrix composed of the coordinates of each corresponding pixel of the output license plate image; By solving the transformation matrix, we can obtain a 3*3 perspective inverse transformation matrix A.

4. The license plate image automatic correction module driven by weakly supervised license plate classification information according to claim 1 is characterized by: In the grid generation module, First, receive the output perspective inverse transformation matrix A and the pixel coordinates of the output license plate image, and use the above information to calculate the pixel coordinates of the input license plate image; Next, the perspective inverse transformation matrix A is applied to each pixel point in the input license plate image to transform it to obtain the pixel point coordinates of the output license plate image; Finally, through the grid generation module, the corresponding position relationship between the input license plate image and the output license plate image can be determined, and the value of each position is determined from which specific position in the input license plate image the value is mapped, thereby obtaining the corresponding relationship of all positions, and the mapping relationship between the input license plate image and the output license plate image is recorded as T M .

5. A license plate straight and tilt classification system based on the license plate image automatic correction module of any one of claims 1 to 4, characterized in that: It contains a dataset of positive license plate images and a dataset of tilted license plate images. These dataset images are then resized, normalized, and data augmented. During the training process, all parameters in the license plate image automatic correction module are frozen.

6. A training strategy for the automatic license plate image correction module according to any one of claims 1 to 4, characterized in that The following steps are involved: Step 1: First, a classifier is trained separately to determine whether the license plate is correctly corrected. After the classifier is trained, it is used as an independent module to feed back its recognition results to the correction network PTN; Step 2: After the initial training of the classifier described in step 1, its parameters are frozen to obtain a frozen classification network so that it is not updated in subsequent training; Step 3: Randomly select the original license plate image, use the trained classification network to classify the license plate image data, and retain the license plate images classified as positive as new training samples A; Step 4: Determine whether the image classified as tilted license plate is caused by image distortion, and retain the distorted license plate image as a new training sample B; Step 5: Reorganize the extracted new training samples A and B according to a certain number to obtain a new data set for training the correction module; Step 6: Finally, combine the correction module trained in step 5 with the classification network that has been frozen in step 2, and then fine-tune the entire network.

Images