Double-row license plate image automatic correction module based on weak supervision identification information driving and training strategy of double-row license plate image automatic correction module

Abstract

The invention discloses a double-row license plate image automatic correction module driven based on weak supervision identification information and a training strategy thereof. A license plate coordinate point regression subnet is used for extracting license plate feature information; the transformation matrix solving module is used for calculating a perspective transformation inverse matrix through perspective inverse transformation according to the coordinate points obtained by the license plate coordinate point regression subnet and the corrected and output image size; and the grid generation module finds the corresponding relation between the input license plate image and the target license plate image position and determines the mapping relation between the input license plate image and the target license plate image position. And the interpolation sampling module processes the pixel points of the input license plate image, and performs pixel filling on the corresponding points of the output license plate image according to the pixel values of the corresponding coordinate points of the input license plate image to obtain a transformed image. According to the invention, a double-row license plate identification problem can be automatically converted into a single-row license plate identification problem, extra license plate character region labeling information is not needed, and only weak supervision information fed back by a license plate identification network is needed; therefore, the accuracy of license plate recognition is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of license plate recognition, and in particular to an automatic correction module for double-row license plate images driven by weakly supervised recognition information and a training strategy thereof. Background Art

[0002] Non-binding license plate recognition technology does not require any constraints or restrictions on moving vehicles and has a very wide range of application scenarios, such as traffic management, security monitoring, vehicle management, toll collection systems, etc. However, due to factors such as shooting angle and lighting conditions, various distortions and excessive inclination angles often appear in the license plate image, which greatly limits the potential of non-binding license plate recognition technology.

[0003] Most of the current license plate recognition research is aimed at single-line license plates. On the one hand, the widely used large-scale license plate image datasets only contain single-line license plate images. Even if they contain double-line license plates, their number accounts for a very small proportion of the entire dataset. For example, CCPDv1 only contains blue single-line license plates; CPDv2 only contains green single-line license plates; and the proportion of double-line license plate images in CBLPRD-330K is only 15.8%. On the other hand, license plate recognition algorithms mainly use OCR (optical character recognition) models with CRNN (a combination of CNN and RNN) architectures, such as PaddleOCR, Trocr, Tesseractocr, etc., or pure CNN architecture models, such as LPRNet and its variants, and are combined with CTC loss. These algorithms only support the recognition of single-line characters.

[0004] With the continuous development of license plate recognition technology, some research and exploration work on double-row license plate recognition has also emerged in recent years. Literature [1]: Chen Zhenyu. Research on double-row license plate recognition technology [D]. Northwest Normal University, 2021. A license plate segmentation method based on adaptive threshold is proposed. First, the license plate image is converted into HSV color space, the image is binarized and the license plate characters are segmented using the adaptive projection method, and finally the segmented license plate characters are recognized. Although this method can handle double-row character license plates, the problem of character loss will occur after the image is binarized. The character segmentation accuracy is not high, and subsequent recognition will also be greatly affected.

[0005] Reference [2]: Liu Junru. Double-row license plate segmentation and recognition algorithm based on convolutional neural network [D]. Wenzhou University, 2019. A double-row license plate segmentation algorithm based on convolutional neural network (CNN) is proposed. This method can automatically segment a double-row license plate into two single-row license plates, and then use the CRNN network structure to recognize the characters in the two single-row license plates. This method can only achieve good results for double-row license plates with good positioning. On the other hand, since the number and types of characters in the upper and lower areas of double-row license plates are also very different, it is necessary to train two different networks to recognize the upper and lower areas of double-row license plates respectively, which not only increases the computational burden, but also fails to utilize some existing excellent single-row license plate recognition algorithms such as LPRNet. In order to better solve the problem of license plate rotation and tilt, the STN spatial transformation network that can achieve spatial correction has received widespread attention in license plate recognition. STN is a network structure that can improve the spatial invariance of convolutional neural networks. It can not only adjust the spatial structure of input data during training, but also be jointly trained with the entire deep learning model end-to-end, which improves the model's attention to input data and enables the geometric transformation and feature extraction parts of the model to be learned together.

[0006] Reference [3]: Li Shiwei. Object detection and license plate recognition system based on YOLOv5 algorithm [J]. Electronic Technology and Software Engineering, 2022, (01): 138-141. A new license plate recognition algorithm RD-LPRNET is proposed. The algorithm constructs a smaller image copy through interpolation operation, and uses the created copy instead of the original image input to predict the affine transformation parameters. The affine transformation parameters are used to obtain the offset parameters and apply them to the original image to achieve license plate correction.

[0007] Reference [4]: ​​Fang Haocheng. Research on key technologies of vehicle recognition system based on deep learning [D]. Huazhong University of Science and Technology, 2021. and reference [5]: Xu Wangming, Yuan Shixin, He Qin. Lightweight foggy license plate detection and recognition algorithm based on image adaptive enhancement [J]. Journal of Wuhan University of Science and Technology, 2024, 47(02): 144-153. Both are spatial correction networks designed based on the original STN. When regressing the spatial transformation matrix, this STN only considers the affine transformation matrix, that is, only simple transformations on the plane such as cropping, rotation, and scaling are realized.

[0008] However, in addition to affine transformations such as rotation and scaling in the plane, license plates taken in open environments also undergo perspective transformations in space. This transformation will change the position of the license plate vertices in the image, and the parallel relationship between the license plate boundary lines will be broken. Standard affine transformations cannot effectively solve such problems. Figure 1As shown in the figure, part a is the original license plate image, and part b is the license plate image after traditional STN correction. The effect before and after correction using standard affine transformation is not obvious. Figure 1 The top row is the data set used in this patent, and the bottom row is the double-row license plate after the original STN correction. Although in theory the STN network can be extended to estimate the parameters of a variety of complex spatial transformation matrices and correct the corresponding highlighted areas. However, in practical applications, complex spatial transformations such as perspective transformations, due to the mutual influence between their 8 key parameters, can easily lead to the failure of STN training convergence with license plate recognition results as weak supervision signals. Therefore, the STN networks that are widely used now are all corrections to affine transformations. For example, the STN-specific python module provided by the official website of Pytorch is only for affine transformations. From a mathematical point of view, affine transformation is a special case of perspective transformation, so the perspective transformation correction network is also applicable to the correction of affine transformation. Summary of the invention

[0009] Aiming at the recognition problem of double-row character license plates in open scenes, the present invention proposes a double-row license plate image automatic correction module and its training strategy driven by weakly supervised recognition information. The automatic correction module designs a PTN network that can effectively solve the perspective transformation correction. The PTN network can automatically and synchronously find the two sets of vertex coordinates corresponding to the upper and lower character areas of the double-row license plate, and use these coordinates to convert the inverse matrix of the perspective transformation, thereby transforming the two areas into rectangular areas of the same size and splicing them into a single-row character license plate image. The spliced ​​image can be correctly identified by the single-row license plate recognition network, cleverly transforming the double-row license plate recognition problem into a single-row license plate recognition problem.

[0010] The technical solution adopted by the present invention is:

[0011] The automatic correction module for double-row license plate images driven by weakly supervised recognition information includes:

[0012] License plate coordinate point regression subnet, transformation matrix solution module, grid generation module, interpolation sampling module;

[0013] The license plate coordinate point regression subnet includes seven convolutional layers and three fully connected layers. The convolutional layers are used to extract the feature information of the six points of the license plate coordinates. The three fully connected layers regress the coordinates of the six points of the license plate by integrating the feature information.

[0014] The transformation matrix solving module calculates the perspective transformation inverse matrix through perspective inverse transformation according to the six coordinate points obtained by the license plate coordinate point regression subnet and the corrected output image size;

[0015] The grid generation module takes the two 3*3 perspective transformation inverse matrices M1 and M2 obtained by the transformation matrix solving module and the pixel coordinates of the target license plate image as inputs, obtains the pixel coordinates of the input license plate image by calculation, then uses the perspective transformation inverse matrix M1 to process each pixel point of the upper half of the input license plate image to obtain the pixel coordinates of the upper half of the target license plate image, and then uses the perspective transformation inverse matrix M2 to process each pixel point of the lower half of the input license plate image to obtain the pixel coordinates of the lower half of the target license plate image; finally, the corresponding relationship between the positions of the input license plate image and the target license plate image is found, and the mapping relationship between the positions of the input license plate image and the target license plate image is determined.

[0016] The interpolation sampling module uses the mapping relationship T obtained by the grid generation module M1 and T M2 The pixel points of the input license plate image are processed, and the corresponding points of the output license plate image are filled with pixels according to the pixel values ​​of the corresponding coordinate points of the input license plate image to obtain the transformed image.

[0017] By the mapping relation T M1 The output license plate image is the rectified image of the first line of characters in the double-line license plate. M2 The output license plate image is the corrected image of the second line of characters in the double-line license plate. The target license plate image is obtained by splicing the two corrected images from left to right.

[0018] In the transformation matrix solving module, 4 coordinate points are needed to calculate an inverse perspective transformation matrix, and two of the 6 coordinate points are shared. The transformation matrix solving module divides the double-row license plate into two upper and lower areas and calculates two inverse perspective transformation matrices.

[0019] in:

[0020] U i (X ui ,Y ui )(i=1,2,3,4) are the top four points of the double-row license plate in the input license plate image;

[0021] V i (X vi ,Y vi )(i=1,2,3,4) are the following four points of the double-row license plate in the input license plate image;

[0022] D ui (X di ,Y di ) are the coordinates of the four corner points of the output image of the upper half of the double-row license plate;

[0023] D vi (X di,Y di ) are the coordinates of the four corner points of the output image of the lower half of the double-row license plate;

[0024] M1 and M2 are the inverse perspective transformation matrices;

[0025] A1 is the coordinate set of all points in the quadrilateral area surrounded by the four points on the license plate in the input license plate image;

[0026] A2 is the coordinate set of all points in the quadrilateral area surrounded by the four points below the license plate in the input license plate image;

[0027] B1 and B2 are matrices consisting of the coordinates of the corresponding pixels of the output license plate image.

[0028] Figure 6 Solve the flow chart for the transformation matrix, such as Figure 6 The following steps are shown:

[0029] STEP 1: Determine U i and D ui Are the dimensions of V equal and greater than or equal to 4? i and D vi Are the dimensions equal and greater than or equal to 4?

[0030] STEP2: Initialize matrices A1 and B1; A2 and B2;

[0031] STEP3: Solve the linear equations A1*M1=B1; A2*M2=B2, and use the perspective inverse transformation formula to solve M1 and M2;

[0032] STEP4: Post-process the inverse matrix of perspective transformation;

[0033] STEP5: Return the inverse perspective transformation matrices M1 and M2;

[0034] There are 6 sets of coordinate points, and the middle two sets of coordinate points are shared by the upper and lower lines of characters in a double-line license plate. Taking the upper line of a double-line character license plate as an example, the upper line of a double-line character license plate has 4 sets of coordinate points: U i (X ui ,Y ui )(i=1,2,3,4), which is recorded as (X ui ,Y ui )(i=1,2,3,4); Output the coordinates of the four corner points of the image D ui (X di ,Y di )(i=1,2,3,4), which is recorded as (X dui ,Y dui )(i=1,2,3,4). (X ui ,Yui ) corresponds to (X dui ,Y dui )(i=1,2,3,4), and thus calculate the linear equation system, that is, equation (1). In the calculation process, the first matrix on the left side of the equal sign is recorded as A1, the second matrix on the left side of the equal sign is recorded as M1, and the matrix on the right side of the equal sign is recorded as B1. The linear equation system solution method M1=A1 -1 B1 solves 8 parameters, and the solved 8 parameters (m 11 ,m 21 ,m 31 ,m 12 ,m 22 ,m 32 ,m 13 ,m 23 ) are the first 8 parameters of the perspective transformation inverse matrix M1. The 9th parameter of the perspective transformation inverse matrix is ​​the scale factor, which is usually set to 1. Convert these nine parameters into a 3*3 matrix to obtain the perspective transformation inverse matrix M1. Similarly, follow the same steps to calculate the perspective transformation inverse matrix M2.

[0035]

[0036] In the interpolation sampling module, I is the input license plate image; M1 and M2 are the inverse affine transformation matrices corresponding to the upper and lower halves of the double-row license plate, respectively; H and W are the height and width of the input license plate image I; G1 and G2 are coordinate grids ranging from [0, H) × [0, W); G1′ and G2′ are transformed grids obtained by applying the inverse perspective transformation matrices M1 and M2; I1 and I2 are transformed images; I′ is the image after splicing I1 and I2, that is, splicing the upper and lower parts of the double-row license plate left and right; Figure 8 Solve the process for the interpolation sampling module, such as Figure 8 The following steps are shown:

[0037] Step 1: Initialize the empty output images I1 and I2 with a size of 47*24. The recognition model used in the present invention is lprnet, and its input image size is 94*24. Therefore, the size of the two corrected images is fixed to 47*24, and the size after left and right splicing is 94*24;

[0038] Step 2: Get the height H and width W of the input image I

[0039] Step 3: Construct coordinate grids G1 and G2

[0040] Step 4: Apply the inverse perspective transformation matrices M1 and M2 to the meshes G1 and G2 to obtain the transformed meshes G1′ and G2′

[0041] Step 5: For each point in the transformed grids G1′ and G2′:

[0042] Calculate its position in the input image I

[0043] Use bilinear interpolation to get the pixel value at that location

[0044] Put the pixel values ​​into the corresponding positions of I1 and I2

[0045] Step 6: Return the transformed images I1 and I2, and splice I1 and I2 left and right to get the target image I′

[0046] The present invention adopts the widely used bilinear interpolation method to determine the pixel value of the target image. The bilinear interpolation method calculates the pixel value of a point of the target image by using the pixel values ​​of four points around the original image, as shown in equations (2) to (4).

[0047]

[0048] Among them, Q 11 (x1,y1),Q 12 (x1,y2),Q 21 (x2,y1),Q 22 (x2, y2) are four points of the original image (with known coordinates and pixel values). These four points are the four points closest to the point on the target image mapped back to the original image. P(x, y) is a point on the target image mapped back to the original image (only the coordinates of this point are known). Linear interpolation is first performed in the direction of the X axis using formulas (2) to (3), and then linear interpolation is performed in the direction of the Y axis using formula (4) to achieve bilinear interpolation and calculate the pixel value of this point. The pixel values ​​of the remaining points of the target image are also calculated using the formula.

[0049] Training strategy,In the first stage of training, all training samples are used to train the license plate recognition network. In this process, the last fully connected layer of the license plate coordinate point regression subnet in the automatic correction module is forced to output six points A, B, C, D, E, and F. The license plate image is directly fed into the license plate recognition network for training;

[0050] In the second stage of training, the parameters of the license plate recognition network trained in the first stage are frozen, and the parameters of the license plate recognition network are no longer updated in the second stage of training. This method can keep the feature extraction method learned by the recognition network unchanged. The present invention adopts a positive and negative sample training image set extraction process to obtain a data set for training an automatic correction module.

[0051] First, randomly select the original license plate image and put it into the license plate recognition network trained in the first stage of training to determine whether it is recognized correctly. The correctly recognized license plate image is retained as the new training data A;

[0052] Secondly, for the wrongly recognized license plate image, it is determined again whether the recognition error is caused by image distortion, and the distorted license plate image is retained as new training data B;

[0053] Finally, the new training data A and B are combined in the same amount to form a new data set, which is used to train the automatic correction module;

[0054] The automatic correction module performs spatial transformation on the distorted license plate images in the input data set to obtain an input format that is more suitable for the license plate recognition network. Afterwards, the trained automatic correction module is combined with the frozen license plate recognition network, and the entire network is fine-tuned.

[0055] The present invention provides a two-row license plate image automatic correction module and its training strategy driven by weakly supervised recognition information, and the technical effects are as follows:

[0056] 1) Different from the existing research on double-line character license plates, the license plate image automatic correction module proposed in this invention uses the coordinates of 6 points as parameters, replacing the parameters of the original STN network, and automatically segments and splices the double-line license plates using the input 6-point parameters. On this basis, the inverse matrix solution module of the perspective transformation is added to calculate the matrix parameters, which solves the problem that the original STN network cannot effectively process double-line character license plates, and also improves the interpretability of the results of each part of the correction algorithm module.

[0057] 2) To address the problem of insufficient double-row license plate data set, the present invention uses a method of mixing a synthetic double-row license plate data set with a real double-row license plate data set to build a data set. Figure 2 As shown in the figure. Since the CCDP dataset contains multiple subsets, the license plate images have multiple scenes and multiple angles, so the present invention uses the CCDP dataset as the background of the synthetic dataset. According to the accurate coordinates of the four vertices of the license plate in the CCDP dataset label, the synthesized double-row license plate image is mapped onto the CCPD, as shown in the figure. Figure 3-1 As shown, the double-line character license plate image containing the background is cut out, such as Figure 3-2 As shown in , the synthetic two-row license plate with background and the real two-row license plate are finally mixed together to form a self-built data set, as shown in Figure 3-3 As shown, the two on the left are synthetic data sets, and the two on the right are real data sets.

[0058] 3) In view of the problem that the original STN network cannot effectively correct the perspective transformation, the present invention proposes a double-row license plate image automatic correction module driven by weakly supervised recognition information, and the license plate image automatic correction module is based on the PTN network. The PTN network consists of four parts: a license plate coordinate point regression subnet, a transformation matrix solving module, a grid generation module, and an interpolation sampling module. The PTN network can automatically and synchronously find the two sets of vertex coordinates corresponding to the upper and lower areas of the double-row character license plate, and use these coordinates to convert the inverse matrix of the perspective transformation, thereby transforming the upper and lower rows of the double-row character license plate into rectangular areas of the same size, and finally splicing the two rectangular areas left and right into a single-row license plate image. The spliced ​​image can be correctly identified by the single-row license plate recognition network, cleverly transforming the double-row license plate recognition problem into a single-row license plate recognition problem.

[0059] 4) Due to the lack of existing double-row character license plate data sets, the training of deep learning models is severely restricted. Therefore, the present invention uses the CCPD data set as the background of the synthetic data set, and maps the synthetic double-row character license plate image to the CCPD according to the accurate coordinates of the four vertices of the license plate in the precise positioning label of the CCPD data set. The double-row character license plate image containing the background is then cut out according to the annotation information of the CCPD data set, and finally the synthetic double-row license plate containing the background is mixed with the real double-row license plate to form a self-built data set. In order to further improve the convergence of the proposed double-row license plate character area automatic segmentation and splicing algorithm module driven by weakly supervised recognition information in training, the present invention proposes a new training strategy. Using this training strategy, the input license plate image can be adaptively geometrically corrected according to the recognition accuracy, and the spatial transformation parameters can be learned with the recognition accuracy as the center, thereby enhancing the accuracy of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0061] Figure 1 This is the correction effect diagram of the traditional STN network.

[0062] Figure 2 To synthesize a double-row license plate image.

[0063] Figure 3-1 This is the texture effect diagram;

[0064] Figure 3-2 This is the cutout effect picture.

[0065] Figure 3-3 This is a self-built dataset diagram.

[0066] Figure 4 This is the structural diagram of the double-row license plate automatic segmentation and splicing algorithm module.

[0067] Figure 5-1 This is a schematic diagram of the license plate coordinate point regression network;

[0068] Figure 5-2 Schematic diagram of the position of regression coordinate points.

[0069] Figure 6 Solving the transformation matrix flowchart

[0070] Figure 7 Generate schematic for mesh

[0071] Figure 8 Interpolation sampling module algorithm flow chart

[0072] Fig. 9 Training policy graph for the auto-correction module

[0073] Fig.10 Initialize the six-point coordinate map for the correction algorithm module

[0074] Fig.11 Flowchart for extracting positive and negative sample training image sets for training the automatic correction module

[0075] Fig.12 This is a comparison of the license plate before and after correction.

[0076] Fig.12 In the figure, part a is the original license plate image, and part b is the license plate image corrected by PTN.

[0077] Fig.13 To identify the effect diagram. DETAILED DESCRIPTION

[0078] In view of the problem that the original STN network cannot effectively correct the perspective transformation, the present invention proposes a PTN (Perspective Transformation Network), which can automatically find the coordinates of the six points corresponding to the upper and lower areas of the double-line character license plate, and use these coordinates to convert the inverse matrix of the perspective transformation, divide the upper and lower lines of the double-line character license plate into areas of the same size and splice them left and right into a single-line character license plate image; in view of the problem of insufficient double-line character license plate data sets. The present invention mixes the synthetic data set with the real data set to form a self-built data set. Since the CCDP data set contains multiple subsets, the license plate images have multiple scenes and multiple angles, the present invention uses the CCPD data set as the background to map the synthetic double-line license plate, and then uses the annotation information of CCPD to obtain the synthetic data set. The real data set comes from the double-line license plate image in CBLPRD-330K.

[0079] The present invention proposes an automatic segmentation and splicing algorithm for double-row license plate character regions driven by weakly supervised recognition information. The core of the algorithm is the perspective transformation correction network PTN. The PTN network consists of four parts: a license plate coordinate point regression subnet, a transformation matrix solution module, a grid generation module, and an interpolation sampling module. The overall composition of the algorithm is as follows: Figure 4 As shown:

[0080] (I) Composition of the automatic correction algorithm for double-row license plates:

[0081] 1. License plate coordinate point regression subnet:

[0082] The license plate coordinate point regression subnet structure is as follows Figure 5-1 As shown, (Xi, Yi) (i = 1, 2, 3, 4, 5, 6) represents the coordinates of the four vertices of the license plate. The entire subnet can regress 12 parameters, representing the specific coordinates of the 6 points on the license plate.

[0083] The input license plate image of the license plate coordinate point regression subnet comes from the license plate positioning. The present invention has no restrictions on the license plate detection and positioning algorithm, and is applicable to license plates detected by any license plate positioning algorithm. The input license plate area size used in the present invention is 36*94. If the license plate area size obtained by the license plate positioning algorithm is different, it needs to be processed by a simple size conversion, which will not be repeated here. Figure 5-1 It can be seen that the input license plate image first passes through the feature extraction layer to obtain the corresponding license plate features, and then extracts high-level features through the fully connected layer. The license plate coordinate point regression subnet consists of seven convolutional layers and three fully connected layers. The convolutional layer is used to extract the key features related to the coordinates of the six points of the license plate. The three fully connected layers regress the coordinates of the six points of the license plate by integrating the feature information. Figure 5-2 This is a schematic diagram of the positions of the six points on the license plate obtained by the license plate coordinate point regression network.

[0084] 2. Transformation matrix solution module:

[0085] This module calculates the perspective transformation inverse matrix based on the 6 coordinate points obtained by the coordinate point regression subnet and the corrected output image size through the perspective inverse transformation formula. Calculating an inverse perspective transformation matrix requires 4 coordinate points, and two of the 6 coordinate points are shared, such as Figure 5-2 This module divides the double-row license plate into two upper and lower regions and calculates two perspective transformation inverse matrices.

[0086] Transformation matrix solving module such as Figure 6 As shown, where:

[0087] U i (X ui ,Y ui)(i=1,2,3,4) are the top four points of the double-row license plate in the input license plate image;

[0088] V i (X vi ,Y vi )(i=1,2,3,4) are the following four points of the double-row license plate in the input license plate image;

[0089] D ui (X di ,Y di ) are the coordinates of the four corner points of the output image of the upper half of the double-row license plate;

[0090] D vi (X di ,Y di ) are the coordinates of the four corner points of the output image of the lower half of the double-row license plate;

[0091] M1 and M2 are the inverse perspective transformation matrices;

[0092] A1 is the coordinate set of all points in the quadrilateral area surrounded by the four points on the license plate in the input license plate image;

[0093] A2 is the coordinate set of all points in the quadrilateral area surrounded by the four points below the license plate in the input license plate image;

[0094] B1 and B2 are matrices consisting of the coordinates of the corresponding pixels of the output license plate image.

[0095] 3. Mesh generation module:

[0096] This module uses the two 3*3 perspective transformation inverse matrices M1 and M2 obtained by the above transformation matrix solving module and the pixel coordinates of the target license plate image as input, obtains the pixel coordinates of the input license plate image by calculation, and then uses the perspective transformation inverse matrix M1 to process each pixel point of the upper half of the input license plate image to obtain the pixel coordinates of the upper half of the target image, and then uses the perspective transformation inverse matrix M2 to process each pixel point of the lower half of the input license plate image to obtain the pixel coordinates of the lower half of the target image. Finally, the corresponding relationship between the input license plate image and the target license plate image position is found, and it is determined to which position of the target license plate image a certain position of the input license plate image is mapped.

[0097] The mapping relationship between the input license plate image and the output license plate image is denoted as T M1 and T M2 , when the pixel point of the upper half of the input license plate image is (X s1 ,Y s1 ), the pixel coordinates of the target image are (X t1 ,Y t1); When the pixel point in the lower half of the input license plate image is (X s2 ,Y s2 ), the pixel coordinates of the target image are (X t2 ,Y t2 ), the mapping relationship is as follows Figure 7 shown.

[0098] 4. Interpolation sampling module:

[0099] This module uses the mapping relationship T obtained by the grid generation module. M1 and T M2 Process the pixels of the input license plate image. According to the pixel values ​​of the corresponding coordinate points of the input license plate image, fill the corresponding points of the output license plate image with pixels to obtain the transformed image.

[0100] In order to ensure that the input license plate image and the output license plate image space match, this process requires adjusting the position of each point of the input license plate image. The pixel value of the interpolation point is calculated using a bilinear interpolation algorithm and assigned to the output license plate image. The bilinear interpolation algorithm can avoid the problem that the position correspondence between the output image and the input image may be a decimal. Since the bilinear interpolation algorithm is a widely used algorithm, the present invention will not be repeated here. The interpolation sampling module can not only retain image details and improve image resolution, but also maintain high computational efficiency and good locality, thereby enhancing the robustness of the model to geometric transformations in the input data. The interpolation sampling module algorithm flow chart is shown below. Figure 8 As shown, where: I is the input license plate image; M1 and M2 are the inverse affine transformation matrices corresponding to the upper and lower halves of the double-row license plate respectively; H and W are the height and width of the input license plate image I; G1 and G2 are coordinate grids ranging from [0,H)×[0,W); G1′ and G2′ are the transformed grids obtained by applying the inverse perspective transformation matrices M1 and M2; I1 and I2 are the transformed images; I′ is the image after splicing I1 and I2, that is, splicing the upper and lower parts of the double-row license plate left and right.

[0101] (II) Training of the automatic license plate image correction algorithm module:

[0102] In order to further improve the convergence of the automatic segmentation and splicing algorithm of double-row license plate character area driven by weakly supervised recognition information during the training process, the present invention proposes a method such as Fig. 9 The training strategy shown.

[0103] In the first stage of training, all training samples are used to train the license plate recognition network. The present invention uses the widely used LPRNet recognition network. In this process, the last fully connected layer of the license plate coordinate point regression subnet in the automatic correction module is forced to output six points A, B, C, D, E, and F, such as Fig.10As shown, in order to ensure that the automatic correction module does not correct the license plate image, all parameters in the automatic correction module need to be frozen. The license plate image actually directly enters the license plate recognition network for training.

[0104] In the second stage of training, the parameters of the license plate recognition network that has undergone preliminary training are frozen, and the parameters of the recognition network are no longer updated in the first stage of training. This method can keep the feature extraction method that the license plate recognition network has learned unchanged.

[0105] The present invention adopts the positive and negative sample training image set extraction process to obtain the data set for training the license plate image automatic correction module, such as Fig.11 As shown. Specifically: First, randomly select the original license plate image and put it into the license plate recognition network trained in the first stage of training to determine whether the recognition is correct, and retain the correctly recognized license plate image as the new training data A. Secondly, for the wrongly recognized license plate image, determine again whether the recognition error is caused by image distortion, and retain the distorted license plate image as the new training data B. Finally, the new training data A and B are combined in the same number to form a new data set, which is used to train the license plate image automatic correction module. The license plate image automatic correction module performs spatial transformation on the distorted license plate images in the input data set to obtain an input format that is more suitable for the recognition network. After that, the trained correction module is combined with the frozen license plate recognition network, and then the entire network is fine-tuned. The learning rate in this process is set very small, with the aim of carefully adjusting the network parameters to improve performance while retaining the existing knowledge.

[0106] The present invention proposes a two-row license plate image automatic correction module based on weakly supervised recognition information, which effectively solves the recognition problem of two-row license plates. The recognition network used in the present invention is the Lprnet recognition network, and the final recognition accuracy is 99.2%. The comparison of the effects before and after the license plate correction is shown in the figure below. Fig.12 As shown, Fig.12 Part a in the figure is the original license plate image, and part b is the license plate image after PTN correction. Fig.12 It can be seen that the double-row license plate with spatial distortion is effectively converted into a single-row license plate after PTN correction, and the double-row character license plate problem is cleverly converted into the single-row character license plate. The recognition effect is shown in the figure Fig.13 As shown, after PTN correction, the recognition network can well recognize double-row license plates. In addition, the correction module proposed in the present invention can be adapted to any recognition network and can be flexibly adjusted according to the usage scenario.

[0107] This invention is a two-row license plate image automatic correction module driven by weak supervision recognition information, which has the following characteristics:

[0108] 1. Aiming at the problem of correction and recognition of double-row license plates, the present invention proposes a double-row license plate character area automatic segmentation and splicing module driven by weakly supervised recognition information, the core of which is the perspective transformation correction network PTN. In PTN, firstly, a license plate character area coordinate point regression subnet is introduced to facilitate faster convergence of the spatial correction network, which is used to predict the positions of the six coordinate points of the double-row license plate. The six point coordinates are used to divide the license plate into two upper and lower areas, corresponding to the upper and lower character areas of the double-row character license plate. Next, a transformation matrix solving module is used to directly calculate the relevant parameters of the two perspective transformation inverse matrices according to the six coordinate points obtained in the previous step. Then, the double-row license plate is processed by the two obtained perspective transformation inverse matrices, and the upper and lower character areas in the double-row character license plate image are corrected to the same size image matrix images through the perspective inverse transformation inverse matrix, effectively separating the two rows of characters; finally, the obtained rectangular upper and lower license plate character areas are spliced ​​left and right to form a single-row license plate. This method can not only automatically convert the double-row license plate recognition problem into a single-row license plate recognition problem, but also does not require additional license plate character area annotation information, and only requires weak supervision information fed back by the license plate recognition network.

[0109] 2. In view of the serious shortage of double-row license plate data sets, which leads to the inability to effectively train deep learning network models, the present invention builds a double-row character license plate image data set by synthesis and mapping. In order to meet the multi-angle and multi-scene requirements of double-row license plate image data sets in unconstrained environments, the present invention maps the synthesized double-row license plate images to the CCPD data set, and then uses the annotation information of CCPD to extract a sufficient number of double-row license plate images with unconstrained scenes and various angles, and finally mixes the synthesized double-row license plates with real double-row license plates to form a self-built data set.

[0110] 3. In order to further improve the convergence of the proposed weakly supervised recognition information driven double-row license plate character area automatic segmentation and splicing algorithm module in training, the present invention proposes a new training strategy. The first step is to use all training samples to perform initial training on the recognition network. At this time, the initial parameters of the correction module are in a frozen state, that is, the correction module will not perform any form of transformation or correction on the license plate image. All training samples actually enter the recognition network directly for training. The second step is to freeze the parameters of the recognition network that has undergone preliminary training. In addition, a positive and negative sample training image set extraction method is used to use the trained recognition network to screen out the new data A that is correctly recognized and the data set B that is incorrectly recognized due to spatial distortion, and take out equal amounts of license plate images from data sets A and B and remix them into a new data set. The third step is to freeze the recognition network parameters and use the mixed data set to train the correction module to better adapt to the input format required by the recognition network. This training strategy allows the spatial transformation parameters to be learned with recognition accuracy as the center, thereby enhancing the accuracy of license plate recognition.

Claims

1. The double-row license plate image automatic correction module driven by weakly supervised recognition information is characterized by include: License plate coordinate point regression subnet, transformation matrix solution module, grid generation module, interpolation sampling module; The license plate coordinate point regression subnet includes seven convolutional layers and three fully connected layers. The convolutional layers are used to extract the feature information of the six points of the license plate coordinates. The three fully connected layers regress the coordinates of the six points of the license plate by integrating the feature information. The transformation matrix solving module calculates the perspective transformation inverse matrix through perspective inverse transformation according to the six coordinate points obtained by the license plate coordinate point regression subnet and the corrected output image size; The grid generation module takes the two perspective transformation inverse matrices M1 and M2 obtained by the transformation matrix solving module and the pixel coordinates of the target license plate image as inputs, obtains the pixel coordinates of the input license plate image by calculation, and then uses the perspective transformation inverse matrix M1 to process each pixel point of the upper half of the input license plate image to obtain the pixel coordinates of the upper half of the target license plate image, and then uses the perspective transformation inverse matrix M2 to process each pixel point of the lower half of the input license plate image to obtain the pixel coordinates of the lower half of the target license plate image; finally, the corresponding relationship between the positions of the input license plate image and the target license plate image is found, and the mapping relationship between the positions of the input license plate image and the target license plate image is determined; The interpolation sampling module uses the mapping relationship T obtained by the grid generation module M1 and T M2 The pixel points of the input license plate image are processed, and the corresponding points of the output license plate image are filled with pixels according to the pixel values ​​of the corresponding coordinate points of the input license plate image to obtain the transformed image.

2. The double-row license plate image automatic correction module driven by weakly supervised recognition information according to claim 1 is characterized by: By the mapping relation T M1 The output license plate image is the rectified image of the first line of characters in the double-line license plate. M2 The output license plate image obtained is the image after the second line of character image area of ​​the double-line character license plate is rectified; the target license plate image is obtained by splicing the two rectified images from left to right.

3. The double-row license plate image automatic correction module based on weakly supervised recognition information drive according to claim 1 is characterized by: In the transformation matrix solving module, 4 coordinate points are required to calculate an inverse perspective transformation matrix, and two of the 6 coordinate points are shared. The transformation matrix solving module divides the double-row license plate into two upper and lower regions, and calculates two inverse perspective transformation matrices; in: U i (X ui ,Y ui )(i=1,2,3,4) are the top four points of the double-row license plate in the input license plate image; V i (X vi ,Y vi )(i=1,2,3,4) are the following four points of the double-row license plate in the input license plate image; D ui (X di ,Y di ) are the coordinates of the four corner points of the output image of the upper half of the double-row license plate; D vi (X di ,Y di ) are the coordinates of the four corner points of the output image of the lower half of the double-row license plate; M1 and M2 are the inverse perspective transformation matrices; A1 is the coordinate set of all points in the quadrilateral area surrounded by the four points on the license plate in the input license plate image; A2 is the coordinate set of all points in the quadrilateral area surrounded by the four points below the license plate in the input license plate image; B1 and B2 are matrices consisting of the coordinates of the corresponding pixels of the output license plate image.

4. The double-row license plate image automatic correction module driven by weakly supervised recognition information according to claim 3 is characterized by: The transformation matrix solution process includes the following steps: STEP 1: Determine U i and D ui Are the dimensions of V equal and greater than or equal to 4? i and D vi Are the dimensions equal and greater than or equal to 4? STEP2: Initialize matrices A1 and B1; A2 and B2; STEP3: Solve the linear equations A1*M1=B1; A2*M2=B2, and use the perspective inverse transformation formula to solve M1 and M2; STEP4: Post-process the inverse matrix of perspective transformation; STEP5: Return the inverse perspective transformation matrices M1 and M2.

5. The double-row license plate image automatic correction module driven by weakly supervised recognition information according to claim 1 is characterized by: In the interpolation sampling module, I is the input license plate image; M1 and M2 are the inverse affine transformation matrices corresponding to the upper and lower halves of the double-row license plate respectively; H and W are the height and width of the input license plate image I; G1 and G2 are coordinate grids ranging from [0,H)×[0,W); G1′ and G2′ are transformed grids obtained by applying the inverse perspective transformation matrices M1 and M2; I1 and I2 are transformed images; I′ is the image after splicing I1 and I2, that is, splicing the upper and lower parts of the double-row license plate left and right.

6. The double-row license plate image automatic correction module driven by weakly supervised recognition information according to claim 5 is characterized by: The interpolation sampling module solution includes the following steps: Step 1: Initialize empty output images I1 and I2; Step 2: Get the height H and width W of the input image I; Step 3: Construct coordinate grids G1 and G2; Step 4: Apply the inverse perspective transformation matrix M1 and M2 to the grids G1 and G2 to obtain the transformed grids G1′ and G2′; Step 5: For each point in the transformed grids G1′ and G2′: Calculate its position in the input image I; Use bilinear interpolation method to get the pixel value at that position; Put the pixel values ​​into the corresponding positions of I1 and I2; Step 6: Return the transformed images I1 and I2, and splice I1 and I2 left and right to obtain the target image I′.

7. The double-row license plate image automatic correction module driven by weakly supervised recognition information according to claim 6 is characterized by: The bilinear interpolation method is used to determine the pixel value of the target image. The bilinear interpolation method calculates the pixel value of a point in the target image through the pixel values ​​of four points around the original image, as shown in equations (2) to (4): Among them, Q 11 x1,y1,Q 12 x1,y2,Q 21 x2,y1,Q 22 x2, y2 are four points of the original image, whose coordinates and pixel values ​​are known. These four points are the four points closest to the points mapped back to the original image from the target image; Px, y is a point mapped back to the original image from the target image, and only the coordinates of the point are known; By using equations (2) and (3) to perform linear interpolation in the direction of the X axis, and then by using equation (4) to perform linear interpolation in the direction of the Y axis, bilinear interpolation is implemented to obtain the pixel value of the point.

8. A license plate recognition network training strategy based on any one of the two-row license plate image automatic correction modules of claims 1 to 7, characterized in that: In the first stage of training, all training samples are used to train the license plate recognition network. In this process, the last fully connected layer of the license plate coordinate point regression subnet in the automatic correction module is forced to output six points A, B, C, D, E, and F. The license plate image is directly fed into the license plate recognition network for training. In the second stage of training, the parameters of the license plate recognition network trained in the first stage are frozen, and the parameters of the license plate recognition network are no longer updated in the second stage of training. The positive and negative sample training image set extraction process is adopted to obtain the data set for training the automatic correction module.

9. The license plate recognition network training strategy according to claim 8 is characterized by: First, randomly select the original license plate image and put it into the license plate recognition network trained in the first stage of training to determine whether it is recognized correctly. The correctly recognized license plate image is retained as the new training data A; Secondly, for the wrongly recognized license plate image, it is determined again whether the recognition error is caused by image distortion, and the distorted license plate image is retained as new training data B; Finally, the new training data A and B are combined in the same amount to form a new data set, which is used to train the automatic correction module.

10. A license plate recognition network training strategy based on any one of the two-row license plate image automatic correction modules of claims 1 to 7, characterized in that: The automatic correction module performs spatial transformation on the distorted license plate images in the input data set to obtain an input format that is more suitable for the license plate recognition network; then, the trained automatic correction module is combined with the frozen license plate recognition network, and the entire network is fine-tuned.

Images