Plug-and-play license plate image automatic correction module suitable for open scene and training strategy of plug-and-play license plate image automatic correction module

Abstract

The invention discloses a plug-and-play license plate image automatic correction module suitable for an open scene and a training strategy thereof. The license plate image automatic correction module comprises a license plate vertex offset regression subnet, a transformation matrix solving module, a grid generation module and an interpolation sampling module. The license plate vertex offset regression subnet is connected with a transformation matrix solving module, the transformation matrix solving module is connected with a grid generation module, and the grid generation module is connected with an interpolation sampling module; inputting a license plate image into a license plate vertex offset regression subnet and an interpolation sampling module; and the interpolation sampling module outputs the transformed license plate image to a license plate recognition network. The input license plate image can be geometrically corrected in a self-adaptive manner on the basis of the recognition accuracy; the license plate image automatic correction module has excellent performance when processing the license plate image with an improper shooting angle.

Description

Technical Field

[0001] The present invention relates to the technical field of license plate image correction based on computer vision technology, and in particular to a plug-and-play license plate image automatic correction module suitable for open scenarios and a training strategy thereof. Background Art

[0002] In constrained scenarios, such as parking lots and toll booths, the mainstream method for license plate positioning and correction is to use traditional computer vision technology because the shooting environment is fixed. Reference [1]: Yang Dingding, Chen Shiqiang, Liu Jingyi. License plate positioning algorithm based on license plate background and character color features [J]. Computer Applications and Software, 2018, 35(12): 216-221. A license plate correction method based on license plate background and color features is proposed. This method uses a color model to roughly locate the license plate according to the rules of the fixed color matching of the license plate background and characters, and then uses the improved Canny edge detection algorithm to obtain the license plate edge, thereby obtaining the accurate license plate position. However, this method has strict requirements on the license plate background, is easily affected by noise, is insensitive to weak edge information, and is prone to miss weak edge information, resulting in misjudgment of edge detection results.

[0003] Reference [2]: Alhussein M, Aurangzeb K, Haider S I. Vehicle license plate detection and perspective rectification [J]. Elektronika ir Elektrotechnika, 2019, 25(5): 47-56. A vehicle image perspective transformation correction method is proposed. This work first obtains the license plate area based on the traditional threshold segmentation algorithm, then uses the edge characteristics of the license plate to find the four boundaries of the license plate, and fits the edge by selecting 2 detection points on each boundary. Finally, the intersection of the four edge lines is used as the four vertices of the license plate to calculate the perspective change matrix, thereby realizing license plate correction.

[0004] The above-mentioned license plate image correction method based on traditional computer vision technology is effective when the license plate image quality is good; however, for unconstrained open scenes, due to the many factors affecting the shooting environment, the obtained license plate image often shows blurred edges, high noise and deformation of the overall shape of the license plate. Traditional computer vision methods are not able to cope with this problem well. The license plate image correction algorithm based on deep learning technology has hierarchical image feature extraction capabilities and end-to-end learning capabilities, which enable it to cope with various noise interference problems in open scenes well, and is currently the mainstream.

[0005] In order to better solve the license plate correction problem, the STN spatial transformer network that can achieve spatial correction has received widespread attention in license plate recognition. For example, the document [3]: Jaderberg M, Simonyan K, Zisserman A. Spatial transformer networks [J]. Advances in neural information processing systems, 2015, 28. STN is a type of network structure designed to improve the spatial invariance of convolutional neural networks. It can adjust the spatial structure of input data during training, thereby increasing the model's attention to input data. It can also be trained end-to-end with the entire deep learning model, so that the geometric transformation and feature extraction parts of the model can be learned together to improve model performance.

[0006] Reference [4]: ​​Li Wenjie, Zhang Zusheng, Zhou Kunxiao, et al. License plate recognition algorithm RD-LPRNet based on image correction and denoising[J]. Journal of Dongguan University of Technology, 2023, 30(05): 22-33. A license plate recognition network RD-LPRNet based on the combination of STN and denoising module is proposed. The network first predicts the affine transformation parameters by creating a smaller image copy, and then applies these parameters to the original image, and finally realizes the positioning and correction of the target area.

[0007] Reference [5]: Dai Wang. Research and implementation of license plate detection and recognition based on deep learning [D]. Nanjing University of Posts and Telecommunications, 2022.

[0008] Reference [6]: Niu Zhigang, Zhang Ruiquan, Gao Shujing, et al. End-to-end license plate detection and recognition based on deep learning [J]. Electronic Design Engineering, 2023, 31(10): 147-151. The original STN network is directly used for license plate correction. The STNs used in these works are all the original STNs of the spatial correction network designed for handwritten character recognition in reference [3]. This type of STN, including the STN network model that comes with the mainstream deep learning framework Pytorch, only considers the affine transformation matrix when regressing the spatial transformation matrix, that is, only simple transformations on the plane such as cropping, rotation, and scaling are realized.

[0009] However, the license plate captured in an open scene not only has affine transformations such as rotation and scaling in the plane, but also perspective transformation in space. This transformation will cause the spatial relationship of the four vertices of the license plate to change, and the parallel relationship of the license plate edge lines will no longer be maintained. In this case, it is impossible to solve the problem by relying solely on the 2×3 affine transformation matrix used in the original STN network.

[0010] like Figure 1 As shown, Figure 1Part a in the figure is the license plate image before correction, and part b is the license plate effect image after correction using the traditional STN. It can be seen that the original STN network cannot effectively solve the perspective transformation of the license plate image. Since the perspective transformation will cause the positions of the vertices of the license plate image to change in the image, and the parallel relationship between the license plate boundary lines cannot be maintained, the standard affine transformation cannot accurately correct this deformation. Figure 1 The effect before and after correction is not obvious.

[0011] License plate recognition technology in open environments has a wide range of applications. However, due to the complexity of the shooting scenes, various distortions often appear in license plate images, which brings great challenges to the subsequent improvement of recognition accuracy. Summary of the invention

[0012] In view of the common problems of the above-mentioned license plate correction algorithm, the present invention proposes a plug-and-play license plate image automatic correction module and its training strategy suitable for open scenarios by improving the STN network structure, which can adaptively perform geometric correction on the input license plate image based on recognition accuracy; the license plate image automatic correction module has excellent performance when processing license plate images with incorrect shooting angles.

[0013] The technical solution adopted by the present invention is:

[0014] A plug-and-play license plate image automatic correction module suitable for open scenarios, which includes: a license plate vertex offset regression subnet, a transformation matrix solution module, a grid generation module and an interpolation sampling module:

[0015] The license plate vertex offset regression subnet is connected to the transformation matrix solving module, the transformation matrix solving module is connected to the grid generation module, and the grid generation module is connected to the interpolation sampling module;

[0016] The license plate image is input to the license plate vertex offset regression subnet and interpolation sampling module;

[0017] The interpolation sampling module outputs the transformed license plate image to the license plate recognition network.

[0018] The license plate vertex offset regression subnet includes seven convolutional layers and three fully connected layers. The input license plate image of the license plate vertex offset regression subnet comes from the license plate positioning image. The convolutional layer is used to extract key features related to the license plate vertex coordinate offset. The three fully connected layers regress the offset of the required license plate image coordinates by integrating feature information.

[0019] Among them, (X i ,Y i ) represents the coordinates of the four vertices of the license plate,

[0020] (X i +ΔX i ,Yi +ΔY i ) represents the coordinates of the four corner points of the rectangular frame surrounding the license plate;

[0021] X i (i=1,2,3,4) represents the four vertices of the license plate, which are the horizontal coordinates of the upper left, upper right, lower left, and lower right vertices;

[0022] Y i (i=1,2,3,4) represents the four vertices of the license plate, which are the ordinates of the upper left, upper right, lower left, and lower right vertices;

[0023] ΔX i (i=1,2,3,4) represents the four vertices X of the license plate. i (i=1,2,3,4) lateral offset;

[0024] ΔY i (i=1,2,3,4) represents the four vertices Y of the license plate respectively i The longitudinal offset of (i=1,2,3,4).

[0025] The entire license plate vertex offset regression subnet can regress 8 parameters, which represent the offset of the four vertices of the license plate relative to the coordinates of the four corner points of the license plate image.

[0026] The transformation matrix solving module calculates the perspective transformation inverse matrix M through the perspective inverse transformation formula according to the coordinates of the four vertices of the license plate converted by the license plate vertex offset regression subnet and the image size of the corrected output, the image size of the corrected output is a fixed known size of 94*24; wherein the coordinates of the four vertices of the license plate are obtained by adding the coordinates of the four corner points of the license plate image and the four sets of offsets obtained by regression;

[0027] Among them, S i (X si ,Y si ) are the source point coordinates, i.e. the coordinates of the four vertices of the license plate in the input image;

[0028] D i (X di ,Y di ) are the coordinates of the four corner points in the output image,

[0029] X si Represents the source point coordinates, i.e. the horizontal coordinates of the four vertices of the license plate in the input image;

[0030] Y si Represents the source point coordinates, i.e., the ordinates of the four vertices of the license plate in the input image;

[0031] X diare the coordinates of the four corner points in the output image, which are called the horizontal coordinates of the target point coordinates;

[0032] Y di are the coordinates of the four corner points in the output image, which are called the ordinates of the target point coordinates here;

[0033] M is the inverse perspective transformation matrix;

[0034] A is the coordinate set of each point in the quadrilateral area surrounded by the coordinates of the four vertices of the license plate in the input license plate image;

[0035] B is the matrix composed of the coordinates of the corresponding pixels of the output license plate image;

[0036] The perspective inverse transformation formula is as follows:

[0037] The corresponding points of the four license plate groups are: (U1, V1) corresponds to (X1, Y1), (U2, V2) corresponds to (X2, Y2), (U3, V3) corresponds to (X3, Y3), (U4, V4) corresponds to (X4, Y4), where (U i ,V i )(i=1,2,3,4) represent the coordinates of the four corners of the license plate, namely the upper left, upper right, lower left and lower right. i ,Y i )(i=1,2,3,4) represent the coordinates of the four vertices of the upper left, upper right, lower left and lower right of the license plate. At this time, the four points are substituted into the derivation of formula (1) to obtain 8 equations, which can be used to solve the inverse matrix of the perspective transformation and obtain the calculation formula in matrix form as shown in formula (1). In the calculation process, let the first matrix on the left side of formula (1) be denoted as A, (t 11 ,t 21 ,t 31 ,t 12 ,t 22 ,t 32 ,t 13 ,t 23 ) is denoted as T, (U1, V1, U2, V2, U3, V3, U4, V4) is denoted as B, and the linear equation system is solved by the method of T = A -1 B, solve 8 parameters, (t 11 ,t 21 ,t 31 ,t 12 ,t 22 ,t 32 ,t 13 ,t 23 ) The 8 parameters are the offsets of the 4 corner points, which is the updated θ. At the same time, the first 8 parameters in the perspective matrix are the results of the above formula (t 11 ,t 21 ,t 31 ,t12 ,t 22 ,t 32 ,t 13 ,t 23 )8 parameters, the ninth parameter is w, w is the scale factor, which is usually set to 1 in perspective correction, so that the inverse matrix of perspective transformation is M.

[0038]

[0039] The grid generation module,

[0040] First, the perspective transformation inverse matrix M output by the transformation matrix solving module and the pixel coordinates of the input license plate image are received, and the pixel coordinates of the input image are calculated;

[0041] Then, the perspective transformation inverse matrix M is applied to each pixel of the input license plate image, thereby transforming the pixel coordinates of the target license plate image.

[0042] Finally, the grid generation module finds the corresponding relationship between the input license plate image and the target license plate image position, determines that the value of each position is mapped from the value of a certain position of the input license plate image, and thus finds all the corresponding relationships between the positions. The mapping relationship between the input and output license plate images is recorded as T M ; When the pixel coordinate point of the input license plate image is (X U1 , X u2 ), X U1 The horizontal coordinate of the pixel coordinate point of the input image, X u2 The vertical coordinate of the pixel coordinate point of the input image; the pixel coordinate point of the target license plate image is (X v1 , X v2 ), X v1 The horizontal coordinate of the pixel coordinate of the target image, X v2 The vertical coordinate of the pixel point coordinates of the target image.

[0043] The interpolation sampling module obtains the mapping relationship T through the grid generation module. M Act on the pixel coordinates of the input license plate image, take the pixel value of the corresponding coordinate point of the input license plate image, and fill the pixel at the corresponding point of the output license plate image to obtain the transformed license plate image.

[0044] The training strategy of the license plate character recognition network based on the plug-and-play license plate image automatic correction module is as follows:

[0045] In the first stage of training, all training samples are used to train the license plate character recognition network. In this process, the bias of the last fully connected layer of the license plate vertex offset regression subnet in the automatic correction module is initialized to all 0, and the connection weight parameters are also initialized to 0, which can force the output of the last fully connected layer to be all 0, which means that the four vertices of the license plate coincide with the four corners of the image without any offset. At the same time, all parameters in the automatic correction module are frozen. It is ensured that the automatic correction module will not perform any form of correction on the license plate image, and the license plate image directly enters the license plate character recognition network for training;

[0046] After the initial training of the license plate character recognition network, its parameters are frozen, that is, these parameters will not be updated in the subsequent training process, and the feature extraction method that the license plate character recognition network has learned will remain unchanged; then, the positive and negative sample training image set extraction process is adopted to obtain the data set for training the license plate image automatic correction module.

[0047] The training strategy of the license plate character recognition network based on the plug-and-play license plate image automatic correction module includes the following steps:

[0048] The first step is to randomly select the original license plate image through Figure 4 The license plate character recognition network trained in training 1 shown in the figure determines whether the license plate image is recognized correctly, and retains the correctly recognized license plate image as a new training data set A;

[0049] In the second step, the wrongly recognized license plate image is judged again to determine whether the recognition error is caused by the distortion of the license plate image, and the distorted license plate image is retained as new training data B;

[0050] In the third step, the extracted training data sets A and B are recombined into new data sets in the same quantity for training the automatic correction module. The automatic correction module performs spatial transformation on the distorted license plate images in the input data set to better adapt to the input format required by the license plate character recognition network.

[0051] Finally, the trained automatic correction module is combined with the frozen license plate character recognition network, and the two networks combined by the automatic correction module and the license plate character recognition network are fine-tuned.

[0052] The present invention provides a plug-and-play license plate image automatic correction module and its training strategy suitable for open scenarios, and the technical effects are as follows:

[0053] 1) Aiming at the problem that the original STN network cannot effectively regress the transformation matrix parameters when learning complex license plate image features, the present invention proposes a plug-and-play license plate image automatic correction module suitable for open scenarios. The automatic correction module consists of four parts: a license plate vertex offset regression subnet, a transformation matrix solution module, a grid generation module, and an interpolation sampling module. In the automatic correction module, a license plate vertex offset subnet that is more conducive to the convergence of the spatial correction network is used to replace the positioning subnet in the original STN, and a transformation matrix solution module is used to directly calculate the change matrix parameters. This solution not only improves the interpretability and effectiveness of the results of each part of the correction module, but also does not affect the calculation speed.

[0054] 2) In order to further improve the convergence of the proposed plug-and-play license plate image automatic correction algorithm module in open scenarios during training, the present invention proposes a training strategy. Through this training method, the input license plate image can be geometrically corrected adaptively based on the recognition accuracy, and the learning of spatial transformation parameters centered on the recognition accuracy is realized, thereby enhancing the accuracy of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0056] Figure 1 This is the correction effect diagram of the traditional STN network.

[0057] Figure 2 This is a structural diagram of a plug-and-play license plate image automatic correction module of the present invention.

[0058] Figure 3 Schematic diagram of the license plate vertex offset regression network.

[0059] Figure 4 Training policy graph for the autocorrection module.

[0060] Figure 5 Flowchart for extracting the positive and negative training image sets used to train the automatic correction module.

[0061] Figure 6 This is a comparison of the license plate before and after correction.

[0062] Figure 6 In the figure, part a is the license plate image before correction, and part b is the license plate effect image after correction of the present invention.

[0063] Figure 7 This is the license plate recognition effect diagram;

[0064] Figure 7 In the figure, part a is a recognition effect diagram before correction, and part b is a recognition effect diagram after correction of the present invention. DETAILED DESCRIPTION

[0065] Aiming at the problem of correcting license plate images in open scenes, the present invention proposes a plug-and-play license plate image automatic correction module suitable for open scenes. Different from the widely used STN network, the proposed correction module adopts a license plate vertex offset subnet that is more conducive to the convergence of the spatial correction network, replacing the positioning subnet in the original STN network. On this basis, a change matrix solution module is added, and the change matrix parameters are directly calculated by it. This solves the problem that the original STN network is difficult to effectively regress the perspective transformation parameters when processing complex license plate image features without affecting the calculation speed, and improves the interpretability of the results of each part of the correction algorithm module. In order to further improve the convergence of the plug-and-play license plate image automatic correction module suitable for open scenes in the training process. The present invention proposes a training strategy, which not only uses the positive and negative sample training image set extraction method to extract the data set of the training correction module, but also achieves the purpose of learning the spatial transformation parameters centered on the recognition accuracy.

[0066] A plug-and-play license plate image automatic correction module suitable for open scenarios. The module consists of four parts: Vertex-offset Regression Network (hereinafter referred to as Vertex-offset RN), transformation matrix solution module, grid generation module and interpolation sampling module. The overall structure is as follows Figure 2 shown.

[0067] (I): License plate image automatic correction module composition:

[0068] 1. License plate vertex offset regression subnet:

[0069] The structure of the license plate vertex offset regression subnet is as follows: Figure 3 As shown, where (X i ,Y i )(i=1,2,3,4) represents the coordinates of the four vertices of the license plate, (X i +ΔX i ,Y i +ΔY i )(i=1,2,3,4) represents the coordinates of the four corner points of the rectangular frame circumscribing the license plate. The entire subnet can regress 8 parameters, which represent the offset of the four vertices of the license plate relative to the coordinates of the four corner points of the license plate image.

[0070] The input image of the license plate vertex offset regression subnet comes from the license plate positioning image. The present invention does not impose any restrictions on the license plate positioning or license plate detection algorithm, and is applicable to the license plate area detected by any license plate positioning algorithm. The input size set in the present invention is 24*94. If the output result of the license plate positioning algorithm adopted is not of this size, it can be processed by a simple size transformation, which is not repeated here. In the license plate image area, the features of the input license plate are first extracted by the feature extraction layer, and then the high-level features are extracted by the fully connected layer. The subnet-related parameters are shown in Table 1. The main body consists of seven convolutional layers and three fully connected layers. The convolutional layer is used to extract key features related to the license plate vertex coordinate offset. The last three fully connected layers can integrate the feature information to regress the required coordinate offset.

[0071] Table 1 License plate vertex offset regression network composition and related parameters

[0072]

[0073] 2. Transformation matrix solution module:

[0074] This module calculates the perspective transformation inverse matrix based on the four vertex coordinates of the license plate converted by the vertex license plate vertex offset regression subnet and the corrected output license plate image size through the perspective inverse transformation formula. The four vertex coordinates of the license plate are obtained by adding the four corner point coordinates of the license plate image and the four sets of offsets regressed.

[0075] The algorithm steps of the transformation matrix solution module are as follows:

[0076]

[0077] Where S i (X si ,Y si )(i=1,2,3,4) are the source point coordinates, i.e., the coordinates of the four vertices of the license plate in the input license plate image;

[0078] D i (X di ,Y di )(i=1,2,3,4) are the coordinates of the four corner points in the output license plate image, here called the target point coordinates; M is the inverse perspective transformation matrix; A is the coordinate set of each point in the quadrilateral area surrounded by the coordinates of the four vertices of the license plate in the input license plate image; B is the matrix composed of the coordinates of each corresponding pixel of the output license plate image.

[0079] 3. Mesh generation module:

[0080] The aforementioned transformation matrix solving module solves a perspective transformation inverse matrix M with a parameter of 3*3. In order to generate a corrected image, the grid generation module needs to first receive the perspective transformation inverse matrix M output by the transformation matrix solving module and the pixel coordinates of the target license plate image, and calculate the pixel coordinates of the input license plate image. Then the transformation matrix M is applied to each pixel point of the input license plate image, thereby transforming the pixel coordinates of the target license plate image. Finally, the grid finds the corresponding relationship between the positions of the input license plate image and the target license plate image, determines which position of the input license plate image the value of each position is mapped from, and thus finds all the position correspondences. The mapping relationship between the input and output images is recorded as T. M When the pixel coordinate point of the input license plate image is (X U1 , X u2 ), the pixel coordinates of the target license plate image are (X v1 , X v2 ), the mapping relationship between the two is as follows Figure 2 The mesh generation module is shown in .

[0081] 4. Interpolation sampling module:

[0082] The interpolation sampling module is mainly based on the mapping relationship T obtained by the grid generation module. M Act on the pixel coordinates of the input license plate image, take the pixel value of the corresponding coordinate point of the input license plate image, and fill the pixel at the corresponding point of the output license plate image to obtain the transformed license plate image.

[0083] This process involves adjusting the position of each point in the input license plate image so that they match the transformed image space. It uses an interpolation algorithm to calculate pixel values ​​based on position mapping. Since the position correspondence of the input license plate image in the input license plate image may be a decimal, a bilinear interpolation algorithm is used to calculate the pixel value of the interpolation point and assign it to the output image. Since the bilinear interpolation algorithm is a widely used algorithm, the present invention will not be described in detail here. The interpolation sampling module algorithm is as follows:

[0084]

[0085] In this way, the interpolation sampling module can effectively improve the resolution of the image while maintaining image details, while maintaining high computational efficiency and good locality, thereby improving the robustness of the model to geometric changes in the input data and helping to improve the overall recognition or detection performance.

[0086] (II): Training of the license plate image automatic correction module:

[0087] In order to further improve the convergence of the plug-and-play license plate image automatic correction module in the open scene during the training process, the present invention proposes a method such as Figure 4 The training strategy shown.

[0088] In the first stage of training, all training samples are used to train the recognition network. The recognition network of the present invention uses the widely adopted LPRNet recognition network. In this process, the bias of the last fully connected layer of the license plate vertex offset regression subnet in the automatic correction module is initialized to all 0, and the connection weight parameters are also all initialized to 0. This allows the last fully connected layer to force the output to be all 0, which means that the four vertices of the license plate coincide with the four corners of the image without any offset, and all parameters in the automatic correction module are frozen at the same time. This ensures that the automatic correction module will not perform any form of correction on the license plate image, and the license plate image actually enters the recognition network directly for training.

[0089] After the recognition network is initially trained, its parameters are frozen, that is, these parameters are no longer updated in the subsequent training process. The purpose of this is to keep the feature extraction method that the recognition network has learned unchanged. Then, use the following method: Figure 5 The positive and negative sample training image set extraction process is shown to obtain the data set for training the license plate image automatic correction module.

[0090] Specifically:

[0091] In the first step, the original license plate image is randomly selected, and the license plate character recognition network trained in training 1 is used to determine whether the license plate image is recognized correctly, and the correctly recognized license plate image is retained as the new training data set A.

[0092] In the second step, the incorrectly recognized license plate image is judged again to determine whether the recognition error is caused by the distortion of the license plate image, and the distorted license plate image is retained as new training data B.

[0093] In the third step, the extracted training data sets A and B are recombined into new data sets according to the same number for training the correction module. The goal of the license plate image automatic correction module is to perform spatial transformation on the distorted license plate images in the input data set so as to better adapt to the input format required by the recognition network.

[0094] Finally, the trained correction module is combined with the frozen recognition network to fine-tune the entire network. The learning rate in this process is set very small, with the aim of fine-tuning the network parameters to improve performance while retaining existing knowledge.

[0095] (III): Verification Example:

[0096] In order to compare the effectiveness of the plug-and-play license plate image automatic correction module proposed in the present invention, which is suitable for open scenarios, it is evaluated using the large-scale dataset CCPD focused on license plate recognition that is widely used in the industry, and the statistical results shown in Table 2 are obtained.

[0097] Table 2 Comparative experiment of CCPD recognition (%)

[0098]

[0099] The “LPRnet” mentioned in Table 2 is recorded in the literature [7]: Zherzdev S, Gruzdev A. Lprnet: License plate recognition via deep neural networks [J / OL]. https: / / arxiv.org / pdf / 1806.10447.pdf, [2023-4-20].

[0100] As can be seen from Table 2, the license plate recognition accuracy rate increased by about 6% after correction, especially in the Rotate subset with a larger license plate inclination angle, the recognition accuracy rate increased by 28% after correction, including significant improvements in other subsets. In addition, the recognition accuracy rate of the license plate corrected by the present invention is improved compared with the recognition accuracy rate of the original STN network corrected in each CCPD subset, while the processing speed is kept unaffected, further demonstrating the effectiveness of the solution proposed by the present invention.

[0101] The comparison of license plate correction before and after is shown below: Figure 6 As shown, from Figure 6 It can be seen that the license plate correction effect is obvious, and the difficulty of license plate identification is significantly reduced. The license plate recognition effect is as follows Figure 7 As shown, after license plate correction, the recognition effect is significantly improved. Such correction can greatly reduce the error rate and ensure that the license plate recognition system can correctly identify license plate information in various scenarios.

[0102] In addition, the license plate image automatic correction module and recognition network proposed in the present invention can be trained synchronously and can be adapted to any recognition network. Therefore, the license plate recognition network after the correction network proposed in the present invention is not limited to one type, but is widely applicable to multiple recognition networks and can be flexibly adjusted according to the usage scenario.

Claims

1. A plug-and-play license plate image automatic correction module suitable for open scenarios, characterized by This module includes: license plate vertex offset regression subnet, transformation matrix solution module, grid generation module and interpolation sampling module: The license plate vertex offset regression subnet is connected to the transformation matrix solving module, the transformation matrix solving module is connected to the grid generation module, and the grid generation module is connected to the interpolation sampling module; The license plate image is input to the license plate vertex offset regression subnet and interpolation sampling module; The interpolation sampling module outputs the transformed license plate image to the license plate recognition network.

2. According to claim 1, a plug-and-play license plate image automatic correction module suitable for open scenarios is characterized by: The license plate vertex offset regression subnet includes seven convolutional layers and three fully connected layers. The input license plate image of the license plate vertex offset regression subnet comes from the license plate positioning image. The convolutional layer is used to extract key features related to the license plate vertex coordinate offset. The three fully connected layers regress the offset of the required license plate image coordinates by integrating feature information. Among them, (X i ,Y i ) represents the coordinates of the four vertices of the license plate; (X i +ΔX i ,Y i +ΔY i ) represents the coordinates of the four corner points of the rectangular frame surrounding the license plate; X i (i=1,2,3,4) represents the four vertices of the license plate, which are the horizontal coordinates of the upper left, upper right, lower left, and lower right vertices; Y i (i=1,2,3,4) represents the four vertices of the license plate, which are the ordinates of the upper left, upper right, lower left, and lower right vertices; ΔX i (i=1,2,3,4) represents the four vertices X of the license plate. i (i=1,2,3,4) lateral offset; ΔY i (i=1,2,3,4) represents the four vertices Y of the license plate respectively i The longitudinal offset of (i=1,2,3,4); The entire license plate vertex offset regression subnet can regress 8 parameters, which represent the offset of the four vertices of the license plate relative to the coordinates of the four corner points of the license plate image.

3. According to claim 1, a plug-and-play license plate image automatic correction module suitable for open scenarios is characterized by: The transformation matrix solving module calculates the perspective transformation inverse matrix M through the perspective inverse transformation formula according to the coordinates of the four vertices of the license plate converted by the license plate vertex offset regression subnet and the image size of the corrected output; wherein the coordinates of the four vertices of the license plate are obtained by adding the coordinates of the four corner points of the license plate image and the four sets of offsets obtained by regression; Among them, S i (X si ,Y si ) are the source point coordinates, i.e. the coordinates of the four vertices of the license plate in the input image; D i (X di ,Y di ) are the coordinates of the four corner points in the output image, X si Represents the source point coordinates, i.e. the horizontal coordinates of the four vertices of the license plate in the input image; Y si Represents the source point coordinates, i.e., the ordinates of the four vertices of the license plate in the input image; X di are the coordinates of the four corner points in the output image, which are called the horizontal coordinates of the target point coordinates here; Y di are the coordinates of the four corner points in the output image, which are called the ordinates of the target point coordinates here; M is the inverse perspective transformation matrix; A is the coordinate set of each point in the quadrilateral area surrounded by the coordinates of the four vertices of the license plate in the input license plate image; B is a matrix composed of the coordinates of the corresponding pixels of the output license plate image.

4. According to claim 1, a plug-and-play license plate image automatic correction module suitable for open scenarios is characterized by: The perspective inverse transformation is as follows: The corresponding points of the four license plate groups are: (U1, V1) corresponds to (X1, Y1), (U2, V2) corresponds to (X2, Y2), (U3, V3) corresponds to (X3, Y3), (U4, V4) corresponds to (X4, Y4), where (U i ,V i )(i=1,2,3,4) represent the coordinates of the four corners of the license plate, namely the upper left, upper right, lower left and lower right. i ,Y i )(i=1,2,3,4) represent the coordinates of the four vertices of the upper left, upper right, lower left and lower right of the license plate respectively; at this time, the four points are substituted into the derivation of formula (1) to obtain 8 equations, and the inverse matrix of the perspective transformation can be solved inversely, and the calculation formula in matrix form is shown in formula (1); In the calculation process, let the first matrix on the left side of equation (1) be denoted as A, (t 11 ,t 21 ,t 31 ,t 12 ,t 22 ,t 32 ,t 13 ,t 23 ) is denoted as T, (U1, V1, U2, V2, U3, V3, U4, V4) is denoted as B, and the linear equation system is solved by the method of T = A -1 B, solve 8 parameters, (t 11 ,t 21 ,t 31 ,t 12 ,t 22 ,t 32 ,t 13 ,t 23 ) The 8 parameters are the offsets of the 4 corner points, which is the updated θ. At the same time, the first 8 parameters in the perspective matrix are the results of the above formula (t 11 ,t 21 ,t 31 ,t 12 ,t 22 ,t 32 ,t 13 ,t 23 )8 parameters, so the inverse matrix of perspective transformation is M; 5. According to claim 1, a plug-and-play license plate image automatic correction module suitable for open scenarios is characterized by: The grid generation module, First, the perspective transformation inverse matrix M output by the transformation matrix solving module and the pixel coordinates of the input license plate image are received, and the pixel coordinates of the input image are calculated; Then, the perspective transformation inverse matrix M is applied to each pixel of the input license plate image, thereby transforming the pixel coordinates of the target license plate image; Finally, the grid generation module finds the corresponding relationship between the input license plate image and the target license plate image position, determines that the value of each position is mapped from the value of a certain position of the input license plate image, and thus finds all the corresponding relationships between the positions. The mapping relationship between the input and output license plate images is recorded as T M ; When the pixel coordinate point of the input license plate image is (X U1 , X u2 ), X U1 The horizontal coordinate of the pixel coordinate point of the input image, X u2 The vertical coordinate of the pixel coordinate point of the input image; the pixel coordinate point of the target license plate image is (X v1 , X v2 ), X v1 The horizontal coordinate of the pixel coordinate of the target image, X v2 The vertical coordinate of the pixel point coordinates of the target image.

6. According to claim 1, a plug-and-play license plate image automatic correction module suitable for open scenarios is characterized by: The interpolation sampling module obtains the mapping relationship T through the grid generation module. M Act on the pixel coordinates of the input license plate image, take the pixel value of the corresponding coordinate point of the input license plate image, and fill the pixel at the corresponding point of the output license plate image to obtain the transformed license plate image.

7. A license plate character recognition network training strategy based on the license plate image automatic correction module of any one of claims 1 to 6, characterized in that: In the first stage of training, all training samples are used to train the license plate character recognition network. In this process, the bias of the last fully connected layer of the license plate vertex offset regression subnet in the automatic correction module is initialized to all 0, and the connection weight parameters are also initialized to 0, so that the last fully connected layer can be forced to output all 0, which means that the four vertices of the license plate coincide with the four corners of the image without any offset. At the same time, all parameters in the automatic correction module are frozen to ensure that the automatic correction module will not perform any form of correction on the license plate image, and the license plate image directly enters the license plate character recognition network for training; After the initial training of the license plate character recognition network, its parameters are frozen, that is, these parameters will not be updated in the subsequent training process, and the feature extraction method that the license plate character recognition network has learned will remain unchanged; then, the positive and negative sample training image set extraction process is adopted to obtain the data set for training the license plate image automatic correction module.

8. A license plate character recognition network training strategy based on the license plate image automatic correction module of any one of claims 1 to 6, characterized in that The following steps are involved: In the first step, the original license plate image is randomly selected, and the license plate character recognition network that has been trained is used to determine whether the license plate image is correctly recognized, and the correctly recognized license plate image is retained as the new training data set A; In the second step, the wrongly recognized license plate image is judged again to determine whether the recognition error is caused by the distortion of the license plate image, and the distorted license plate image is retained as new training data B; The third step is to reassemble the extracted training data sets A and B into new data sets according to the same number for training the automatic correction module. The automatic correction module performs spatial transformation on the distorted license plate images in the input data set to better adapt to the input format required by the license plate character recognition network. Finally, the trained automatic correction module is combined with the frozen license plate character recognition network, and the two networks combined by the automatic correction module and the license plate character recognition network are fine-tuned.

Images