A license plate positioning method, device and storage medium for license plate recognition

Abstract

The present invention discloses a license plate positioning method, device and storage medium for license plate recognition, the method comprising: obtaining an image to be recognized, wherein the image to be recognized contains at least one license plate to be recognized; inputting the image to be recognized into a license plate positioning model for positioning detection, and obtaining four license plate corner point coordinates corresponding to the license plate to be recognized, so as to complete the license plate positioning in the image to be recognized according to the four license plate corner point coordinates, wherein the license plate positioning model comprises a MobileNetV1‑0.25 network, an FPN network, an SSH network and a head network; the present invention can directly obtain the four corner points of the license plate, realize end-to-end license plate positioning, avoid the large error and low precision problems existing in target detection and image processing technology, and the present invention adopts the MobileNetV1‑0.25 network as the backbone network, realizes license plate positioning with almost no loss of precision, reduces the parameter amount and calculation amount of the overall model, provides conditions for terminal deployment of license plate recognition system, and is suitable for large-scale promotion and application.

Description

Technical Field

[0001] The invention belongs to the technical field of license plate recognition, and in particular relates to a license plate positioning method, a device and a storage medium for license plate recognition. Background Art

[0002] With the development of computer vision, digital image processing technology and intelligent transportation technology, license plate recognition technology is being used more and more widely in the field of intelligent transportation, such as parking gates, highway toll stations and traffic violation identification. In license plate recognition technology, license plate positioning and license plate correction are the primary issues and are also important factors affecting the accuracy of license plate recognition.

[0003] Among the existing license plate positioning technologies, traditional image processing algorithms and target detection algorithms based on deep learning are mainly used. Among them, the license plate recognition technology based on traditional image processing algorithms mainly locates and segments the license plate by means of image binarization, license plate color features and license plate edge features, and then corrects the license plate by the tilt features of the license plate characters; and the target detection algorithms based on deep learning, such as Yolo (You Only Look Once, a target detection system based on a single neural network proposed by Joseph Redmon and Ali Farhadi et al. in 2015), SSD (Single Shot MultiBoxDetector, object detection network) and R-CNN (Region-CNN (Convolutional Neural Networks, convolutional neural network), is the first algorithm that successfully applied deep learning to target detection) and other target detection algorithms, have developed rapidly in the detection field, and have been applied to license plate positioning with rich theoretical support and experience references. This method has the advantages of adapting to a wide range of license plates, high recall rate and accurate positioning.

[0004] However, the above two positioning methods have the following shortcomings: due to the large number of styles and colors of Chinese license plates, including gradient green, blue, black and white styles, and the influence of factors such as lighting in actual scenes, it is difficult to locate license plates with traditional image processing algorithms, and the accuracy and robustness are insufficient; in addition, in actual scenes, there is an uncertain angle between the license plate recognition camera and the car, and the license plate located by the target detection algorithm has a tilt problem, which requires the combination of image technology or deep learning technology for tilt correction, interference filtering and other post-processing. Due to the influence of interference, it is difficult to achieve high-precision license plate positioning in practice, which also leads to low license plate recognition accuracy; therefore, it is urgent to provide a high-precision license plate positioning method. Summary of the invention

[0005] The purpose of the present invention is to provide a license plate positioning method, device and storage medium for license plate recognition, so as to solve the problem of low accuracy of existing license plate positioning methods.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a license plate positioning method for license plate recognition, comprising:

[0008] Acquire an image to be identified, wherein the image to be identified includes at least one license plate to be identified;

[0009] Input the image to be identified into the license plate positioning model for positioning detection, and obtain the coordinates of the four license plate corner points corresponding to the license plate to be identified, so as to complete the license plate positioning in the image to be identified according to the four license plate corner point coordinates, wherein the license plate positioning model includes MobileNetV1-0.25 network, FPN network, SSH network and head network;

[0010] The MobileNetV1-0.25 network is used as the backbone feature extraction network of the license plate positioning model, and is used to extract features of the image to be identified under three receptive field conditions to obtain three first feature maps;

[0011] The FPN network is used to upsample and fuse the three first feature maps to obtain three second feature maps;

[0012] The SSH network includes three parallel convolutional layers, which are used to perform convolution processing on the three second feature maps using the three parallel convolutional layers to obtain three third feature maps;

[0013] The head network is used to predict the key points of the license plate on the three third feature maps to output an 8-dimensional key point vector so as to obtain the coordinates of the four license plate corner points according to the key point vector.

[0014] Based on the above disclosed content, the present invention pre-builds a license plate positioning model by using MobileNetV1-0.25 network, FPN network, SSH network and head network, so that the image to be identified is input into the model for feature extraction, that is, the MobileNetV1-0.25 network is first used to extract the backbone features, and then the FPN network is used to perform feature fusion on the extracted backbone features to obtain a feature map with richer feature information. Then, the three parallel convolutional layers in the SSH network are used to perform multi-scale detection on the feature map output by the FPN network, and the feature extraction is further strengthened to enhance the context information of the extracted features, thereby improving the quality of the extracted license plate features and achieving the purpose of improving the license plate positioning accuracy. Finally, the head network is used to predict key points and output an 8-dimensional key point vector, that is, the present invention changes the output dimension of the key point branch in the model from 10 dimensions to 8 dimensions to match the 4 corner points of the license plate. Therefore, the present invention can directly obtain the 4 corner points of the license plate, realize end-to-end license plate positioning, avoid the large errors and low precision problems existing in target detection and image processing technology, and further improve the license plate positioning accuracy.

[0015] Through the above design, the present invention can directly obtain the four corner points of the license plate, realize end-to-end license plate positioning, avoid the large errors and low precision problems existing in target detection and image processing technologies, and the present invention adopts the MobileNetV1-0.25 network as the backbone network, realizes license plate positioning with almost no loss of precision, reduces the number of parameters and calculation amount of the overall model, provides conditions for terminal deployment of license plate recognition system, and is suitable for large-scale promotion and application.

[0016] In a possible design, the loss function of the license plate location model is:

[0017]

[0018] In the above formula, L represents the loss function of the license plate positioning model, L cls represents the target classification loss function, L box represents the detection box regression loss function, L pts represents the key point regression loss function, p i represents the probability that anchor point i is predicted as a license plate corner point, When it is 1, it indicates the true label of the positive anchor point. When it is 0, it indicates the true label of the negative anchor point, t i represents the predicted coordinates of the prediction box in the anchor box, represents the true coordinates of the predicted box in the anchor box, l i Represents the predicted license plate corner coordinates in the anchor box, represents the coordinates of the corner points of the license plate, λ1, λ2, λ3 and λ4 represent weight coefficients, which are constants;

[0019] L pts1 Represents the corner point distance constraint function, which is used to constrain the distance between the two parallel sides of the license plate to be identified;

[0020] L pts2 Represents the corner point distance ratio constraint function, which is used to constrain the ratio of the length and width of the license plate to be recognized.

[0021] In a possible design, the corner point distance constraint function is:

[0022] L pts1 (l i )=|||l0l1|-|l2l3|||+||l0l2|-|l1l3|||

[0023] In the above formula, l0l1 represents the distance between the first license plate corner point and the second license plate corner point, l2l3 represents the distance between the third license plate corner point and the fourth license plate corner point, l0l2 represents the distance between the first license plate corner point and the third license plate corner point, l1l3 represents the distance between the second license plate corner point and the fourth license plate corner point, wherein the upper left corner of the license plate to be identified is the first license plate corner point, the upper right corner is the second license plate corner point, the lower left corner is the third license plate corner point, and the lower right corner is the fourth license plate corner point.

[0024] In a possible design, the corner point distance ratio constraint function is:

[0025]

[0026] In the above formula, Indicates the distance between the first license plate annotation corner point and the second license plate annotation corner point. Indicates the distance between the first license plate annotation corner point and the third license plate annotation corner point.

[0027] Based on the aforementioned disclosed content, the present invention adds a corner point distance constraint function and a corner point distance ratio constraint function on the basis of the traditional Retinaface network, wherein the corner point distance constraint function is used to constrain the distance between the parallel sides of the license plate to be identified, that is, the distance between the two parallel sides should be equal, that is, the value of the corner point distance constraint function is constrained to 0 or close to 0, and the corner point distance ratio constraint function uses the marked points to constrain the predicted points, that is, the ratio of the length and width of the license plate obtained according to the marked corner points of the license plate should be equal to the ratio of the length and width of the license plate obtained according to the coordinates of the corner points of the license plate, that is, the corner point distance ratio constraint function is constrained to 0 or close to 0; thereby, the positioning accuracy of the corner points of the license plate can be further improved.

[0028] In one possible design, the values ​​of weight coefficients λ1, λ2, λ3, and λ4 are 0.25, 0.5, 0.01, and 0.01, respectively.

[0029] Through the above design, the present invention increases the weight of each loss function, making the corner point iteration more accurate, thereby further improving the corner point positioning accuracy.

[0030] In a possible design, the size of the prior anchor of the license plate positioning model is: 16*8, 32*16, 64*32, 128*64, 256*128 and 512*256, wherein the prior anchor is used to characterize the anchor point box of the license plate positioning model.

[0031] Based on the above disclosed content, the present invention can make the anchor point frame closer to the actual size of the license plate by improving the length and width of the anchor point frame, thereby further improving the corner point positioning accuracy.

[0032] In a possible design, after the image to be identified is input into the license plate positioning model for positioning detection and the coordinates of four license plate corner points corresponding to the license plate to be identified are obtained, the method further includes:

[0033] Obtaining the coordinates of the standard corner points of the license plate in the image to be recognized;

[0034] Performing perspective transformation processing on the image to be identified using the standard corner point coordinates of the license plate and the four corner point coordinates of the license plate to obtain a corrected image to be identified, wherein the corrected image to be identified only contains the license plate to be identified;

[0035] The corrected image to be recognized is input into the license plate recognition model to obtain a license plate recognition result.

[0036] Based on the above disclosed content, the present invention combines the license plate corner point coordinates output by the license plate positioning model with the license plate standard corner point coordinates (i.e., the coordinates preset by the user, which can be specifically set according to the size to be transformed), and performs perspective transformation processing on the image to be identified. This can achieve tilt correction of the license plate so that the corrected image only contains the license plate to be identified, thereby improving the license plate recognition accuracy.

[0037] In a second aspect, the present invention provides a license plate positioning device for license plate recognition, comprising:

[0038] An acquisition unit, used for acquiring an image to be recognized, wherein the image to be recognized contains at least one license plate to be recognized;

[0039] The license plate positioning unit is used to input the image to be identified into the license plate positioning model for positioning detection, and obtain the coordinates of the four license plate corner points corresponding to the license plate to be identified, so as to complete the license plate positioning in the image to be identified according to the four license plate corner point coordinates, wherein the license plate positioning model includes MobileNetV1-0.25 network, FPN network, SSH network and head network.

[0040] In the third aspect, the present invention provides another license plate positioning device for license plate recognition. Taking the device as a computer main device as an example, it includes a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the license plate positioning method for license plate recognition as described in the first aspect or any possible design of the first aspect.

[0041] In a fourth aspect, the present invention provides a storage medium having instructions stored thereon, which, when executed on a computer, executes the license plate positioning method for license plate recognition as in the first aspect or any possible design of the first aspect.

[0042] In a fifth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the license plate positioning method for license plate recognition as described in the first aspect or any possible design of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the structure of the license plate positioning model provided by the present invention;

[0044] Figure 2 A schematic flow chart of the steps of the license plate positioning method for license plate recognition provided by the present invention;

[0045] Figure 3 A schematic diagram of the structure of the SSH network provided by the present invention;

[0046] Figure 4 A schematic diagram of marking the four corner points of the license plate to be identified provided by the present invention;

[0047] Figure 5 A schematic diagram of the license plate positioning effect of the license plate to be identified provided by the present invention;

[0048] Figure 6 A schematic diagram of the steps of license plate recognition provided by the present invention;

[0049] Figure 7 A schematic diagram of the structure of a license plate positioning device for license plate recognition provided by the present invention;

[0050] Figure 8 This is a structural schematic diagram of a computer main device provided by the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that although the description of these embodiments is used to help understand the present invention, it does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0052] It should be understood that although the terms first, second, etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another unit. For example, a first unit can be referred to as a second unit, and similarly, a second unit can be referred to as a first unit without departing from the scope of the exemplary embodiments of the present invention.

[0053] It should be understood that the term "and / or" that may appear in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this article describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B can represent two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0054] Example

[0055] See also Figure 1As shown, a license plate positioning model is provided for the application. The license plate positioning model is improved on the traditional Retinaface detection network, which includes: MobileNetV1-0.25 network, FPN network, SSH network and head network, wherein the license plate positioning model adopts MobileNetV1-0.25 network as the backbone feature extraction network, which is used to extract features of the input image to be identified under three receptive field conditions to obtain three first feature maps (that is, feature information is extracted in different dimensions to obtain feature maps in different dimensions), and the FPN network is used to upsample and fuse the three first feature maps to obtain three second feature maps (that is, feature maps with richer feature information are obtained through feature fusion); at the same time, the SSH network in this model is provided with three parallel convolutional layers, that is, three second feature maps are respectively convolved using three parallel convolutional layers to obtain three third feature maps (the third feature maps are respectively convolved using three parallel convolutional layers to obtain three third feature maps (the third feature maps are respectively convolved using three parallel convolutional layers to obtain three third feature maps). The essence is: using convolution processing to perform multi-scale feature detection, thereby enhancing the contextual information of the extracted features to improve the quality of the extracted features, and then improve the accuracy of corner point positioning); finally, using the head network to predict the key points of the license plate on the three third feature maps, thereby outputting an 8-dimensional key point vector, so as to obtain 4 license plate corner point coordinates according to the key point vector (that is, each license plate corner point coordinate is represented by a 2-dimensional vector (that is, the horizontal and vertical coordinates), and the 8-dimensional vector corresponds to 4 corner points); thus, through the above design, the present invention can directly obtain the 4 corner points of the license plate, realize end-to-end license plate positioning, avoid the large errors and low precision problems existing in target detection and image processing technology, and the present invention adopts the MobileNetV1-0.25 network as the backbone network, which not only realizes the license plate positioning with almost no loss of accuracy, but also reduces the parameter amount and calculation amount of the overall model, providing conditions for the terminal to deploy the license plate recognition system, which is suitable for large-scale promotion and application.

[0056] The following describes the license plate positioning method provided by the embodiment of the present application in conjunction with the accompanying drawings, and applies the method to Figure 1 The license plate positioning model structure shown is explained in detail.

[0057] like Figure 2 As shown, the license plate positioning method for license plate recognition provided in the first aspect of this embodiment is applicable to license plate positioning in any scenario (such as parking lot entry and exit gates, highway toll station entry and exit gates, and traffic violation capture, etc.), which may include but is not limited to the following steps S101 and S102.

[0058] S101. Acquire an image to be identified, wherein the image to be identified includes at least one license plate to be identified.

[0059] Step S101 is the process of obtaining the image to be identified, so that it can be subsequently input into the license plate positioning model to extract the coordinates of the four license plate corner points of the license plate, thereby completing the positioning of the license plate in the image; in specific implementation, it can be but is not limited to: license plate photos taken when entering and exiting the parking lot and / or license plate photos taken when entering and exiting the highway toll station as the image to be identified, or any picture in a license plate video as the image to be identified (that is, the video is processed frame by frame to obtain multiple images), and the image containing the license plate to be identified can also be directly uploaded by the working personnel, that is, the image to be identified can choose different acquisition methods according to the specific application scenario, which is not specifically limited here.

[0060] After obtaining the image to be recognized, the aforementioned Figure 1 The model shown performs image recognition on the image to locate the license plate in the image, as shown in the following step S102.

[0061] S102. Input the image to be identified into a license plate positioning model for positioning detection to obtain the coordinates of four license plate corner points corresponding to the license plate to be identified, so as to complete the license plate positioning in the image to be identified based on the four license plate corner point coordinates.

[0062] As mentioned above, the license plate positioning model provided in this embodiment is improved on the traditional Retinaface detection network, that is, the MobileNetV1-0.25 network is used as the backbone feature extraction network of the Retinaface detection network, and the FPN network, SSH network and head network in the traditional Retinaface detection network are retained, so as to realize the extraction of the coordinates of the license plate corner points to complete the license plate positioning in the image to be identified.

[0063] See also Figure 1 As shown, the license plate positioning model is specifically described as follows:

[0064] First of all, the MobileNet network is a lightweight deep neural network proposed by Google for embedded devices such as mobile phones. The core idea it uses is depthwise separable convolution. It has three structures: V1, V2 and V3. The MobileNetV1-0.25 network simplifies the structure based on MobileNetV1, that is, compresses its number of channels to 1 / 4 of the MobileNetV1 network, thereby greatly reducing the amount of calculation and parameters of the model with a small loss of accuracy, providing favorable conditions for deploying the model on various embedded devices.

[0065] In specific implementation, the MobileNetV1-0.25 network will perform feature extraction on the image to be identified under the conditions of three different receptive fields, thereby obtaining the first feature maps with three different numbers of channels, that is, the first feature extraction is performed on the image to be identified, and the receptive field is the size of the mapping area of ​​the pixels on the feature map on the original image (i.e., the image to be identified), that is, the larger the receptive field, the larger the range of the extracted original image, and the extracted image contains richer feature information; in this embodiment, three receptive fields are used for feature extraction, which is equivalent to extracting feature information in three dimensions, thereby ensuring the comprehensiveness of feature information extraction.

[0066] In this embodiment, sampling can be performed under the conditions of, but not limited to, 8 times (i.e., using a convolution operation to perform convolution processing on the image to be recognized, such as using a convolution kernel of size 2*2 for convolution operation), 16 times, and 32 times (feature sampling can be performed using, but not limited to, DW (deepthwise) convolution), so as to obtain the first feature map with 64 channels, 128 channels, and 256 channels (i.e., Figure 1 C3, C4 and C5 in the above figure), that is, outputting the three first feature maps to the FPN network for feature fusion to further improve the extracted feature information.

[0067] In specific implementation, the FPN network (Feature Pyramid Networks) can greatly improve the detection performance of objects without increasing the computational complexity of the model, that is, the number of channels of the first feature map output by the MobileNetV1-0.25 network is first changed, and then upsampling and feature fusion are performed in sequence to obtain a second feature map with richer feature information; optionally, the FPN network can, but is not limited to, use a 1*1 convolution kernel to perform convolution operations on the aforementioned first feature maps with 64 channels, 128 channels, and 256 channels, thereby adjusting the number of channels of the three to 64 channels, wherein the 1*1 convolution does not need to consider the relationship between pixels and surrounding pixels, and it can linearly combine pixels on different channels, and then perform nonlinear operations to complete the function of dimensionality reduction (that is, while reducing the number of channels, the width and height of the image are not changed, thereby achieving the purpose of reducing the computational complexity).

[0068] After the three first feature maps are channel-adjusted, upsampling (i.e., image enlargement) and feature fusion can be performed. In specific implementation, upsampling can be performed using, but not limited to, the nearest interpolation method, bilinear interpolation method, mean interpolation method, or median interpolation method. When performing feature fusion, the specific operations are as follows: Figure 1As shown, the first feature map C5 is used as the bottom feature map, and after passing through the FPN network, it is directly used as the second feature map P5; then the first feature map C5 is upsampled and feature fused with the second feature map C4 to obtain the second feature map P4; finally, the second feature map P4 is upsampled and feature fused with the first feature map C3 to obtain the second feature map P3, thereby obtaining three second feature maps, thereby realizing the fusion of features in three dimensions, which not only avoids the loss of feature information, but also obtains richer feature information, thereby increasing the ability to locate the license plate in the image.

[0069] In this embodiment, channel fusion is used to implement the feature fusion of the three first feature maps. Channel fusion is also called concat channel fusion, which is to merge the number of channels of the convolutional layer, and is a commonly used feature fusion method in various neural network models.

[0070] After obtaining the three second feature maps, they can be input into the SSH network (which is an improvement based on the Visual Geometry Group Network (VGG-16 network)) for convolution processing to obtain features with more detailed information; the SSH network is equipped with three parallel convolution layers, that is, the three second feature maps can be convolved separately using three parallel convolution layers to achieve multi-scale feature detection, thereby strengthening feature extraction and improving the contextual information of the feature map to improve the quality of the extracted features and the accuracy of corner location.

[0071] See also Figure 3 As shown in the figure, the three parallel convolutional layers in the SSH network are: one 3*3 convolution on the left, two 3*3 convolutions in the middle, and three 3*3 convolutions on the right. Figure 3 Conc2d ks=3filters=32 in it means that a 3*3 convolution kernel is used and conv2d convolution is performed to obtain a feature map with 32 channels. In specific implementation, the three second feature maps are respectively input into three parallel convolution layers to obtain three third feature maps.

[0072] After the features of the above three networks are extracted, the head network can be used to predict the key points of the license plate, that is, to detect the prior box (also called anchor box or detection box). The head network is a network that obtains the network output content. It can use the features extracted by the above three networks to make predictions, thereby obtaining prediction results.

[0073] In this embodiment, the head network predicts the key points of the license plate, including the following three aspects: (1) classification prediction, which is used to determine whether the prior frame contains the target (i.e., the license plate); (2) regression of the prior frame, which is used to adjust the prior frame to obtain the predicted frame, requiring four parameters, the first two of which are used to adjust the center of the prior frame, and the last two are used to adjust the width and height of the prior frame. The adjustment of the prior frame can be achieved by, but is not limited to, a 1*1 convolution; (3) regression of the key points of the license plate, which is used to adjust the predicted frame within the prior frame to obtain the coordinates of the key points of the license plate, requiring 8 parameters. Each key point of the license plate requires two adjustment parameters (i.e., horizontal and vertical coordinates), i.e., adjusting the x and y axes of the center of the prediction frame to obtain the coordinates of the key points of the license plate, i.e., outputting an 8-dimensional key point vector, and matching the 8-dimensional key point vector to obtain 4 license plate key point coordinates (i.e., 4 license plate corner point coordinates), i.e., each license plate corner point coordinate is represented by a two-dimensional vector of (x, y). Of course, the final output result is represented by coordinates.

[0074] At the same time, in the specific implementation, this embodiment further optimizes the model parameters of the aforementioned license plate positioning model, including: adding new model constraints on corner points, improving the size of the prior frame and the weights of each constraint.

[0075] First, this embodiment sets the size of the prior anchor of the license plate positioning model to: 16*8 (length*width), 32*16, 64*32, 128*64, 256*128 and 512*256, wherein the prior anchor is used to characterize the anchor point frame of the license plate. Through the above design, the anchor point frame can be closer to the actual size of the license plate, thereby further improving the corner point positioning accuracy.

[0076] Secondly, on the basis of the original constraints of the Retinaface detection network, this embodiment adds a corner point distance constraint and a corner point distance ratio constraint. The reason is that the traditional Retinaface detection network is a face recognition network, which only performs key point regression supervision, and has the following disadvantages: there is a relative position between the key points. For a human face, the key points corresponding to the left and right eyes are on the top, the key points corresponding to the nose are in the middle, and the key points corresponding to the left and right corners of the mouth are on the bottom. However, the traditional Retinaface detection network does not supervise the position of the key points, which will cause a small probability of misalignment error in the position of the key points finally identified, thereby affecting the positioning accuracy. This disadvantage has a great impact when positioning the license plate, which will directly lead to license plate positioning and recognition errors. Therefore, this embodiment adds the above two key point constraints to constrain the key point position. At the same time, since the appearance of the license plate is a standard rectangle in geometry, it is a standard parallelogram even when tilted. Therefore, this constraint can also be applied when tilted.

[0077] Among them, the corner distance constraint function is used to constrain the distance between the two parallel sides of the license plate to be identified (that is, to ensure that the distance between the two parallel sides is equal), and the corner distance ratio constraint function is used to constrain the ratio of the length and width of the license plate to be identified, that is, the ratio of the length and width of the identified license plate should be equal to the ratio of the actual length and width of the license plate. The function for setting the specific constraint conditions (that is, the loss function of the entire model) is as follows:

[0078]

[0079] In the above formula, L represents the loss function of the license plate positioning model, L cls represents the target classification loss function, L box represents the detection box regression loss function, L pts represents the key point regression loss function, p i represents the probability that anchor point i (the smallest unit point on the feature map) is predicted as a license plate corner point, When it is 1, it indicates the true label of the positive anchor point. When it is 0, it indicates the true label of the negative anchor point, t i represents the predicted coordinates of the prediction box in the anchor box, represents the true coordinates of the predicted box in the anchor box, l i Represents the predicted license plate corner coordinates in the anchor box, represents the coordinates of the corner points of the license plate annotation, λ1, λ2, λ3 and λ4 represent weight coefficients, which are constants.

[0080] L pts1 Represents the corner point distance constraint function, which is used to constrain the distance between the two parallel sides of the license plate to be recognized.

[0081] L pts2 Represents the corner point distance ratio constraint function, which is used to constrain the ratio of the length and width of the license plate to be recognized.

[0082] In (1), L pts1 The specific function expression is:

[0083] The corner point distance constraint function is:

[0084] L pts1 (l i )=|||l0l1|-|l2l3|||+||l0l2|-|l1l3||| (2)

[0085] In the above formula (2), l0l1 represents the distance between the first license plate corner point and the second license plate corner point, l2l3 represents the distance between the third license plate corner point and the fourth license plate corner point, l0l2 represents the distance between the first license plate corner point and the third license plate corner point, l1l3 represents the distance between the second license plate corner point and the fourth license plate corner point, where, see Figure 4 As shown, the upper left corner of the license plate to be identified is the first license plate corner point (denoted as p0), the upper right corner is the second license plate corner point (denoted as p1), the lower left corner is the third license plate corner point (denoted as p2), and the lower right corner is the fourth license plate corner point (denoted as p3).

[0086] pass Figure 4 It can be seen that the distance between the first license plate corner point and the second license plate corner point, and the distance between the third license plate corner point and the fourth license plate corner point are the two length sides of the license plate to be identified, and the distance between the first license plate corner point and the third license plate corner point, and the distance between the second license plate corner point and the fourth license plate corner point are the two width sides of the license plate to be identified; therefore, the constraint expressed by the above formula (2) means that the distance between the two parallel sides should be equal, that is, L pts1 The final value should be constrained to 0 or close to 0, so that the identified corner points are the most accurate.

[0087] Similarly, the corner point distance ratio constraint function is:

[0088]

[0089] In the above formula (3), Indicates the distance between the first license plate annotation corner point and the second license plate annotation corner point. It represents the distance between the first license plate marked corner point and the third license plate marked corner point, where the license plate marked corner points represent the actual corner point coordinates of the license plate, which are measured in advance by the staff.

[0090] Therefore, the constraint meaning of the above formula (3) is that the ratio of the length to the width of the license plate obtained based on the coordinates of the four license plate corner points predicted by the model should be equal to the ratio of the actual length to the width of the license plate, that is, the value of the corner point distance ratio constraint function should ultimately be constrained to 0 or close to 0.

[0091] In this embodiment, the distance is obtained according to the coordinates of the corner points of the license plate, which can be obtained by using, but not limited to, a coordinate distance formula, which will not be elaborated herein.

[0092] Therefore, through the aforementioned improvement of the model constraints, the regression accuracy of the four corner points of the license plate can be improved and the risk of corner point errors can be reduced.

[0093] Finally, in the traditional Retinaface detection network, λ1=0.25, λ2=0.1, and the weight of the key point regression loss function is relatively low. In the license plate positioning detection, the license plate corner point is more important than the license plate position frame. Therefore, this embodiment sets λ2=0.5, and sets the weight coefficients of the newly added constraint items λ3=0.01, λ4=0.01; thereby, the corner point regression can be jointly constrained to further improve the positioning accuracy of the license plate corner points.

[0094] Therefore, through the above detailed description of the license plate positioning model, the present invention has the following advantages:

[0095] (1) The present invention adopts a lightweight MobileNetV1-0.25 network as the backbone network, which can not only realize the license plate corner point positioning with almost no loss of accuracy, but also reduce the calculation amount and parameter amount of the model, providing favorable conditions for the terminal to deploy the license plate recognition system.

[0096] (2) The output of the key point branch is improved from the traditional 10-dimensional vector to an 8-dimensional vector, which can directly obtain the four license plate corner points, realizing end-to-end license plate positioning and avoiding the problem of error amplification in step-by-step positioning of target detection.

[0097] (3) By improving the size of the prior frame, adding corner point distance constraints, adding corner point distance ratio constraints, and increasing the weight of the key point regression loss function, the positioning accuracy of the license plate corner points can be significantly improved, further improving the license plate recognition accuracy.

[0098] See also Figure 6 As shown, the second aspect of this embodiment provides a license plate recognition method based on the coordinates of the license plate corner points on the basis of the first aspect of the first embodiment, as shown in the following steps S201 to S205.

[0099] S201. Obtain an image to be identified, wherein the image to be identified includes at least one license plate to be identified.

[0100] S202. Input the image to be identified into a license plate positioning model for positioning detection to obtain the coordinates of four license plate corner points corresponding to the license plate to be identified.

[0101] S203. Obtain the coordinates of the standard corner points of the license plate in the image to be recognized.

[0102] S204. Perform perspective transformation processing on the image to be identified using the standard corner point coordinates of the license plate and the four corner point coordinates of the license plate to obtain a corrected image to be identified, wherein the corrected image to be identified only contains the license plate to be identified.

[0103] S205. Input the corrected image to be recognized into a license plate recognition model to obtain a license plate recognition result.

[0104] The license plate recognition principle of the above steps is:

[0105] Use the getPerspectiveTransform function in opencv (a cross-platform computer vision and machine learning software library released under the Apache2.0 license (open source)) and build the perspective transformation matrix M based on the coordinates of the four license plate corner points and the four standard license plate corner points. Then use the warpPerspective function in opencv, pass in the image to be recognized and the matrix M, and you can get the corrected image to be recognized.

[0106] In specific implementation, the coordinates of the four standard corner points of the license plate can be, but are not limited to, preset by the user according to the size of the license plate to be transformed. For example, if the license plate needs to be transformed into a 94*24 size, then the coordinates of the four standard corner points of the license plate are: p0(0,0), p1(94,0), p2(24,0), p3(94,24). At the same time, the LPRnet (License Plate Recognition via Deep Neural Networks) license plate recognition network can be used for license plate recognition.

[0107] See also Figure 5 As shown, Figure 5 This is a schematic diagram of the license plate positioning and recognition effect using the positioning model and recognition method provided in this embodiment, from Figure 5 It can be clearly seen that end-to-end license plate positioning can be achieved, and tilt correction can be performed to obtain an image containing only the license plate to be identified, which significantly improves the positioning and recognition accuracy of the license plate.

[0108] like Figure 7 As shown, the third aspect of this embodiment provides a hardware device for implementing the license plate positioning method for license plate recognition described in the first aspect of the embodiment, including:

[0109] The acquisition unit is used to acquire the image to be recognized, wherein the image to be recognized contains at least one license plate to be recognized.

[0110] The license plate positioning unit is used to input the image to be identified into the license plate positioning model for positioning detection, and obtain the coordinates of the four license plate corner points corresponding to the license plate to be identified, so as to complete the license plate positioning in the image to be identified according to the four license plate corner point coordinates, wherein the license plate positioning model includes MobileNetV1-0.25 network, FPN network, SSH network and head network.

[0111] The working process, working details and technical effects of the hardware device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0112] like Figure 8 As shown, the fourth aspect of this embodiment provides another license plate positioning device for license plate recognition, taking the device as a computer main device as an example, comprising: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the license plate positioning method for license plate recognition as described in the first aspect of the embodiment.

[0113] For example, the memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. Specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). At the same time, the processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state.

[0114] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may not be limited to a microprocessor of the STM32F105 series, a reduced instruction set computer (RSIC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a general packet radio service technology (General Packet Radio Service, GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver and / or a 5G transceiver, etc. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0115] The working process, working details and technical effects of the computer main device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0116] The fifth aspect of this embodiment provides a storage medium that stores instructions for the license plate positioning method for license plate recognition described in the first aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the license plate positioning method for license plate recognition described in the first aspect is executed.

[0117] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, a CD, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0118] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0119] The sixth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the license plate positioning method for license plate recognition as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0120] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A license plate positioning method for license plate recognition, characterized in that: include: Acquire an image to be identified, wherein the image to be identified includes at least one license plate to be identified; Input the image to be identified into the license plate positioning model for positioning detection, and obtain the coordinates of the four license plate corner points corresponding to the license plate to be identified, so as to complete the license plate positioning in the image to be identified according to the four license plate corner point coordinates, wherein the license plate positioning model includes MobileNetV1-0.25 network, FPN network, SSH network and head network; The MobileNetV1-0.25 network is used as the backbone feature extraction network of the license plate positioning model, and is used to extract features of the image to be identified under three receptive field conditions to obtain three first feature maps; The FPN network is used to upsample and fuse the three first feature maps to obtain three second feature maps; The SSH network includes three parallel convolutional layers, which are used to perform convolution processing on the three second feature maps using the three parallel convolutional layers to obtain three third feature maps; The head network is used to predict the key points of the license plate on the three third feature maps to output an 8-dimensional key point vector so as to obtain the coordinates of the four license plate corner points according to the key point vector.

2. The method according to claim 1, characterized in that The loss function of the license plate positioning model is: In the above formula, L represents the loss function of the license plate positioning model, L cls represents the target classification loss function, L box represents the detection box regression loss function, L pts represents the key point regression loss function, p i represents the probability that anchor point i is predicted as a license plate corner point, When it is 1, it indicates the true label of the positive anchor point. When it is 0, it indicates the true label of the negative anchor point, t i represents the predicted coordinates of the prediction box in the anchor box, represents the true coordinates of the predicted box in the anchor box, l i Represents the predicted license plate corner coordinates in the anchor box, represents the coordinates of the corner points of the license plate, λ1, λ2, λ3 and λ4 represent weight coefficients, which are constants; L pts1 Represents the corner point distance constraint function, which is used to constrain the distance between the two parallel sides of the license plate to be identified; L pts2 Represents the corner point distance ratio constraint function, which is used to constrain the ratio of the length and width of the license plate to be recognized.

3. The method according to claim 2, characterized in that The corner point distance constraint function is: L pts1 (l i )＝|||l0l1|-|l2l3|||+|||l0l2|-|l1l3||| In the above formula, l0l1 represents the distance between the first license plate corner point and the second license plate corner point, l2l3 represents the distance between the third license plate corner point and the fourth license plate corner point, l0l2 represents the distance between the first license plate corner point and the third license plate corner point, l1l3 represents the distance between the second license plate corner point and the fourth license plate corner point, wherein the upper left corner of the license plate to be identified is the first license plate corner point, the upper right corner is the second license plate corner point, the lower left corner is the third license plate corner point, and the lower right corner is the fourth license plate corner point.

4. The method according to claim 3, characterized in that The corner point distance ratio constraint function is: In the above formula, Indicates the distance between the first license plate annotation corner point and the second license plate annotation corner point. Indicates the distance between the first license plate annotation corner point and the third license plate annotation corner point.

5. The method according to claim 2, characterized in that The values ​​of weight coefficients λ1, λ2, λ3 and λ4 are 0.25, 0.5, 0.01 and 0.01 respectively.

6. The method according to claim 1, characterized in that The sizes of the prior anchors of the license plate positioning model are: 16*8, 32*16, 64*32, 128*64, 256*128 and 512*256, wherein the prior anchors are used to characterize the anchor point frame of the license plate positioning.

7. The method according to claim 1, characterized in that After the image to be identified is input into the license plate positioning model for positioning detection and the coordinates of four license plate corner points corresponding to the license plate to be identified are obtained, the method further includes: Obtaining the coordinates of the standard corner points of the license plate in the image to be recognized; Performing perspective transformation processing on the image to be identified using the standard corner point coordinates of the license plate and the four corner point coordinates of the license plate to obtain a corrected image to be identified, wherein the corrected image to be identified only contains the license plate to be identified; The corrected image to be recognized is input into the license plate recognition model to obtain a license plate recognition result.

8. A license plate positioning device for license plate recognition, characterized in that: Used to execute the license plate positioning method for license plate recognition according to any one of claims 1 to 7, wherein the license plate positioning device comprises: An acquisition unit, used for acquiring an image to be recognized, wherein the image to be recognized contains at least one license plate to be recognized; The license plate positioning unit is used to input the image to be identified into the license plate positioning model for positioning detection, and obtain the coordinates of the four license plate corner points corresponding to the license plate to be identified, so as to complete the license plate positioning in the image to be identified according to the four license plate corner point coordinates, wherein the license plate positioning model includes MobileNetV1-0.25 network, FPN network, SSH network and head network.

9. A license plate positioning device for license plate recognition, characterized in that: include: A memory, a processor and a transceiver which are sequentially communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program to execute the license plate positioning method for license plate recognition as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed on a computer, the license plate positioning method for license plate recognition according to any one of claims 1 to 7 is executed.

Images