A method, device, equipment and medium for detecting corner points of license plates

Abstract

The present invention provides a method, device, computer equipment and computer-readable storage medium for license plate corner point detection, including: obtaining a license plate detection data set, and obtaining a training set, a test set and an evaluation set based on the license plate detection data set; obtaining a feature enhancement module A and a feature enhancement module B; obtaining a plurality of first feature vectors with different semantics and resolutions based on the training set; obtaining a first loss function based on the first feature vector and the feature enhancement module A; obtaining a second loss function based on the first feature vector and the feature enhancement module B; calculating a total loss function based on the first loss function and the second loss function; obtaining a training model based on the test set and the total loss function; and obtaining license plate corner point coordinates and evaluation indicators based on the evaluation set, the feature enhancement module A and the training model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for detecting license plate corner points, a device for detecting license plate corner points, a computer device and a computer-readable storage medium. Background Art

[0002] In the existing technology, there are mainly two technical routes: (1) Single character recognition. First, there is a sensor sensing area on the ground. When a vehicle passes by, it automatically takes a photo. Then, the photo is preprocessed to become a grayscale image, remove noise, etc., remove some interference factors, and reduce the size of the image to facilitate edge extraction in the later stage; secondly, the edge extraction technology is used to extract the image contour, and most of the unsuspected ones are removed according to the aspect ratio of the license plate of about 1:2.5; then, the image grayscale value is projected in the horizontal and vertical directions to cut out the characters; finally, template matching or a trained deep learning model is used for recognition, and each character is recognized separately and then connected in series to obtain the entire license plate number. The bottleneck of this method is how to accurately cut out each character on the license plate, especially for tilted license plates. When cutting, the characters will be divided into two or even multiple parts, which will seriously affect the effect of license plate recognition and greatly increase the probability of misrecognition. (2) Entire license plate recognition. First, use the sensor to sense the vehicle and take a photo of the entire vehicle; second, use the trained license plate detection algorithm to detect the license plate (detection frame and four corner points). To improve the accuracy of license plate recognition, image preprocessing technology can be used to preprocess the image and reduce interference. Finally, use the trained deep learning model to recognize the entire license plate. The bottleneck of this method is whether the four corner points of the license plate detection can be accurately predicted in the tilted license plate, which in turn affects the subsequent recognition effect of the entire license plate.

[0003] For the whole license plate recognition, firstly, the sensor is used to sense the vehicle, take a picture of the front part of the license plate, and then obtain a picture containing the complete license plate; secondly, Figure 1 As shown in the figure, the license plate detection module is trained, which mainly includes feature extraction, feature enhancement and network output layer (license plate rectangular frame coordinates, category probability of the detected target being a license plate and four corner point coordinates); then the original image is corrected according to the license plate correction module, and a fixed-size image containing only the license plate is output; finally, the license plate recognition module is trained to recognize the corrected image and output the recognition result.

[0004] Due to the limitations of memory and computing power on low-power chips, only networks with small computational load and small weight parameters can be used, which will greatly reduce the performance of the existing license plate detection module, and thus affect the performance of license plate recognition, especially in the recognition of license plates with large angles. The detection effect of the four corner point coordinates directly affects the performance of the license plate correction module and the license plate recognition module. Therefore, improving the license plate detection module on low-power chips is crucial to the overall license plate recognition performance.

[0005] Therefore, it is necessary to provide a method for license plate corner point detection, an apparatus for license plate corner point detection, a computer device and a computer-readable storage medium to effectively solve the above problems. Summary of the invention

[0006] The present invention provides a method for detecting corner points of a license plate, a device for detecting corner points of a license plate, computer equipment and a computer-readable storage medium.

[0007] An embodiment of the present invention provides a method for detecting corner points of a license plate, comprising:

[0008] Obtain a license plate detection data set, and obtain a training set, a test set, and an evaluation set based on the license plate detection data set;

[0009] Acquire feature enhancement module A and feature enhancement module B;

[0010] Based on the training set, a plurality of first feature vectors with different semantics and resolutions are obtained;

[0011] Obtaining a first loss function based on the first feature vector and the feature enhancement module A;

[0012] Obtaining a second loss function based on the first feature vector and the feature enhancement module B;

[0013] Calculate a total loss function based on the first loss function and the second loss function;

[0014] Based on the test set and the total loss function, obtaining a training model;

[0015] Based on the evaluation set, the feature enhancement module A and the training model, the license plate corner point coordinates and evaluation indicators are obtained.

[0016] Preferably, obtaining a plurality of first feature vectors with different semantics and resolutions based on the training set includes:

[0017] Preprocessing the training set to obtain a first network input picture;

[0018] Perform feature extraction on the first network input picture to obtain the first feature vector.

[0019] Preferably, obtaining a first loss function based on the first feature vector and the feature enhancement module A includes:

[0020] Using the feature enhancement module A to perform feature fusion on each of the first feature vectors to obtain a first fused feature;

[0021] Based on the first fusion feature, a first detection frame feature map, a first classification feature map and a first corner point feature map are obtained through a Detection Head module;

[0022] Based on the first detection box feature map, the first classification feature map and the first corner point feature map, respectively obtaining a detection box loss, a category loss and a corner point loss;

[0023] The first loss function is calculated based on the detection box loss, the category loss and the corner point loss.

[0024] Preferably, obtaining the second loss function based on the first feature vector and the feature enhancement module B includes:

[0025] Using the feature enhancement module B to perform feature fusion on each of the first feature vectors to obtain a second fused feature;

[0026] Based on the second fusion feature, a probability map, a threshold map and a binary map are obtained through a Seg Head module;

[0027] Based on the probability map, the threshold map and the binary map, respectively obtain a probability map loss, a threshold map loss and a binary map loss;

[0028] The second loss function is calculated based on the probability map loss, the threshold map loss, and the binary map loss.

[0029] Preferably, obtaining the license plate corner point coordinates and evaluation index based on the evaluation set, the feature enhancement module A and the training model includes:

[0030] Based on the evaluation set, a plurality of second feature vectors with different semantics and resolutions are obtained;

[0031] Using the feature enhancement module A to perform feature fusion on the second feature vector to obtain a second fused feature;

[0032] Based on the second fusion feature, a second detection frame feature map, a second classification feature map and a second corner point feature map are obtained through the Detection Head module;

[0033] Based on the training model, the second detection frame feature map, the second classification feature map and the second corner point feature map are decoded and maximum value suppressed to obtain the license plate corner point coordinates;

[0034] The evaluation index is obtained based on the true label of the evaluation set and the model weight parameter.

[0035] Preferably, obtaining a plurality of second feature vectors with different semantics and resolutions based on the evaluation set includes:

[0036] Preprocessing the evaluation set to obtain a second network input picture;

[0037] Perform feature extraction on the second network input picture to obtain the second feature vector.

[0038] Preferably, a MobileNet0.25 lightweight backbone network is used to perform the feature extraction.

[0039] The present application also provides a device for detecting license plate corner points, comprising:

[0040] An acquisition module is used to acquire a license plate detection data set, and obtain a training set, a test set and an evaluation set based on the license plate detection data set; and acquire a feature enhancement module A and a feature enhancement module B;

[0041] A feature extraction module, used to obtain a plurality of first feature vectors with different semantics and resolutions based on the training set;

[0042] A first branch processing module, used for obtaining a first loss function based on the first feature vector and the feature enhancement module A;

[0043] A second branch processing module, used for obtaining a second loss function based on the first feature vector and the feature enhancement module B;

[0044] A model processing module, configured to calculate a total loss function based on the first loss function and the second loss function; and obtain a training model based on the test set and the total loss function;

[0045] The inference module is used to obtain the coordinates of the corner points of the license plate and the evaluation index based on the backbone network, the evaluation set, the feature enhancement module A and the training model.

[0046] Furthermore, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-mentioned methods when executing the computer program.

[0047] Furthermore, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0048] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0049] The method for license plate corner point detection, the device for license plate corner point detection, the computer equipment and the computer-readable storage medium provided by the embodiments of the present invention introduce a new bypass supervision branch in the training stage of the license plate detection module to supervise the main branch of corner point detection and improve the accuracy of corner point detection, but remove the bypass supervision branch in the reasoning stage of the license plate detection module. In this way, the accuracy of the coordinates of the four corner points of the license plate tilted at a large angle is greatly increased without increasing the weight parameter size and calculation amount of the backbone network, and accurate corner point coordinates are provided for the subsequent license plate correction module, thereby improving the recognition performance of the license plate recognition network and greatly reducing missed characters and misrecognition of characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention, not all embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0051] Figure 1 A schematic diagram of a license plate corner point recognition process provided by the prior art;

[0052] Figure 2 A flowchart of a method for detecting license plate corner points provided by an embodiment of the present invention;

[0053] Figure 3 A flowchart of a method for detecting license plate corner points provided by an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a general multi-scale feature fusion module FPN provided by the prior art;

[0055] Figure 5 A schematic diagram of a feature enhancement module A for improved multi-scale feature fusion provided in some embodiments of the present application;

[0056] Figure 6 A schematic diagram of the SSH Module structure provided for some embodiments of the present application;

[0057] Figure 7A schematic diagram of the structure of a feature enhancement module B provided for some embodiments of the present application;

[0058] Figure 8 A schematic diagram of the structure of a Seg Head module provided for some embodiments of the present application;

[0059] Fig. 9 A schematic diagram of a probability graph provided for some embodiments of the present application;

[0060] Fig.10 A schematic diagram of a threshold map provided for some embodiments of the present application;

[0061] Fig.11 A schematic diagram of a binary image provided for some embodiments of the present application;

[0062] Fig.12 A schematic diagram of an original license plate image provided for some embodiments of the present application;

[0063] Fig.13 A schematic diagram of a resized license plate image provided in some embodiments of the present application;

[0064] Fig.14 A schematic diagram of a license plate image after resizing and left and right padding provided in some embodiments of the present application;

[0065] Fig.15 A schematic diagram of the performance of a license plate detection algorithm provided for some embodiments of the present application;

[0066] Fig.16 A schematic diagram of the performance of a license plate detection and license plate recognition algorithm provided for some embodiments of the present application;

[0067] Figures 17a-17d A schematic diagram of license plate reasoning results in the reasoning stage provided by multiple embodiments of the present application;

[0068] Fig.18 A module schematic diagram of a device for detecting license plate corner points provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0071] Based on the problems existing in the prior art, the embodiments of the present invention provide a method for detecting license plate corner points, an apparatus for detecting license plate corner points, a computer device and a computer-readable storage medium.

[0072] Figure 2 A flowchart of a method for detecting license plate corner points is provided for one embodiment of the present invention. Figure 3 A flowchart of a method for detecting license plate corner points is provided in accordance with an embodiment of the present invention. Figure 2 and 3 As shown, an embodiment of the present invention provides a method for detecting corner points of a license plate, comprising:

[0073] Step S11, obtaining a license plate detection data set, and obtaining a training set, a test set and an evaluation set based on the license plate detection data set.

[0074] In the specific implementation, license plate images can be obtained through cameras, photoelectric detection equipment, etc. The position and angle of the license plate images can be different, so that multiple different license plate images are combined into a license plate detection data set. Then, the license plate detection data set is cleaned and divided into a training set, a test set, and an evaluation set.

[0075] The license plate corner point detection method includes a training phase and an inference phase. The training phase includes the following steps S13 to S17, and the inference phase may include the following step S18.

[0076] The training phase includes a training set and a test set. Specifically, it means: the training set is used for model training. During the training process, the values ​​of the evaluation indicators are calculated multiple times in the test set. For example, the evaluation indicators are calculated every 5 training rounds (epochs). Each time, the training model of the epoch with the best evaluation indicator and its corresponding model weight parameters are saved for model inference in the inference phase. The inference phase uses the evaluation set for inference, so as to better evaluate the generalization ability of the model.

[0077] Step S12, obtaining feature enhancement module A and feature enhancement module B.

[0078] In the specific implementation, feature enhancement module A and feature enhancement module B are different feature fusion modules, which are used in the subsequent training stage and inference stage to perform feature fusion calculation on feature vectors of different resolutions extracted by the backbone network.

[0079] Step S13, obtaining a plurality of first feature vectors with different semantics and resolutions based on the training set.

[0080] In some embodiments, step S13 includes:

[0081] Step S131, preprocessing the training set to obtain a first network input image; and step S132, performing feature extraction on the first network input image to obtain a first feature vector.

[0082] In a specific implementation, preprocessing the training set may include resizing and standardizing the license plate images in the training set to obtain the first network input image.

[0083] Then, for example, an improved MobileNet0.25 lightweight backbone network can be used to extract features from the first network input image to obtain a plurality of first feature vectors with different semantics and resolutions. Then, two branches can be calculated for each first feature vector. Branch 1 includes step S14, and branch 2 includes step S15.

[0084] Step S14, obtaining a first loss function based on the first feature vector and the feature enhancement module A.

[0085] In some embodiments, step S14 includes:

[0086] Step S141: Use the feature enhancement module A to perform feature fusion on each first feature vector to obtain a first fused feature.

[0087] Step S142, based on the first fusion feature, a first detection box feature map, a first classification feature map and a first corner point feature map are obtained through the Detection Head module.

[0088] Step S143, based on the first detection box feature map, the first classification feature map and the first corner point feature map, obtain the detection box loss, the category loss and the corner point loss respectively.

[0089] Step S144, calculating a first loss function based on the detection box loss, the category loss and the corner point loss.

[0090] In the specific implementation, the feature enhancement module A uses an improved version of the general multi-scale feature fusion module FPN, which only uses feature maps of three scales (H / 8×W / 8, H / 16×W / 16, H / 32×W / 32). The detection box feature map, classification feature map and corner feature map can refer to the corresponding feature vectors respectively. Figure 4 A schematic diagram of a general multi-scale feature fusion module FPN provided in the prior art. Figure 5A schematic diagram of a feature enhancement module A for an improved multi-scale feature fusion provided for some embodiments of the present application. Figure 6 A schematic diagram of the SSH Module structure provided for some embodiments of the present application. Figures 4 to 6 There are three main improvements in the feature enhancement module A for improved multi-scale feature fusion. First, only three layers of feature maps are selected, with the aim of further reducing network parameters and network calculations, so that it can run at a higher frame rate (above 15FPS) on low-power chips. Second, the resizing method in the upsampling stage (upsample) is changed from the nearest neighbor to the bilinear difference (bilinear difference) method, with the aim of improving the model effect. Third, a convolution with a convolution kernel of 3×3 is replaced with an SSH Module module. The SSH Module is used for feature fusion of contextual information, which can better predict targets of different scales, thereby improving the model's detection effect on targets of different sizes. Branch 1 uses the feature enhancement module A to perform multi-scale feature fusion on the first feature vectors with different semantics and resolutions.

[0091] Then, the first fused feature is passed through the convolution layer of the detection head module to obtain the first detection frame feature map, the first classification feature map and the first corner point feature map. The first corner point feature map contains the four corner points of the license plate. The first loss function of branch 1 is calculated as shown in formula (1).

[0092] (1)

[0093] Among them, Loss_box is the detection box loss, Loss_cls is the category loss, Loss_landmark is the corner point loss, and Loss1 is the first loss function.

[0094] Step S15, obtaining a second loss function based on the first feature vector and feature enhancement module B.

[0095] In some embodiments, step S15 includes:

[0096] Step S151, using the feature enhancement module B to perform feature fusion on each first feature vector to obtain a second fused feature.

[0097] Step S152, based on the second fusion feature, a probability map, a threshold map and a binary map are obtained through a Seg Head module.

[0098] Step S153, based on the probability map, the threshold map and the binary map, respectively obtain the probability map loss, the threshold map loss and the binary map loss.

[0099] Step S154, calculating a second loss function based on the probability map loss, the threshold map loss and the binary map loss.

[0100] In the specific implementation, Figure 7 Schematic diagram of the structure of feature enhancement module B. Figure 8 This is a schematic diagram of the structure of the Seg Head module. Fig. 9 A schematic diagram of the probability graph. Fig.10 Schematic diagram of the threshold map. Fig.11 This is a schematic diagram of a binary image. Figures 7 to 11 , branch 2 uses the feature enhancement module B to fuse the second feature vectors with different semantics and resolutions, and then passes the fused second fusion features through the segmentation head (Seg Head) module to obtain the probability map, threshold map and binary map. The second loss function of branch 2 is calculated as shown in formula (2).

[0101] (2)

[0102] Among them, Loss_s is the probability map loss (specifically Binary Cross Entropy Loss), Loss_t is the threshold map loss (specifically L1 distance), and Loss_b is the binary map loss (specifically Binary Cross Entropy Loss).

[0103] Among them, Fig. 9 , the probability map represents the probability of each pixel belonging to the text region. The standard binary cross entropy loss function is used to supervise the training process of the probability map. Specifically, for each pixel, if it belongs to the text region, its label is set to 1, otherwise it is set to 0. Then the cross entropy loss between the probability map output by the network and the true label is calculated.

[0104] like Fig.10 , the threshold map represents the binarization threshold of each pixel. In order to enable the network to adaptively learn a suitable binarization threshold, the threshold map loss is introduced. Specifically, for each pixel, the L1 loss between its true threshold and the threshold output by the network is calculated.

[0105] like Fig.11 The binary map is obtained by comparing the probability map and the threshold map pixel by pixel. In order to enable the network to better learn the binary rules during the training process, the binary map loss is introduced. Specifically, for each pixel, its true label is compared with the binary result output by the network pixel by pixel, and the difference between them is calculated.

[0106] Step S16, calculating a total loss function based on the first loss function and the second loss function.

[0107] In a specific implementation, the total loss function calculation formula is as shown in formula (3).

[0108] (3)

[0109] Among them, Loss is the total loss function, Loss2 is the first loss function, and Loss2 is the second loss function.

[0110] Step S17, obtaining a training model based on the test set and the total loss function.

[0111] In the specific implementation, after each round of training, the mAP index is calculated on the test set, and the model weight parameters of the training model are saved according to the mAP index. mAP (mean Average Precision) is a common evaluation index for target detection - average precision. The calculation of mAP is roughly broken down into the following steps:

[0112] 1. Output model detection results. First, you need to complete target decoding and generate a standardized target list, including the category of each target, the coordinates of the bounding box, and the confidence level.

[0113] 2. For a single target, first group the GT (Ground Truth) and Predictions according to their respective categories. In each category, match the two sets of data, GT and Predictions, based on IoU. The matching results may be as follows: (1) TP (True Positive): a correct detection, IoU >= threshold. That is, the predicted target classification is correct, and the bounding box and GT overlap sufficiently. (2) FP (False Positive): an incorrect detection, the target does not match the GT or IoU < threshold. That is, the predicted target classification is incorrect, or the bounding box and GT overlap sufficiently. (3) FN (False Negative): a missed GT, that is, a Ground Truth that is not matched to the target.

[0114] 3. After completing the judgment of each target in the entire test set, the Precision and Recall are statistically divided into categories. (1) Precision, the ability of the model to find only the correct target. Precision = TP / (TP + FP), where TP + FP are all the targets found by the model. (2) Recall, the ability of the model to find all targets. Recall = TP / (TP + FN), where TP + FN are all GTs. When different confidence thresholds are set, the number of detected targets output is different, and the Precision and Recall calculated are also different. According to different thresholds, a series of Precision and Recall values ​​can be obtained, and the PR curve can be obtained by connecting them.

[0115] 4. So far, a PR curve is calculated for each category. The area enclosed by the PR curve and the X-axis is the AP value of the current category. The mean of the AP values ​​of all categories is mAP. In the present invention, there is only one category.

[0116] Step S18, based on the backbone network, the evaluation set, the feature enhancement module A and the training model, the license plate corner point coordinates and the evaluation index are obtained.

[0117] In some embodiments, step S18 includes:

[0118] Step S181, obtaining a plurality of second feature vectors with different semantics and resolutions based on the evaluation set.

[0119] In some embodiments, the evaluation set includes corresponding license plate images and true labels. Step S181 includes: preprocessing the evaluation set to obtain a second network input image; and extracting features from the second network input image to obtain a second feature vector.

[0120] In a specific implementation, preprocessing the evaluation set may include resizing and standardizing the license plate images in the evaluation set to obtain the second network input image.

[0121] Then, for example, an improved MobileNet0.25 lightweight backbone network may be used to perform feature extraction on the second network input image to obtain multiple second feature vectors with different semantics and resolutions.

[0122] Step S182: Use feature enhancement module A to perform feature fusion on the second feature vector to obtain a second fused feature.

[0123] Step S183, based on the second fusion feature, a second detection box feature map, a second classification feature map and a second corner point feature map are obtained through the Detection Head module.

[0124] Step S184, based on the training model, the second detection frame feature map, the second classification feature map and the second corner point feature map are decoded and maximum value suppressed to obtain the license plate corner point coordinates.

[0125] In a specific implementation, the feature enhancement module A is used to perform feature fusion on the second feature vector to obtain the second fused feature, and then the fused second fused feature is passed through the Detection Head convolution layer to obtain the second detection frame feature map, the second classification feature map and the second corner point feature map (including the four corner points of the license plate), and the feature map is decoded (to obtain the point coordinates on the original image) and the maximum value is suppressed using the training model including but not limited to the model weight parameters to obtain the model inference results of the coordinates of the four corner points of the final license plate, and then the evaluation index is calculated according to the real label of the evaluation set and the model weight parameters. The evaluation index is as follows: Fig.15 shown.

[0126] In some specific implementations, the training phase of the license plate detection module may include the following steps:

[0127] (i) Input the license plate images in the training set, resize and standardize them to obtain the first network input image, whose shape is "(batch_size, 3, H, W)". Batch_size is the number of images in a batch of training.

[0128] Among them, the license plate image can be adjusted to the network input size (H, W) by using bilinear interpolation and padding, as follows: (1) Multiply the width and height of the original image by the same coefficient (resize coefficient), and resize the image using bilinear interpolation. The purpose of multiplying by the same coefficient here is to avoid image distortion and keep the aspect ratio of the image before and after resizing unchanged, that is, to keep the target in the original image from being deformed to the greatest extent possible.

[0129] If the values ​​of H and W (i.e., the size of the network input) are increased, the amount of computation required for network reasoning will increase, and it will be difficult to run at a high frame rate on a low-power chip. If the values ​​of H and W are reduced, the performance of the algorithm will be reduced. In the present invention, considering the performance of the algorithm and the high frame rate, H=W=320 is set.

[0130] Fig.12 A schematic diagram of an original license plate image provided for some embodiments of the present application. Fig.12, whose width and height are 720 and 1160 respectively. The width and height of the first network input image are both set to 320, and the calculated resize coefficient is 0.275862, the left and right padding values ​​are 60 and 61 respectively, and the top and bottom padding are both 0. Fig.13 for Fig.12 The image after resizing (width and height are 199 and 320 respectively). Fig.14 for Fig.12 The image after resizing and padding on the left and right (width and height are 320 and 320 respectively).

[0131] The labeling process refers to the standardization of the three channels of the RGB image after resizing and padding according to the mean value calculated in the ImageNet dataset, that is, subtracting the mean value respectively. The calculation formula is formula (4).

[0132] (4)

[0133] Among them, img is the image after resizing and padding, and mean is the mean of the images in ImageNet (104, 117, 123). The standardized data can accelerate the network convergence during training and effectively improve the generalization ability of the network model.

[0134] For the first network input image, its shape is (batch_size, 3, H, W), where batch_size is the number of images in a training batch, and 3 represents the number of channels. It means that a batch uses batch_size images for training, and each network input image has 3 channels, and its shape is (H, W, 3). H and W of the network input can be 320 or 640, whichever is divisible by an integer multiple of 32. In the present invention, H=W=320.

[0135] (ii) Use the improved MobileNet0.25 lightweight backbone network to extract features from the first network input image to obtain first feature vectors of different semantics and resolutions; the shapes of the first feature vectors are: (batch_size, 32, H / 4, W / 4), indicating that the width and height (i.e., resolution) of the feature vector are respectively one-fourth of the model input size, where 32 represents the number of channels; (batch_size, 64, H / 8, W / 8), indicating that the width and height (i.e., resolution) of the feature vector are respectively one-eighth of the model input size, where 64 represents the number of channels; (batch_size, 128, H / 16, W / 16), indicating that the width and height (i.e., resolution) of the feature vector are respectively one-sixteenth of the model input size, where 128 represents the number of channels; (batch_size, 256, H / 32, W / 32), indicating that the width and height (i.e., resolution) of the feature vector are respectively one-thirty-second of the model input size, where 256 represents the number of channels;

[0136] The above feature vectors are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the model input size, which are different resolutions and also represent different semantic information. The deeper the feature, the stronger the high-level semantics and the stronger the resolution.

[0137] (iii) Perform two branch calculations on the first eigenvector.

[0138] Among them, branch 1 uses feature enhancement module A to fuse the first feature vectors with different semantics and resolutions. The shapes of the first fused features after feature fusion are: (batch_size, 64, H / 8, W / 8), (batch_size, 64, H / 16, W / 16), (batch_size, 64, H / 32, W / 32); then the fused first fused features are passed through the Detection Head convolution layer to obtain the first detection box feature map (shape is batch_size × 6300 × 4), the first classification feature map (shape is batch_size × 6300 × 2) and the first corner feature map (shape is batch_size × 6300 × 8).

[0139] Among them, the calculation process of 6300: the shapes of the fused feature vectors are (batch_size, 64, H / 8, W / 8), (batch_size, 64, H / 16, W / 16), and (batch_size, 64, H / 32, W / 32). Here, the number of anchors corresponding to each feature point of each layer of feature map is 3. According to the following formula, H / 8× W / 8×3 + H / 16×W / 16 ×3 + H / 32×W / 32×3 = 6300 can be obtained (H and W here are both 320).

[0140] Branch 2 uses the feature enhancement module B to fuse the second feature vectors with different semantics and resolutions. The shape of the second fused feature after feature fusion is: (batch_size, 96, H / 4, W / 4); then the fused features are passed through the Seg Head convolution layer to obtain the probability map (batch_size, 1, H, W), the threshold map (batch_size, 1, H, W) and the binary map (batch_size, 1, H, W). Finally, the probability map, the threshold map and the binary map are merged in the channel dimension to obtain the feature map (batch_size, 3, H, W).

[0141] The license plate corner point detection method provided in the present application introduces a new bypass supervision branch in the training stage of the license plate detection module to supervise the main branch of corner point detection and improve the accuracy of corner point detection, but the bypass supervision branch is removed in the reasoning stage of the license plate detection module. In this way, the accuracy of the four corner point coordinates of the license plate tilted at a large angle is greatly increased without increasing the weight parameter size and calculation amount of the backbone network, and accurate corner point coordinates are provided for the subsequent license plate correction module, thereby improving the recognition performance of the license plate recognition network and greatly reducing missed characters and misrecognition of characters.

[0142] Furthermore, the feature enhancement module A used in the main branch of the license plate detection module during training is an improved FPN module, and the feature enhancement module B in the bypass supervision branch is also an improved FPN module, while the feature enhancement module B and the Seg Head branch are deleted in the inference stage; this change improves the evaluation index Hmean of the license plate detection model from 0.8868 to 0.9927 (an increase of 10.6%) compared to the model without adding the bypass branch, and improves the accuracy of the license plate detection + license plate recognition model from 87.46% to 99.58% (an increase of 12.12%), which can effectively improve misrecognition such as missing characters and wrong characters.

[0143] Further, such as Fig.15 and 16On a low-power chip, the lightweight backbone network MobileNet0.25 (with only 428.56K weight parameters and 518M computational complexity) achieves or even exceeds the corner detection performance of some large backbone networks.

[0144] Specifically, Fig.15 The evaluation indicators Hmean / Recall / Precision are all calculated on the public dataset CCPD for license plate detection (a total of 12787 images). The public dataset CCPD is divided into three parts, namely the training set, the test set and the evaluation set; the training set and the test set are used in the training phase, while the evaluation set is used in the inference evaluation phase. The Hmean indicator (i.e. F1score) is the harmonic mean of Recall and Precision, i.e. Hmean = 2 / (1 / Recall + 1 / Precision). The value range of the three indicators Hmean / Recall / Precision is [0,1], and the closer the value is to 1, the better the model performance. Among them, Params represents the number of parameters in the inference phase of the license plate detection network, and Flops represents the amount of calculation in the inference phase of the license plate detection network.

[0145] like Fig.16 The evaluation index Accuracy is calculated on the public dataset CCPD for license plate recognition (a total of 12787 images). The value range of the Accuracy index (accuracy, calculated as the number of correctly recognized license plates / total number of license plates) is [0,1], and the closer the value is to 1, the better the model performance. Among them, Params represents the number of parameters in the license plate recognition network reasoning stage, and Flops represents the amount of calculation in the license plate recognition network reasoning stage;

[0146] Furthermore, the present invention has good scalability and robustness, and other backbone networks (such as ResNet series networks) can be used to replace the MobileNet0.25 network used in the present invention, so as to further improve the corner point detection performance of license plates with large angle tilts in various complex scenarios (running on the server).

[0147] Furthermore, in the present invention, each feature point of each feature map layer uses three anchor frames to learn the location information of the license plate to increase the number of positive samples, thereby improving the convergence speed of model training and saving model training time. Moreover, the convolution types used in the present invention are all ordinary convolutions, without complex hole convolutions and deformable convolutions, which are easy to deploy on low-power chips on the end side and can run efficiently.

[0148] Figures 17a-17d A schematic diagram of license plate inference results in the inference stage provided by multiple embodiments of the present application. Figures 17a-17d As shown, the upper left part of each picture is the reasoning result after improvement of this application, the upper right part is the reasoning result before improvement, the lower left part is the picture obtained by the license plate correction module based on the corner point reasoning result after improvement of this application, and the lower right part is the picture obtained by the license plate correction module based on the corner point reasoning result before improvement.

[0149] The final result of the license plate detection + license plate recognition model will be given in the upper left corner of the upper right part of each picture (including corner detection and license plate recognition results), where the first line this application (before improvement)_predict is the license plate number result inferred by the model before improvement of this application + license plate recognition model, the second line this application (after improvement)_predict is the license plate number result inferred by the model after improvement of this application + license plate recognition model, and the third line label is the real license plate number.

[0150] It can be seen that the improved license plate detection model in this application can effectively improve the positions of the four corner points, further improve the effect of license plate recognition, and effectively reduce misrecognition such as wrong characters and missing characters.

[0151] The present application also provides a device for detecting license plate corner points, such as Fig.18 As shown, it includes: an acquisition module 11, used to acquire a license plate detection data set, and obtain a training set, a test set and an evaluation set based on the license plate detection data set; and obtain a feature enhancement module A and a feature enhancement module B; a feature extraction module 12, used to obtain a plurality of first feature vectors with different semantics and resolutions based on the training set; a first branch processing module 13, used to obtain a first loss function based on the first feature vector and the feature enhancement module A; a second branch processing module 14, used to obtain a second loss function based on the first feature vector and the feature enhancement module B; a model processing module 15, used to calculate the total loss function based on the first loss function and the second loss function; and obtain a training model based on the test set and the total loss function; an inference module 16, used to obtain the coordinates of the license plate corner points and the weight parameters of the evaluation index model based on the backbone network, the evaluation set, the feature enhancement module A and the training model.

[0152] The device for license plate corner point detection provided in the present application can implement any embodiment of the above-mentioned license plate corner point detection method.

[0153] Furthermore, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the license plate corner point detection method in any of the above embodiments are implemented.

[0154] Furthermore, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the license plate corner point detection method in any of the above embodiments are implemented.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting license plate corner points, characterized in that: include: Acquire a license plate detection data set, and obtain a training set, a test set, and an evaluation set based on the license plate detection data set; Acquire feature enhancement module A and feature enhancement module B; In the training phase, a plurality of first feature vectors with different semantics and resolutions are obtained based on the training set; two branches are respectively calculated for the first feature vectors, wherein branch 1 is the main branch, including obtaining a first loss function based on the first feature vector and the feature enhancement module A; the first loss function is calculated by formula (1), (1), wherein Loss_box is the detection box loss, Loss_cls is the category loss, Loss_landmark is the corner loss, and Loss1 is the first loss function; branch 2 is a bypass supervision branch, including a second loss function obtained based on the first feature vector and the feature enhancement module B; the second loss function is calculated by formula (2), (2), where Loss_s is the probability map loss, Loss_t is the threshold map loss, and Loss_b is the binary map loss; Calculate a total loss function based on the first loss function and the second loss function; Based on the test set and the total loss function, obtaining a training model; In the inference stage, the bypass supervision branch is removed, and the license plate corner point coordinates and evaluation indicators are obtained based only on the evaluation set, the feature enhancement module A and the training model, including: Based on the evaluation set, a plurality of second feature vectors with different semantics and resolutions are obtained; Using the feature enhancement module A to perform feature fusion on the second feature vector to obtain a second fused feature; Based on the second fusion feature, a second detection frame feature map, a second classification feature map and a second corner point feature map are obtained through a Detection Head module; Based on the training model, the second detection frame feature map, the second classification feature map and the second corner point feature map are decoded and maximum value suppressed to obtain the license plate corner point coordinates; The evaluation index is obtained based on the true label of the evaluation set and the model weight parameter of the training model.

2. The method for detecting license plate corner points according to claim 1, characterized in that: The obtaining of a plurality of first feature vectors with different semantics and resolutions based on the training set comprises: Preprocessing the training set to obtain a first network input picture; Perform feature extraction on the first network input picture to obtain the first feature vector.

3. The method for detecting license plate corner points according to claim 1, characterized in that: The obtaining a first loss function based on the first feature vector and the feature enhancement module A includes: Using the feature enhancement module A to perform feature fusion on each of the first feature vectors to obtain a first fused feature; Based on the first fusion feature, a first detection frame feature map, a first classification feature map and a first corner point feature map are obtained through a Detection Head module; Based on the first detection box feature map, the first classification feature map and the first corner point feature map, respectively obtaining a detection box loss, a category loss and a corner point loss; The first loss function is calculated based on the detection box loss, the category loss and the corner point loss.

4. The method for detecting license plate corner points according to claim 1, characterized in that: The obtaining of a second loss function based on the first feature vector and the feature enhancement module B includes: Using the feature enhancement module B to perform feature fusion on each of the first feature vectors to obtain a second fused feature; Based on the second fusion feature, a probability map, a threshold map and a binary map are obtained through a Seg Head module; Based on the probability map, the threshold map and the binary map, respectively obtain a probability map loss, a threshold map loss and a binary map loss; The second loss function is calculated based on the probability map loss, the threshold map loss, and the binary map loss.

5. The method for detecting license plate corner points according to claim 1, characterized in that: The obtaining of a plurality of second feature vectors with different semantics and resolutions based on the evaluation set comprises: Preprocessing the evaluation set to obtain a second network input picture; Perform feature extraction on the second network input picture to obtain the second feature vector.

6. The method for detecting license plate corner points according to claim 2 or 5, characterized in that: The feature extraction is performed using a MobileNet0.25 lightweight backbone network.

7. A device for detecting license plate corner points, implementing the steps of the method described in any one of claims 1 to 6, characterized in that: include: An acquisition module is used to acquire a license plate detection data set, and obtain a training set, a test set and an evaluation set based on the license plate detection data set; and obtaining feature enhancement module A and feature enhancement module B; A feature extraction module, used to obtain a plurality of first feature vectors with different semantics and resolutions based on the training set; A first branch processing module, used for obtaining a first loss function based on the first feature vector and the feature enhancement module A; A second branch processing module, used for obtaining a second loss function based on the first feature vector and the feature enhancement module B; A model processing module, configured to calculate a total loss function based on the first loss function and the second loss function; and obtain a training model based on the test set and the total loss function; The reasoning module is used to obtain the coordinates of the corner points of the license plate and the evaluation index based on the evaluation set, the feature enhancement module A and the training model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images