A method for license plate detection in low-quality images based on deep neural network

Abstract

The present invention discloses a method for low-quality image license plate detection based on a deep neural network. First, training data is obtained to construct a low-quality image license plate detection network, and then the labeled training data is preprocessed; the corner point difference loss is used to constrain the coordinates of the located corner points; finally, the low-quality image license plate detection network is trained according to a determined loss function, and the license plate area in the image is detected and located by the trained low-quality image license plate detection network. The present invention proposes a dual-pixel type regression strategy using corner points and center points, and designs a relaxed constraint to implicitly integrate the advantages of the two regression types to achieve better detection performance.

Description

Technical Field

[0001] The present invention belongs to the field of image target detection and relates to a method for detecting license plates in low-quality images based on a deep neural network. Background Art

[0002] With the rapid development of computer technology and communication technology, the ability to automatically process information has been enhanced, and many intelligent systems for traffic management have been produced accordingly, such as vehicle navigation systems, electronic police systems, global positioning systems, and license plate recognition systems that are serving the transportation industry. Since the license plate is the only effective symbol of vehicle identity information, it is of great significance to regulate road traffic to accurately and efficiently extract license plate information while the vehicle is driving on the road, making Automatic License Plate Recognition (ALPR) an important component of the intelligent transportation system. Although the existing license plate recognition technology is relatively mature, the license plate detection under low-quality image input represented by fuzzy imaging and bad weather still faces the defect of low accuracy, which leads to errors in license plate character recognition and failure to correctly identify vehicle identity information. Due to the uncertainty of vehicle driving status in real scenes and the complexity and uncontrollability of the external environment, license plate recognition faces the following problems: 1) Due to the improper camera shooting position, the license plate has large-scale tilt and spatial distortion on the geometric plane, making it difficult to accurately locate the license plate area; 2) Due to factors such as rainy and foggy weather, non-uniform lighting and motion blur, the visual features of the license plate are not obvious and difficult to be effectively detected by the detector. Since the visual features of the license plate in low-quality images are very weak, existing methods do not have a special solution for these boundary samples, which is the key to the practical application of license plate detection. Summary of the invention

[0003] In order to overcome the problems existing in license plate recognition in the above-mentioned real scenes, the present invention proposes a method for license plate detection of low-quality images based on deep neural networks. The method improves the above-mentioned problems respectively, including: 1) taking into account that different regression strategies have different focuses on the processing capabilities of different types of boundary samples. Specifically, the corner point regression method has excellent performance in the corner point detection of distorted license plates. However, the visual features of fuzzy license plates are not obvious due to the influence of the surrounding environment. Compared with the corner point regression method, the center point regression method can more efficiently lock the area range of the license plate. Based on this observation, this paper proposes a dual-pixel type regression strategy based on corner points and center points to deal with different forms of noise samples. 2) In order to solve the contradiction between the two regression strategies, this paper encodes the two tasks separately through a decoupled detection head, and designs a constraint with relaxation to decouple the correlation of different regressions. The above-mentioned constraints implicitly integrate the advantages of the two regression types, greatly improving the detection performance of the detector for license plate boundary samples.

[0004] A method for detecting license plates from low-quality images based on a deep neural network comprises the following steps:

[0005] Step 1: Collect low-quality images of vehicles containing license plates, mark the four corner points of the license plates in the image clockwise from the upper left corner, and encode the marking results into the file name corresponding to the image as the label of the training data.

[0006] Step 2: Build a low-quality image license plate detection network (LPCDet);

[0007] The low-quality image license plate detection network (LPCDet) uses ResNet-50 as the feature extractor, i.e. the backbone network, and introduces the feature fusion module (FPEM) to perform multi-scale feature fusion on the feature maps extracted by the backbone network. The multi-scale fused feature maps are input into the decoupled network sub-branches respectively.

[0008] Step 3: Preprocess the labeled training data.

[0009] Step 4: Use the corner point difference loss to constrain the coordinates of the located corner points.

[0010] Step 5: Train the low-quality image license plate detection network according to the determined loss function:

[0011] Step 6: Use the trained low-quality image license plate detection network to detect and locate the license plate area in the image.

[0012] Furthermore, step 2 is specifically implemented as follows:

[0013] The low-quality image license plate detection network (LPCDet) uses ResNet-50 as the feature extractor, i.e. the backbone network, and the feature map size of its output is Where B is the batch size, W and H are the sizes of the input images. A feature fusion module (FPEM) is introduced to perform multi-scale feature fusion on the feature maps extracted by the backbone network to improve the detection network to capture richer license plate features. The multi-scale fused feature maps are input into the decoupled network sub-branches respectively, which respectively constitute the heat map positioning license plate corner point module and the center point based offset positioning license plate corner point module.

[0014] For all network sub-branches, a convolution block is first used to extract features from the input feature map. The convolution block contains a 3×3 convolution, a batch normalization layer, and a Rectified Linear Units (ReLU) activation function. At this time, the number of channels of each feature map is kept unchanged. Furthermore, for the heat map positioning license plate corner module, it includes a corner heat map sub-branch for predicting the corner heat map and a refinement sub-branch for refining the corresponding heat map position. The former adjusts the number of channels of the feature map to 4 through a 1×1 convolution, and uses the Sigmoid activation function to explicitly represent the predicted heat map confidence on the feature map. The output can be expressed as The latter directly uses a 1×1 convolution to adjust the number of channels of the feature map to 8, and the output can be expressed as For the center point offset positioning license plate corner point module, it contains a center point heat map sub-branch for predicting the center point heat map of the license plate and an offset sub-branch for predicting the offset from the center point to the four corner points. The former adjusts the number of channels of the feature map to 1 through a 1×1 convolution, and then uses the Sigmoid activation function to output the confidence of the corresponding center point heat map on the feature map, which can be expressed as The latter directly uses a 1×1 convolution to adjust the number of channels of the feature map to 8, which can be expressed as

[0015] Furthermore, the specific method of step 3 is as follows:

[0016] 3-1. Use bilinear interpolation to adjust the image to the size (512×512) that meets the network (LPCDet) input. In order to avoid image distortion and reduce the computational complexity of model inference, first calculate the maximum side length of the image size, then fill the shorter side length area with grayscale bars, and then scale the processed image to the target size;

[0017] 3-2. The resized license plate images are standardized by channel according to the mean and standard deviation calculated in the ImageNet dataset, that is, the values ​​of each channel are scaled, and then the mean is subtracted and divided by the standard deviation. The calculation formula is:

[0018]

[0019] Among them, I is the resized image, pixel = 255 is the maximum pixel threshold of the image, mean is the mean of the image in the ImageNet dataset, and std is the standard deviation of the image in the ImageNet dataset. The network can converge faster when training with standardized data, and the generalization of the model is effectively improved.

[0020] 3-3. For each corner coordinate p of the license plate label in the image, first calculate a scaled low-resolution corner coordinate R is the scaling factor, which is 4; then a Gaussian kernel is used to map all corner points to the heat map The calculation of mapping the corner points from the Gaussian function to the heat map is:

[0021]

[0022] where σ p is the adaptive standard deviation of the current license plate. In order to reduce the discrete error caused by the output step, the deviation corresponding to the i-th corner point is also calculated

[0023] Calculate the center point of the area from the coordinates of the license plate corner points Get the offset distance from the center point to the four corner points in and Indicates the offset between the i-th corner point and the center point in the x- and y-axis directions.

[0024] Furthermore, the specific method of step 4 is as follows:

[0025] The robustness of corner point localization is improved by a relaxed constraint strategy, which includes a relaxed constraint loss to couple the two regression methods.

[0026] Step 4-1. In the process of heat map positioning license plate corner module regression, the feature map output by the corner point heat map sub-branch Perform feature decoding to obtain the position coordinates of the license plate corner points. Then refine the feature map output by the sub-branch Decoding is performed to calculate the deviation used to accurately determine the position of the corresponding license plate corner point. The precise coordinates of the license plate corner point are obtained based on the position coordinates and deviation of the license plate corner point.

[0027] Step 4-2. In the process of center point offset positioning license plate corner point module regression, firstly, the feature map output by the center point heat map sub-branch is Perform feature decoding to obtain the position coordinates of the center point of the license plate, and then use the feature map output by the offset quantum branch The decoding calculates the offset from the center point to each corner point, and the corner point can be located by adding the corresponding offset to the center point.

[0028] Step 4-3. Relaxed constraint loss based on corner point difference.

[0029] The relaxed constraint loss function couples the regression process of the two modules so that more effective constraints and guidance can be generated between the two modules. The loss function formula is as follows:

[0030]

[0031]

[0032] in, Represents the coordinate value corresponding to the i-th corner point directly predicted based on the heat map, represents the coordinate value corresponding to the i-th corner point predicted based on the offset of the center point of the license plate. α is the modulation factor, and β is the radius of the relaxation boundary. In summary, the total relaxation loss is defined as:

[0033]

[0034] Furthermore, the specific method of step 5 is as follows:

[0035] ResNet-50 uses the pre-trained weights on the ImageNet dataset, uses the CCPD license plate dataset to train the low-quality image license plate detection network (LPCDet), and then uses the training data processed in step 3 for adjustment. The batch size is set to 28 on the Nvidia 3080 GPU, the total training iteration is 50 rounds, and Adam is used as the optimizer, with an initial learning rate of 0.01 and a weight decay of 10 -5 The learning rate is adjusted by the cosine function. As the rounds increase, the learning rate gradually decreases to 5×10 -4 In order to improve the generalization of the model, the imgaug library is used to implement data augmentation. During training, 1 to 3 augmentation operations are randomly selected for each image.

[0036] Further, enhancement operations include brightness adjustment, contrast adjustment, color gamut transformation, histogram equalization, random rotation and cropping, horizontal flipping, mean shift blur, motion blur, affine transformation, perspective transformation, and rainy day simulation.

[0037] Furthermore, the specific method of step 6 is as follows:

[0038] For the input image to be detected, image size adjustment and standardization preprocessing are performed, and the feature map of the corner point heat map is output after the trained model is put into reasoning. And the characteristic map of the deviation Through Calculate the vector maximum index to get the predicted value with the highest confidence, and then substitute the predicted value into Calculate the deviation of the corner point in the Cartesian coordinate system. In summary, the calculation formula for license plate corner point prediction decoding can be expressed as:

[0039]

[0040] where conf i Indicates the heat map area corresponding to the maximum confidence of the i-th corner point, It represents the coordinate value corresponding to the i-th corner point, and mod represents the remainder operation.

[0041] In the implementation process of the present invention, the four corner points of the license plate are annotated with the image as the training data of the low-quality image license plate detection network; the online data enhancement method is used to allow the model to learn the invariance information of the license plate, thereby improving the generalization of the model without increasing the data; the GT coordinates are generated according to the two-dimensional Gaussian function The corresponding heat map is used, and the Gaussian heat map is used to make a soft annotation of the GT coordinates to increase directional guidance for the network training, so that the network converges faster; for the positioning of the license plate corner points, the license plate corner point positioning module based on the offset of the center point can better determine the license plate area, and the license plate corner point direct positioning module using the heat map can more efficiently and accurately locate the license plate corner points. The former guides the latter to form a weak supervision relationship, which effectively solves the problem that the license plate corner points in low-quality images are unclear and difficult to locate; TIoU is introduced as a new license plate detection evaluation indicator to avoid the false high index caused by the traditional IoU metric, so that the detection frame and the real area of ​​the license plate are more matched, which provides a guarantee for subsequent license plate character recognition.

[0042] The beneficial effects of the present invention are as follows:

[0043] Existing methods only consider single pixel type regression to obtain license plate corner points. This solution may not achieve ideal results in various forms of low-quality license plate images. Through experiments, we found that the corner point regression method has excellent performance in detecting corner points of distorted license plates, while the center point regression method can more efficiently lock the license plate area in a blurred image. Based on this observation, the present invention proposes a dual-pixel type regression strategy using corner points and center points, and designs a relaxed constraint to implicitly integrate the advantages of the two regression types to achieve better detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions involved in the present invention, the following briefly introduces the drawings used in the implementation process of the present invention.

[0045] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0046] Figure 2 It is a model structure diagram designed by the method proposed in the present invention;

[0047] Figure 3 It is the structure diagram of feature fusion module (FPEM);

[0048] Figure 4 It is the relaxation loss function curve and the corresponding gradient curve proposed by the present invention. Specific implementation methods

[0049] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0050] Taking the license plate images taken by gate equipment in residential areas, parking lots and other places as an example, the pre-trained ResNet-50 network is used as the feature extraction backbone network of the model, and the present invention is further explained by describing the process of model training and license plate detection. The following description is only for demonstration and explanation, and does not limit the present invention in any form.

[0051] like Figure 1 As shown in the figure, a low-quality image license plate detection method based on deep neural network is implemented as follows:

[0052] Step 1. Use open source annotation tools to annotate the collected license plate data set. The annotation method is to annotate the four corner points along the license plate area in a clockwise direction from the upper left corner. The closed loop area formed is the annotated license plate area. Then decode the content in the generated json annotation file (the coordinates of the four corner points in the license plate and the license plate number) as the file name of the corresponding license plate image, and divide it into training set, validation set and test set in a ratio of 7:2:1 for subsequent detection model training.

[0053] Step 2: Figure 2 As shown in Figure 1, the low-quality image license plate detection network (LPCDet) uses ResNet-50 as the feature extractor, i.e., the backbone network. It uses the direct connection of residual blocks to avoid the problem of serious degradation of the network as the depth increases. It can efficiently extract the feature information of the input image, and the size of its output feature map is Where B is the batch size, W and H are the sizes of the input images. Furthermore, a feature fusion module (FPEM) is introduced to perform multi-scale feature fusion on the feature maps extracted by the backbone network to improve the detection network to capture richer license plate features. Figure 3The FPEM shown in the figure is mainly divided into two stages: up-scale enhancement and down-scale enhancement. In the up-scale enhancement stage, the input feature map is iteratively enhanced with a step size of 32, 16, 8, and 4 pixels. In the down-scale enhancement stage, the feature map of the previous stage is taken as input, and then the features are enhanced from small to large steps from 4 to 32. The output feature map size after feature fusion is Where B is the batch size, W and H are the sizes of the input image. The feature maps after multi-scale fusion are respectively input into the decoupled network sub-branches, which respectively constitute the heat map positioning license plate corner module and the license plate corner positioning module based on the offset of the center point. For all network sub-branches, a convolution block is first used to extract features from the input feature map. The convolution block contains a 3×3 convolution, a batch normalization layer and a RectifiedLinear Units (ReLU) activation function, while keeping the number of channels of each feature map unchanged. Furthermore, for the heat map positioning license plate corner module, it includes a corner heat map sub-branch for predicting the corner heat map and a refinement sub-branch for refining the corresponding heat map position. The former adjusts the number of channels of the feature map to 4 through a 1×1 convolution, and uses the Sigmoid activation function to explicitly represent the heat map confidence predicted on the feature map. The output can be expressed as The latter directly uses a 1×1 convolution to adjust the number of channels of the feature map to 8, and the output can be expressed as For the center point offset positioning license plate corner point module, it contains a center point heat map sub-branch for predicting the center point heat map of the license plate and an offset sub-branch for predicting the offset from the center point to the four corner points. The former adjusts the number of channels of the feature map to 1 through a 1×1 convolution, and then uses the Sigmoid activation function to output the confidence of the corresponding center point heat map on the feature map, which can be expressed as The latter directly uses a 1×1 convolution to adjust the number of channels of the feature map to 8, which can be expressed as

[0054] Step 3: Preprocess the labeled training data.

[0055] 3-1. Use bilinear interpolation to adjust the image to the size (512×512) that meets the network (LPCDet) input. In order to avoid image distortion and reduce the computational complexity of model inference, first calculate the maximum side length of the image size, then fill the shorter side length area with grayscale bars, and then scale the processed image to the target size;

[0056] 3-2. The resized license plate images are standardized by channel according to the mean and standard deviation calculated in the ImageNet dataset, that is, the values ​​of each channel are scaled, and then the mean is subtracted and divided by the standard deviation. The calculation formula is:

[0057]

[0058] Among them, I is the resized image, pixel = 255 is the maximum pixel threshold of the image, mean = [0.40789655 0.44719303 0.47026116] represents the mean of the image in the ImageNet dataset, and std = [0.2886383 0.27408165 0.27809834] represents the standard deviation of the image in the ImageNet dataset. The network can converge faster when training with standardized data, and the generalization of the model is effectively improved.

[0059] 3-3. For each corner coordinate p of the license plate label in the image, first calculate a scaled low-resolution corner coordinate R is the scaling factor, which is 4; then a Gaussian kernel is used to map all corner points to the heat map The calculation of mapping the corner points from the Gaussian function to the heat map is:

[0060]

[0061] where σ p is the adaptive standard deviation of the current license plate (the superscript is the square, and the subscript is the Gaussian function parameter calculated based on the relative size of the current license plate). Since the feature map size (128×128) output by the network (LPCDet) is one-fourth of the input image (512×512), in order to reduce the discrete error caused by the output stride, the deviation corresponding to the i-th corner point is also calculated.

[0062] Calculate the center point of the area from the coordinates of the license plate corner points Get the offset distance from the center point to the four corner points in and Indicates the offset between the i-th corner point and the center point in the x- and y-axis directions.

[0063] Step 4: Use the corner point difference loss to constrain the coordinates of the located corner points.

[0064] In order to solve the problem that the corner features of license plates in low-quality images are not obvious and it is difficult to effectively detect the corner points of license plates, we propose a relaxed constraint strategy to improve the robustness of corner point positioning. This strategy includes a relaxed constraint loss used to couple the two regression methods.

[0065] Step 4-1. In the process of heat map positioning license plate corner module regression, the feature map output by the corner point heat map sub-branch Perform feature decoding to obtain the position coordinates of the license plate corner points. Then refine the feature map output by the sub-branch Decode and calculate the deviation used to accurately determine the position of the corresponding license plate corner point. According to the position coordinates and deviation of the license plate corner point, the precise coordinates of the license plate corner point are obtained. The heat map license plate corner point positioning module is the main module of the model. Ultimately, the output of the model is based on the prediction results of this module.

[0066] The number of channels of the feature map on the diagonal point heat map sub-branch is adjusted to 4, corresponding to the 4 corner points of the license plate to be detected, and a Sigmoid activation function is applied at the end to explicitly represent the probability that the heat map corresponds to the real corner point. The 8 channels correspond to the 8 coordinate parameters regressed from the heat map in the x-axis and y-axis directions, which are used to compensate for the discretization error caused by the output step size, making the predicted license plate corner points more accurate.

[0067] Step 4-2. In the process of center point offset positioning license plate corner point module regression, firstly, the feature map output by the center point heat map sub-branch is Perform feature decoding to obtain the position coordinates of the center point of the license plate, and then use the feature map output by the offset quantum branch Decoding calculates the offset from the center point to each corner point, and the corner point can be located by adding the corresponding offset to the center point. The center point offset license plate corner point positioning module is an auxiliary module of the model. It is only used to assist the heat map direct license plate corner point positioning module training during model training and does not participate in the final model output results.

[0068] Step 4-3. Relaxed constraint loss based on corner point difference.

[0069] Compared with directly predicting the corner points of the license plate, the visual features of the center of the license plate are more obvious in low-quality images, so it is easier to lock the approximate area of ​​the license plate through the center point of the license plate. Specifically, in the early stage of model training, the center point offset positioning license plate corner point module can lock the license plate area more efficiently than the heat map positioning license plate corner point module. At this time, it can provide positive guidance for the regression process of the heat map positioning license plate corner point module. However, as the model gradually converges, the heat map positioning license plate corner point module can accurately locate the corner point position of the license plate. At this time, the constraints imposed by the center point offset positioning license plate corner point module on the heat map positioning license plate corner point module will have a negative impact. The relaxed constraint loss function proposed in this paper couples the regression process of the two modules so that more effective constraints and guidance can be generated between the two modules. The loss function formula is expressed as follows:

[0070]

[0071]

[0072] in, Represents the coordinate value corresponding to the i-th corner point directly predicted based on the heat map, represents the coordinate value corresponding to the i-th corner point predicted based on the offset of the center point of the license plate. α = 0.1 is the modulation factor, and β = 5 is the radius of the relaxation boundary. In summary, the total relaxation loss is defined as:

[0073]

[0074] like Figure 4 As shown in the figure, assuming that the radius of the relaxed boundary is β = 5, samples with small differences in corner coordinates should obtain smaller gradients to eliminate the negative impact of the center point-based corner positioning module on the heat map-based direct corner positioning module in the later stage of training. At the same time, samples with large differences are likely to cause gradient explosion, resulting in incorrect learning.

[0075] Step 5: Train the low-quality image license plate detection network according to the determined loss function:

[0076] The backbone network ResNet-50 in the low-quality image license plate detection network uses the weights pre-trained on the ImageNet dataset. The CCPD license plate dataset is used to train the low-quality image license plate detection network (LPCDet), and then the training data processed in step 3 is used for adjustment, that is, the LPCDet model is loaded with the parameters trained on the CCPD license plate dataset and fine-tuned with the pre-processed labeled training data. The batch size is set to 28 on the Nvidia 3080 GPU, the total training iteration is 50 rounds, and Adam is used as the optimizer, with an initial learning rate of 0.01 and a weight decay of 10 -5 The learning rate is adjusted by the cosine function. As the rounds increase, the learning rate gradually decreases to 5×10 -4 In order to improve the generalization of the model, imgaug is used to implement data augmentation. One to three augmentation operations are randomly selected each time. The augmentation operations include brightness adjustment, contrast adjustment, color gamut transformation, histogram equalization, random rotation and cropping, horizontal flipping, mean shift blur, motion blur, affine transformation, perspective transformation, and rainy day simulation.

[0077] Step 6: Use the trained low-quality image license plate detection network to detect and locate the license plate area in the image.

[0078] For the input image to be detected, image size adjustment and standardization preprocessing are performed, and the feature map of the corner point heat map is output after the trained model is put into reasoning. And the characteristic map of the deviation Through Calculate the vector maximum index to get the predicted value with the highest confidence, and then substitute the predicted value into Calculate the deviation of the corner point in the Cartesian coordinate system. In summary, the calculation formula for license plate corner point prediction decoding can be expressed as:

[0079]

[0080] where conf i Indicates the heat map area corresponding to the maximum confidence of the i-th corner point, It represents the coordinate value corresponding to the i-th corner point, and mod represents the remainder operation.

[0081] Step 7. We introduce the TIoU metric as a detection evaluation indicator. The current evaluation indicator using IoU as a metric cannot accurately represent the accuracy of the detection results. Sometimes, even detection results with missing characters and too much background noise are regarded as correct samples. The TIoU metric we introduced will make the detection results pay more attention to all areas of GT, that is, ensure the integrity of the detection box relative to GT, and penalize the detection box outside GT, so that the detection results with higher scores can have better compactness. Based on the above two points, it can be guaranteed that the detection results with high TIoU must be better than those with low TIoU. The calculation process of recall, precision and F1 score in TIoU will be explained below.

[0082] 7-1. Calculation of TIoU recall rate: First, calculate the undetected area in GT:

[0083] C t =A(G i )-A(D j ∩G i ), C t ∈[0,A(G i )]

[0084] Where A(*) represents the area of ​​the region, G i is the i-th GT region, D j is the area of ​​the jth detection box, then G i The intersection ratio can be calculated by the following formula:

[0085]

[0086] The TIoU recall rate calculation formula is as follows:

[0087]

[0088] 7-2. Calculation of TIoU precision: First, calculate the abnormal area in the detection box that is not in the target GT area:

[0089] O t =A(D j -D j ∩G i ), O t ∈[0,A(D j )]

[0090] Then the correct detection area intersection ratio is:

[0091]

[0092] Similarly, the calculation formula of TIoU precision is as follows:

[0093]

[0094] 7-3. TIoU-based metrics: In order to calculate the final score, the harmonic mean of recall and precision is usually used as the main metric, and the calculation formula is as follows:

[0095]

[0096] The calculation formulas for recall and precision are:

[0097]

[0098]

[0099] Among them, Num gt is the total number of GT boxes, Num dt is the total number of detection boxes. Example

[0100] On the self-built license plate dataset, we compared the detection and recognition accuracy of LPCDet and the baseline model (i.e. CenterNet before improvement) (the same license plate recognition model, the detection results before and after improvement are used as input). As shown in the table below, in the traditional IoU and new TIoU evaluation protocols, LPCDet outperforms the baseline model, with detection performance improved by 3.3% and 5.1%, and recognition accuracy improved by 0.5%. Under the stricter TIoU metric, LPCDet's advantage over the baseline is further amplified, which reflects that the improved model can achieve more efficient and accurate detection.

[0101]

[0102] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0103] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for low-quality image license plate detection based on deep neural network, characterized in that: The steps include: Step 1: Collect low-quality images of vehicles containing license plates, mark the four corner points of the license plates in the image clockwise from the upper left corner, and encode the marking results into the file name corresponding to the image as the label of the training data; Step 2: Build a low-quality image license plate detection network; The low-quality image license plate detection network uses ResNet-50 as the feature extractor, i.e., the backbone network. The feature fusion module is introduced to perform multi-scale feature fusion on the feature maps extracted by the backbone network. The multi-scale fused feature maps are input into the decoupled network sub-branches respectively. Step 3: Preprocess the labeled training data; Step 4: Constrain the coordinates of the located corner points using the relaxed constraint loss based on the difference of the corner points; The relaxed constraint loss function couples the regression process of the two modules so that more effective constraints and guidance can be generated between the two modules. The loss function formula is as follows: in, Represents the coordinate value corresponding to the i-th corner point directly predicted based on the heat map, represents the coordinate value corresponding to the i-th corner point predicted based on the offset of the center point of the license plate; α is the modulation factor, and β is the radius of the relaxation boundary; in summary, the total relaxation loss is defined as: Step 5: Train the low-quality image license plate detection network according to the determined loss function: Step 6: Detect and locate the license plate area in the image through the trained low-quality image license plate detection network; For the input image to be detected, image size adjustment and standardization preprocessing are performed, and the feature map of the corner point heat map is output after the trained model is put into reasoning. And the characteristic map of the deviation Through Calculate the vector maximum index to get the predicted value with the highest confidence, and then substitute the predicted value into Calculate the deviation of the corner point in the Cartesian coordinate system; In summary, the calculation formula for license plate corner point prediction decoding can be expressed as: where conf i Indicates the heat map area corresponding to the maximum confidence of the i-th corner point, It represents the coordinate value corresponding to the i-th corner point, and mod represents the remainder operation.

2. The method for low-quality image license plate detection based on deep neural network according to claim 1, characterized in that: Step 2 is implemented as follows: The low-quality image license plate detection network uses ResNet-50 as the feature extractor, i.e. the backbone network, and the feature map size of its output is Where B is the batch size, W and H are the sizes of the input images; the feature fusion module is introduced to perform multi-scale feature fusion on the feature map extracted by the backbone network to improve the detection network to capture richer license plate features; The multi-scale fused feature maps are respectively input into the decoupled network sub-branches, and the network sub-branches respectively constitute a heat map positioning license plate corner point module and a center point based offset positioning license plate corner point module; For all network sub-branches, a convolution block is first used to extract features from the input feature map. The convolution block contains a 3×3 convolution, a batch normalization layer and a ReLU activation function. At this time, the number of channels of each feature map is kept unchanged. Furthermore, for the heat map positioning license plate corner module, it includes a corner heat map sub-branch for predicting the corner heat map and a precision sub-branch for refining the corresponding heat map position. The former adjusts the number of channels of the feature map to 4 through a 1×1 convolution, and uses the Sigmoid activation function to explicitly represent the heat map confidence predicted on the feature map. The output can be expressed as The latter directly uses a 1×1 convolution to adjust the number of channels of the feature map to 8, and the output can be expressed as For the center point offset positioning license plate corner point module, it contains a center point heat map sub-branch for predicting the center point heat map of the license plate and an offset sub-branch for predicting the offset from the center point to the four corner points; the former adjusts the number of channels of the feature map to 1 through a 1×1 convolution, and then uses the Sigmoid activation function to output the confidence of the corresponding center point heat map on the feature map, which can be expressed as The latter directly uses a 1×1 convolution to adjust the number of channels of the feature map to 8, which can be expressed as 3. The method for low-quality image license plate detection based on deep neural network according to claim 2 is characterized in that: Step 3: 3-1. Use bilinear interpolation to adjust the image to a size that matches the network input. To avoid image distortion and reduce the amount of computation required for model inference, first calculate the maximum side length of the image, then fill in the shorter side length area with grayscale bars, and then scale the processed image to the target size. 3-2. The resized license plate images are standardized by channel according to the mean and standard deviation calculated in the ImageNet dataset, that is, the values ​​of each channel are scaled, and then the mean is subtracted and divided by the standard deviation. The calculation formula is: Where I is the resized image, pixel = 255 is the maximum pixel threshold of the image, mean is the mean of the image in the ImageNet dataset, and std is the standard deviation of the image in the ImageNet dataset. The network can converge faster during training with standardized data, and the generalization of the model is effectively improved. 3-3. For each corner coordinate p of the license plate label in the image, first calculate a scaled low-resolution corner coordinate R is the scaling factor, which is 4; then a Gaussian kernel is used to map all corner points to the heat map The calculation of mapping the corner points from the Gaussian function to the heat map is: where σ p is the adaptive standard deviation of the current license plate; in order to reduce the discrete error caused by the output step, the deviation corresponding to the i-th corner point is also calculated Calculate the center point of the area from the coordinates of the license plate corner points Get the offset distance from the center point to the four corner points in and Indicates the offset between the i-th corner point and the center point in the x- and y-axis directions.

4. The method for low-quality image license plate detection based on deep neural network according to claim 3 is characterized in that: Step 4 also includes: Step 4-1. In the process of heat map positioning license plate corner module regression, the feature map output by the corner point heat map sub-branch Perform feature decoding to obtain the position coordinates of the license plate corner points; then refine the feature map output by the sub-branch Decoding is performed to calculate the deviation used to accurately determine the position of the corresponding license plate corner point; the precise coordinates of the license plate corner point are obtained based on the position coordinates and deviation of the license plate corner point; Step 4-2. In the process of center point offset positioning license plate corner point module regression, firstly, the feature map output by the center point heat map sub-branch is Perform feature decoding to obtain the position coordinates of the center point of the license plate, and then use the feature map output by the offset quantum branch The decoding calculates the offset from the center point to each corner point, and the corner point can be located by adding the corresponding offset to the center point.

5. The method for detecting license plates from low-quality images based on deep neural networks according to claim 4, characterized in that: Step 5: ResNet-50 uses the pre-trained weights on the ImageNet dataset, uses the CCPD license plate dataset to train the low-quality image license plate detection network, and then uses the training data processed in step 3 for adjustment; set the batch size to 28 on the Nvidia 3080 GPU, and the total training iteration is 50 rounds. At the same time, use Adam as the optimizer, with an initial learning rate of 0.01 and a weight decay of 10 -5 The learning rate is adjusted by the cosine function. As the rounds increase, the learning rate gradually decreases to 5×10 -4 ; In order to improve the generalization of the model, the imgaug library is used to implement data augmentation. During training, 1 to 3 augmentation operations are randomly selected for each image each time.

6. The method for detecting license plates from low-quality images based on deep neural networks according to claim 5, characterized in that: Enhancement operations include brightness adjustment, contrast adjustment, color space transformation, histogram equalization, random rotation and cropping, horizontal flipping, mean shift blur, motion blur, affine transformation, perspective transformation, and rain simulation.

Images