Multi-license-plate key point detection method and related equipment

Abstract

The invention provides a multi-license-plate key point detection method. The multi-license-plate key point detection method comprises the following steps: acquiring license plate image data of a target vehicle; performing key point prediction on the license plate image data through the trained license plate key point detection model to obtain initial license plate key points; performing optimization processing on the initial license plate key point through a preset geometric constraint rule to obtain an optimized license plate key point; and performing affine transformation post-processing on the optimized license plate key point, and outputting the corrected license plate key point. The license plate key points are detected through the license plate key point detection model, optimization processing is performed on the license plate key points in combination with the geometric constraint rule, affine transformation post-processing is performed on the optimized license plate key points, and the corrected license plate key points are output. The problems that an existing license plate detection method is poor in performance and inaccurate in license plate detection in complex scenes such as multi-line license plates, shielding and illumination changes based on target detection or semantic segmentation are solved.

Description

Technical Field

[0001] This invention relates to the field of smart city technology, and in particular to a method and related equipment for detecting key points of multiple license plates. Background Technology

[0002] License Plate Detection and Recognition (LPDR) is one of the key technologies. License plate key point detection aims to accurately locate the four corner points of the license plate for subsequent perspective transformation and license plate correction, thereby improving the accuracy of OCR (Optical Character Recognition). Currently, traditional license plate detection methods are usually based on object detection (such as YOLO, Faster R-CNN) or semantic segmentation (such as U-Net), but these methods perform poorly in complex scenes, such as multiple license plates, occlusion, and changes in lighting, resulting in inaccurate license plate detection. Summary of the Invention

[0003] This invention provides a method for detecting key points on multiple license plates, aiming to address the problem that existing license plate detection methods, based on object detection or semantic segmentation, perform poorly in complex scenes, such as those with multiple license plates, occlusion, or varying lighting conditions, resulting in inaccurate license plate detection. The method uses a trained license plate key point detection model to predict key points in the license plate image data of the target vehicle, obtaining initial license plate key points. These initial key points are then optimized using preset geometric constraint rules to obtain optimized license plate key points. Finally, affine transformations are applied to the optimized key points to output corrected license plate key points. This method solves the problem of existing license plate detection methods, based on object detection or semantic segmentation, performing poorly in complex scenes, such as those with multiple license plates, occlusion, or varying lighting conditions, resulting in inaccurate license plate detection.

[0004] In a first aspect, embodiments of the present invention provide a method for detecting key points of multiple license plates, the method comprising the following steps:

[0005] Obtain the license plate image data of the target vehicle;

[0006] The license plate key points are obtained by predicting key points in the license plate image data using a trained license plate key point detection model.

[0007] The initial license plate key points are optimized by using preset geometric constraint rules to obtain optimized license plate key points.

[0008] The optimized license plate key points are then subjected to affine transformation and post-processing to output the corrected license plate key points.

[0009] Optionally, before performing key point prediction on the license plate image data using the trained license plate key point detection model to obtain initial license plate key points, the method further includes:

[0010] Obtain a sample training dataset and an untrained license plate key point detection model. The sample training dataset includes sample license plate image data, confidence label data of license plate key points corresponding to the sample license plate image data, and coordinate label data of license plate key points. The output of the untrained license plate key point detection model is the license plate key point prediction vector, the confidence data corresponding to the license plate key point prediction vector, and the coordinate data corresponding to the license plate key points.

[0011] The untrained license plate key point detection model is fine-tuned and trained using the sample training dataset. Once training is complete, a trained license plate key point detection model is obtained.

[0012] Optionally, the step of fine-tuning the untrained license plate key point detection model using the sample training dataset, and obtaining a trained license plate key point detection model after training, includes:

[0013] The sample license plate image data is input into the untrained license plate key point detection model, and the untrained license plate key point detection model outputs the confidence data of the license plate key points and the coordinate data of the license plate key points.

[0014] A first error loss is calculated between the confidence score data of the license plate key points and the confidence label data of the license plate key points using a first loss function; and a second error loss is calculated between the coordinate data of the license plate key points and the coordinate label data of the license plate key points using a second loss function.

[0015] The total error loss is obtained by weighting the first error loss and the second error loss.

[0016] With minimizing the total error loss as the optimization objective, the model parameters of the untrained license plate key point detection model are adjusted using the backpropagation algorithm. The adjustment process of the model parameters is iterated until the total loss is less than a preset value or the number of iterations reaches a preset number, at which point training stops, and a trained license plate key point detection model is obtained.

[0017] Optionally, the step of predicting key points in the license plate image data using a trained license plate key point detection model to obtain initial license plate key points includes:

[0018] The license plate image data is processed by feature extraction using a trained license plate key point detection model to obtain license plate image features;

[0019] The key point confidence scores of the license plate image features are predicted to obtain the license plate key point confidence scores.

[0020] The key point coordinates of the license plate image are predicted to obtain the key point coordinates of the license plate.

[0021] Based on the confidence level of the license plate key points and the coordinates of the license plate key points, the initial license plate key points are obtained.

[0022] Optionally, the optimization of the initial license plate key points using preset geometric constraint rules to obtain optimized license plate key points includes:

[0023] The initial license plate key points are edge aligned by applying preset geometric constraint rules to obtain the license plate key points after edge alignment constraint.

[0024] The license plate key points after edge alignment constraint are subjected to multi-license plate relative position constraint to obtain optimized license plate key points.

[0025] Optionally, the step of applying multi-license plate relative position constraints to the license plate key points after edge alignment constraints to obtain optimized license plate key points includes:

[0026] The license plate direction is detected for the key points of the license plate after the edge alignment constraint to determine whether the license plate direction is consistent.

[0027] If the license plate directions are consistent, then the spacing of the constrained license plate key points is constrained to obtain optimized license plate key points.

[0028] Optionally, the step of applying spacing constraints to the license plate key points after edge alignment constraints to obtain optimized license plate key points includes:

[0029] Apply a first arrangement spacing constraint to the license plate key points after the edge alignment constraint to obtain the license plate key points after the first sorting spacing constraint.

[0030] Apply a second arrangement spacing constraint to the license plate key points after the edge constraint to obtain the license plate key points after the second arrangement spacing constraint.

[0031] Secondly, embodiments of the present invention also provide a multi-license plate key point detection device, the multi-license plate key point detection device comprising:

[0032] The acquisition module is used to acquire the license plate image data of the target vehicle;

[0033] The prediction module is used to predict key points in the license plate image data using a trained license plate key point detection model to obtain initial license plate key points.

[0034] The first processing module is used to optimize the initial license plate key points according to preset geometric constraint rules to obtain optimized license plate key points.

[0035] The second processing module is used to perform affine transformation on the optimized license plate key points and output the corrected license plate key points.

[0036] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the multi-license plate key point detection method provided in embodiments of the present invention.

[0037] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the multi-license plate key point detection method provided in the embodiments of the present invention.

[0038] In this embodiment of the invention, license plate image data of a target vehicle is acquired; key points are predicted from the license plate image data using a trained license plate key point detection model to obtain initial license plate key points; the initial license plate key points are optimized using preset geometric constraint rules to obtain optimized license plate key points; and the optimized license plate key points are post-processed with an affine transformation to output corrected license plate key points. This invention solves the problem of inaccurate license plate detection in existing license plate detection methods based on object detection or semantic segmentation in complex scenes, such as multi-line license plates, occlusion, and changes in lighting. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a multi-license plate key point detection method provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of a multi-license plate key point detection device provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, Figure 1 This is a flowchart of a multi-license plate key point detection method provided by an embodiment of the present invention. The multi-license plate key point detection method includes the following steps:

[0045] 101. Obtain the license plate image data of the target vehicle.

[0046] In this embodiment of the invention, the aforementioned multi-license plate key point detection method can be applied to a license plate detection platform. This platform can be built on a server-based or distributed architecture and includes a data interface (for sensor or user uploads), a knowledge database, and a knowledge database construction program. The data interface can be used to acquire license plate image data of the target vehicle, and the knowledge database construction program can be used to construct the knowledge database. This knowledge database is specifically designed to provide additional relational information for the identified data entities, thereby enhancing the data recognition system's understanding of the content.

[0047] The aforementioned target vehicles can be understood as vehicles that require multiple license plate detection, such as buses and cars.

[0048] The aforementioned license plate image data can be understood as images of target vehicles containing license plate information captured by surveillance cameras. This license plate image data can be multi-license plate image data, which can contain multiple license plates, such as those found on Hong Kong double-decker buses and Macau commercial vehicles, which may have multiple rows of license plates arranged vertically or horizontally.

[0049] It is understandable that cross-border vehicles need to display license plates from both regions simultaneously, such as a Guangdong Z license plate plus a Hong Kong license plate.

[0050] It should be noted that Guangdong-Hong Kong-Macau license plates (such as Guangdong Z and FV plates) usually use a double-line plate or a combination of multiple plates (such as arranged vertically or horizontally). Taking Hong Kong plates as an example, Hong Kong plates are often single-line or double-line plates with letters and numbers; taking Macau plates as an example, Macau plates are generally 6 digits long, with a "-" symbol connecting them in the middle, and some plates use a two-line layout, etc.

[0051] 102. Predict key points of license plate images using a trained license plate key point detection model to obtain initial license plate key points.

[0052] In this embodiment of the invention, the trained license plate key point detection model can be a license plate key point detection model built based on deep learning or machine learning, such as ResNet, YOLOv5, etc. ResNet (Residual Network) solves the training difficulties and performance degradation problems caused by the increase in the number of layers in traditional deep networks. The core idea of ​​ResNet is to achieve identity mapping by introducing residual blocks, reducing the learning difficulty of the network. YOLOv5 (You Only Look Once version 5) is an object detection algorithm that completes the localization and recognition of objects in an image through a single scan. The core principle of YOLOv5 is to use a single-stage detection mechanism to directly predict bounding boxes and categories from the image without the need for region proposals.

[0053] The pre-trained license plate keypoint detection model described above is obtained by training a pre-trained license plate keypoint detection model on a license plate training dataset. The training dataset includes sample license plate images, confidence-level annotations of the license plate keypoints, and coordinate annotations of the keypoints. These annotations can be understood as adding structured labels to the raw data, enabling the machine learning model to recognize and process both the raw and labeled data. Through these annotations, the model establishes a mapping between the input data and the correct output labels. The pre-trained license plate keypoint detection model outputs the license plate keypoints. This pre-trained model can be a license plate keypoint detection model built based on deep learning or machine learning, such as ResNet or YOLOv5. The training can be supervised training, which uses a set of known-labeled data to train the model. By optimizing the model parameters, the model can predict the labels of new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to adjust the model parameters to minimize the difference between the model's output labels and the input data. The loss function described above measures the difference between the model's predictions and the actual results. Its purpose is to improve prediction accuracy by minimizing the loss function value through adjusting the model parameters. This loss function can be the mean squared error loss function, cross-entropy loss function, etc.

[0054] The trained license plate detection model described above can identify key points of the license plate in image data.

[0055] The aforementioned key point prediction can be understood as the process of identifying key points of license plates in license plate image data through a trained license plate key point detection model.

[0056] The initial license plate key points mentioned above can be obtained by predicting key points in the license plate image data using a trained license plate key point detection model. These key points can be the four corner points of the license plate.

[0057] It should be noted that a pre-trained license plate key point detection model can be used to predict key points on the input license plate image data, predicting the four corner points of the license plate, thereby obtaining the initial license plate key points.

[0058] 103. The initial license plate key points are optimized by using preset geometric constraint rules to obtain optimized license plate key points.

[0059] In this embodiment of the invention, the aforementioned preset geometric constraint rules can be pre-set geometric constraint rules by the system, used to guide and constrain the optimization processing of license plate key points. These preset geometric constraint rules can include license plate edge alignment constraints and multi-license plate relative position constraints to optimize the spatial distribution of license plate key points. The aforementioned license plate edge alignment constraint is used to filter out non-license plate areas, applying constraints only to quadrilaterals with aspect ratios ∈ [2.0, 5.0], forcing opposite sides of the license plate quadrilaterals to be parallel. The aforementioned multi-license plate relative position constraint is used when multiple license plates are detected, constraining the spacing and orientation consistency of the multiple license plates.

[0060] The above optimization process can be understood as constraining and optimizing the initial license plate key points through preset geometric constraint rules to obtain a more accurate license plate key point processing process.

[0061] The optimized license plate key points mentioned above can be understood as the license plate key points obtained by constraining and optimizing the initial license plate key points through preset geometric constraints.

[0062] It should be noted that preset geometric constraint rules can be used to constrain the aspect ratio of the four corner points of the license plate, forcing the opposite sides of the license plate quadrilateral to be parallel, and constraining the spacing and orientation consistency of multiple license plates when multiple license plates are detected. Understandably, the positions of key points on the license plate can be adjusted and optimized based on preset geometric constraint rules, thereby obtaining more accurate and optimized license plate key points.

[0063] 104. Perform affine transformation on the optimized license plate key points and output the corrected license plate key points.

[0064] In this embodiment of the invention, the above-described affine transformation post-processing can be understood as a transformation method that maps a set of points on one two-dimensional plane to a set of points on another two-dimensional plane. It is understood that affine transformations can change the shape and size of an image, but will not change the internal structure of the image.

[0065] The affine transformation post-processing described above is used to correct license plate tilt caused by shooting angle or distance issues. Specifically, by performing affine transformation post-processing on the optimized license plate key points, the license plate can be adjusted into a more regular shape.

[0066] The corrected license plate key points mentioned above can be obtained by performing an affine transformation on the optimized license plate key points. Furthermore, the corrected license plate key points can be used to identify the license plate data of a target vehicle.

[0067] In this embodiment of the invention, license plate image data of a target vehicle is acquired; key points are predicted from the license plate image data using a trained license plate key point detection model to obtain initial license plate key points; the initial license plate key points are optimized using preset geometric constraint rules to obtain optimized license plate key points; and the optimized license plate key points are post-processed with an affine transformation to output corrected license plate key points. This invention solves the problem of inaccurate license plate detection in existing license plate detection methods based on object detection or semantic segmentation in complex scenes, such as multi-line license plates, occlusion, and changes in lighting.

[0068] It is understood that in the specific implementation of this application, data such as image data, key point data, knowledge data, and license plate data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0069] Optionally, before the step of predicting key points in the license plate image data using the trained license plate key point detection model to obtain the initial license plate key points, a sample training dataset and an untrained license plate key point detection model can be obtained; the untrained license plate key point detection model can be fine-tuned using the sample training dataset, and after training is completed, a trained license plate key point detection model is obtained.

[0070] In this embodiment of the invention, the aforementioned sample training dataset includes sample license plate image data, confidence-level annotation data of the license plate key points corresponding to the sample license plate image data, and coordinate annotation data of the license plate key points. The annotation data can be understood as adding structured labels to the original data, enabling the machine learning model to recognize and process both the original data and the annotation data. Through the annotation data, the model can establish a mapping relationship between the input data and the correct output labels.

[0071] The output of the untrained license plate key point detection model is the license plate key point prediction vector, the confidence data corresponding to the license plate key point prediction vector, and the coordinate data corresponding to the license plate key point.

[0072] The aforementioned untrained license plate keypoint detection model can be a license plate keypoint detection model built based on machine learning or deep learning, such as ResNet or YOLOv5. ResNet (Residual Network) addresses the training difficulties and performance degradation issues caused by increasing the number of layers in traditional deep networks. The core idea of ​​ResNet is to achieve identity mapping by introducing residual blocks, reducing the learning difficulty of the network. YOLOv5 (You Only Look Once Version 5) is an object detection algorithm that completes the localization and recognition of objects in an image through a single scan. The core principle of YOLOv5 is to use a single-stage detection mechanism to directly predict bounding boxes and categories from the image, without requiring region proposals.

[0073] The fine-tuning training described above can be supervised training or model parameter fine-tuning. Supervised training uses a set of data with known labels to train the model, optimizing the model parameters so that the model can predict the labels of new data or make decisions based on the characteristics of existing data. During training, the minimum loss function can be used to adjust the model parameters to minimize the difference between the model's output label and the input data. The loss function measures the difference between the model's prediction and the true result, and its purpose is to improve prediction accuracy by minimizing the loss function value by adjusting the model parameters. The aforementioned loss function can be the mean squared error loss function, cross-entropy loss function, etc.

[0074] The trained license plate key point detection model described above can identify the key points of the license plate in the license plate image data.

[0075] Optionally, in the step of fine-tuning the untrained license plate key point detection model using a sample training dataset to obtain a trained license plate key point detection model, sample license plate image data can be input into the untrained license plate key point detection model. The untrained license plate key point detection model outputs the confidence data and coordinate data of the license plate key points. A first error loss is calculated between the confidence data and the confidence label data of the license plate key points using a first loss function. A second error loss is calculated between the coordinate data and the coordinate label data of the license plate key points using a second loss function. The total error loss is obtained by weighting the first and second error losses. With minimizing the total error loss as the optimization objective, the model parameters of the untrained license plate key point detection model are adjusted using a backpropagation algorithm. The adjustment process of the model parameters is iterated until the total loss is less than a preset value or the number of iterations reaches a preset number, at which point training stops, and a trained license plate key point detection model is obtained.

[0076] In this embodiment of the invention, the confidence data of the above-mentioned license plate key points can be the reliability or degree of certainty of the license plate key points.

[0077] The coordinate data of the above-mentioned key points of the license plate can be the coordinate positions of the four corner points of the license plate in space.

[0078] The first loss function mentioned above is used to calculate the error loss between the confidence score data and the labeled confidence score data of the license plate key points. This first loss function can be L1 Loss, which is a regression loss function used to measure the difference between predicted and true values. The error loss between the confidence score data and the labeled confidence score data of the license plate key points can be calculated using the L1 Loss function.

[0079] The first error loss can be the error loss between the confidence data of the license plate key points and the confidence labeling data of the license plate key points.

[0080] The second loss function mentioned above is used to calculate the error loss between the coordinate data of the license plate key points and the coordinate annotation data of the license plate key points. This second loss function can be IOU Loss, an Intersection over Union (IoU) loss function used for object detection that improves localization accuracy by optimizing the Intersection over Union (IoU) ratio between the predicted bounding box and the ground truth bounding box. IOU Loss can be used to calculate the error loss between the coordinate data of the license plate key points and the coordinate annotation data of the license plate key points.

[0081] The aforementioned second error loss can be the error loss between the coordinate data of the license plate key points and the coordinate annotation data of the license plate key points.

[0082] The above weighting can be understood as a calculation scheme that assigns corresponding weights to the first error loss and the second error loss based on their importance or influence.

[0083] The total error loss mentioned above is a measure of the difference between the model's predicted values ​​and the true values, used to guide the adjustment of model parameters to minimize the error.

[0084] In one possible implementation, adaptive weighted L1 loss can be used to assign higher weights to occluded, small, or blurred sample image data.

[0085]

[0086] in, For adaptive weighted L1 loss, N is the number of keypoints. The dynamic weight of the i-th key point. Let i be the true coordinates of the i-th key point. Let be the predicted coordinates of the i-th keypoint, where the weights are... The confidence level is dynamically adjusted based on the key point.

[0087] You can use IOU Loss to constrain the overlap between the predicted bounding box and the ground truth bounding box:

[0088]

[0089] in, For IOU LOSS, Let be the intersection-union ratio function. To predict the bounding box, This represents the true bounding box.

[0090] The total loss function is:

[0091]

[0092] in, and To balance the hyperparameters.

[0093] Specifically, the optimization objective can be minimizing the total error loss. This can be achieved by using the backpropagation algorithm to adjust the parameters of the untrained license plate keypoint detection model. This parameter adjustment process is iterated until the total error loss is less than a preset value, or the number of iterations reaches a preset number, at which point the training process ends, resulting in a trained license plate keypoint detection model. The backpropagation algorithm described above can be understood as updating weights by calculating the gradient of the loss function to minimize the loss.

[0094] In another possible embodiment, occluded, small-target, or blurred sample image data can be used as hard samples, and the confidence threshold for hard samples is τ:

[0095]

[0096] Here, τ is dynamically updated during the training process. After each round of model training, the confidence distribution of the current model on all samples on the validation set is recalculated, and the top 25% is taken as the new τ.

[0097] Weight The implementation can be done in the following ways:

[0098] First, the confidence level is mapped to a weight value, and different weights are dynamically assigned to different confidence levels using the following piecewise formula:

[0099]

[0100] in, This refers to the maximum weight of difficult samples (low confidence), with a value range of [2.0, 5.0]. This is the minimum weight for simple samples (high confidence), with a value range of [0.1, 0.5]; parameters and Initially, the weights are set to fixed values ​​of 2.0 and 0.2 respectively. After the Kth model is trained, the weights are dynamically adjusted, primarily based on the proportion of difficult samples in the validation set. K can be set to 1 / 3 of the total model training weights; as the proportion of difficult samples increases, the weights are increased accordingly. When the proportion of difficult samples decreases, then reduce... ;

[0101] It should be noted that the condition for mapping confidence scores to weight values ​​is as follows:

[0102] 1. Low confidence (difficult samples) → High weight

[0103] 2. High confidence (simple sample) —> Low weight

[0104] The trained license plate key point detection model described above can identify the key points of the license plate in the license plate image data.

[0105] Optionally, in the step of predicting key points in license plate image data using a trained license plate key point detection model to obtain initial license plate key points, the trained license plate key point detection model can be used to perform feature extraction processing on the license plate image data to obtain license plate image features; perform key point confidence prediction on the license plate image features to obtain license plate key point confidence; perform key point coordinate prediction on the license plate image features to obtain license plate key point coordinates; and obtain the initial license plate key points based on the license plate key point confidence and license plate key point coordinates.

[0106] In this embodiment of the invention, the pre-trained license plate keypoint detection model can be a license plate keypoint detection model built based on deep learning or machine learning, such as ResNet, YOLOv5, etc. The pre-trained license plate keypoint detection model is obtained by training a pre-trained license plate keypoint detection model on a license plate training dataset. The license plate training dataset includes sample license plate image data, confidence annotation data of license plate keypoints in the sample license plate image data, and coordinate position annotation data of the license plate keypoints. The annotation data can be understood as adding structured labels to the original data, enabling the machine learning model to recognize and process the original data and the annotation data. Through the annotation data, the model can establish a mapping relationship between the input data and the correct output labels. The pre-trained license plate keypoint detection model outputs the license plate keypoints. The pre-trained license plate keypoint detection model can be a license plate keypoint detection model built based on deep learning or machine learning, such as ResNet, YOLOv5, etc. The training described above can be supervised training. Supervised training uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the labels of new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to adjust the model parameters to minimize the difference between the model's output label and the input data. The loss function mentioned above measures the difference between the model's prediction and the true result. Its purpose is to improve prediction accuracy by minimizing the loss function value by adjusting the model parameters. The loss function can be the mean squared error loss function, cross-entropy loss function, etc.

[0107] The aforementioned license plate image data can be images of target vehicles containing license plate information captured by surveillance cameras. This license plate image data can also be multi-license plate image data, which includes images of multiple license plates, such as those found on Hong Kong double-decker buses and Macau commercial vehicles, which may have multiple rows of license plates arranged vertically or horizontally.

[0108] The above feature extraction process can be understood as the process of extracting relevant features of the license plate from the license plate image data, such as extracting edges and colors.

[0109] The aforementioned license plate image features can be understood as license plate features extracted from license plate image data.

[0110] The aforementioned key point confidence prediction can be understood as the process of predicting the confidence of license plate key points in image features. These license plate key points can be the four corner points of the license plate.

[0111] The above-mentioned license plate key point confidence level is used to measure the reliability of the predicted key points.

[0112] The above key point coordinate prediction can be understood as the process of predicting the coordinates of key points of the license plate in the image features.

[0113] The aforementioned key point coordinates of the license plate can be the coordinate information of the four corner points of the license plate.

[0114] The aforementioned initial license plate key points can be obtained based on the confidence level and coordinates of the license plate key points.

[0115] It should be noted that a trained license plate key point detection model can be used to extract features from license plate image data to obtain license plate image features. Key point confidence can be predicted from these features to obtain the license plate key point confidence. Key point coordinates can be predicted from these features to obtain the license plate key point coordinates. Based on the license plate key point confidence and coordinates, initial license plate key points can be obtained, which can improve the accuracy of license plate key point recognition.

[0116] Optionally, in the step of optimizing the initial license plate key points through preset geometric constraint rules to obtain optimized license plate key points, the initial license plate key points can be edge-aligned by preset geometric constraint rules to obtain edge-aligned license plate key points; and the relative position constraints of multiple license plates can be applied to the edge-aligned license plate key points to obtain optimized license plate key points.

[0117] In this embodiment of the invention, the aforementioned preset geometric constraint rules can be pre-set geometric constraint rules by the system, used to guide and constrain the optimization processing of license plate key points. These preset geometric constraint rules can be license plate edge alignment constraints and relative position constraints of multiple license plates, optimizing the spatial distribution of license plate key points.

[0118] The above license plate edge alignment constraint is to filter out non-license plate areas, and only applies constraints to quadrilaterals with aspect ratios ∈ [2.0, 5.0], forcing the opposite sides of the license plate quadrilateral to be parallel.

[0119] Edge alignment constraints can be applied to the initial license plate key points by minimizing the formula for calculating the opposite side angle deviation, forcing the opposite sides of the quadrilateral formed by the license plate key points to be parallel. (Opposite side angle deviation) The calculation formula is as follows:

[0120] |

[0121] in, It is the angle between adjacent sides.

[0122] It is understandable that there is an angle deviation on opposite sides. The smaller the value, the closer the opposite sides of the license plate's key points are to parallel, meaning it resembles a rectangle more. The larger the difference, the further away the opposite side of the license plate key point is from the parallel.

[0123] It should be noted that the edge alignment preset needs to meet one of the following conditions to be activated:

[0124] (1) Predict the initial IOU of the quadrilateral > Set the threshold to 0.3;

[0125] (2) Key point confidence < Set threshold 0.3.

[0126] The aforementioned relative position constraints of multiple license plates are implemented when multiple license plates are detected, and the spacing and orientation of the license plates are kept consistent.

[0127] It should be noted that edge alignment constraints can be applied to the initial license plate key points using preset geometric constraint rules to obtain the license plate key points after edge alignment constraints. When multiple license plates are detected as the license plate key points after edge alignment constraints, the relative positions of the multiple license plates can be constrained to obtain optimized license plate key points, which can improve the accuracy of license plate key point detection.

[0128] Optionally, in the step of constraining the relative positions of multiple license plates to obtain optimized license plate key points after edge alignment constraint, license plate direction detection can be performed on the license plate key points after edge alignment constraint to determine whether the license plate directions are consistent; if the license plate directions are consistent, then spacing constraint can be applied to the license plate key points after edge alignment constraint to obtain optimized license plate key points.

[0129] In this embodiment of the invention, when multiple license plate key points are detected after edge alignment constraint, license plate direction detection can be performed on the license plate key points after edge alignment constraint to determine whether the license plate directions are consistent.

[0130] The above license plate direction detection can be understood as a process of detecting the license plate direction of key points after edge alignment constraints in order to determine whether the license plate direction is consistent.

[0131] It should be noted that, in order to quickly eliminate angular anomalies, license plate direction detection can be performed on the key points of the license plate after edge alignment constraint to determine whether the license plate direction is consistent.

[0132] In one possible implementation, for example, in a double-lane license plate configuration, the angles of the two license plates are θ1 and θ2, respectively. To check whether the two license plates are tilted in the same direction, the following conditions must be met simultaneously: ① θ1*θ2 > 0; ② The absolute value of the rotation angle of the two license plates, ∆θ = |θ1 - θ2|, must be calculated, and ∆θ < θ_threshold, where θ_threshold refers to the set angle judgment threshold, and the default setting is θ_threshold = 5°. When θ1 and θ2 have the same sign and the difference is less than the threshold, they are determined to be in the same direction.

[0133] Furthermore, when the license plate orientation is consistent, the spacing of the key points of the license plate after edge alignment constraint can be constrained to obtain the optimized license plate key points.

[0134] The aforementioned spacing constraint can be understood as the process of applying spacing constraints to the arrangement of license plate key points after edge alignment constraints. It is understood that spacing constraints can prevent license plate overlap or excessive spacing.

[0135] Optionally, in the step of applying spacing constraints to the license plate key points after edge alignment constraints to obtain optimized license plate key points, a first arrangement spacing constraint can be applied to the license plate key points after edge alignment constraints to obtain license plate key points after first sorting spacing constraints; and a second arrangement spacing constraint can be applied to the license plate key points after edge constraints to obtain license plate key points after second sorting spacing constraints.

[0136] In this embodiment of the invention, the first arrangement spacing constraint can be a left-right arrangement constraint or a top-bottom arrangement constraint. The second arrangement spacing constraint can be a top-bottom arrangement constraint or a left-right arrangement constraint. It should be noted that when the first and second arrangement spacing constraints are different arrangement constraints, the second arrangement spacing constraint is a top-bottom arrangement constraint when the first arrangement spacing constraint is a left-right arrangement constraint, and vice versa.

[0137] Furthermore, the key points of the license plate after edge alignment can be constrained to be arranged horizontally to obtain the key points of the license plate after horizontal spacing constraints; the key points of the license plate after edge alignment can be constrained to be arranged vertically to obtain the key points of the license plate after vertical spacing constraints.

[0138] It should be noted that the conditions for left and right sorting constraints are: (1) the horizontal distance between the center points of the two license plates ∆x = |x1-x2|, w_avg < ∆x < 2 * w_avg, where w_avg is the average width of the two license plates; (2) the vertical offset between the center points of the two license plates ∆y = |y1 - y2|, ∆y < 0.2 * h_avg, where h_avg is the average height of the two license plates.

[0139] The conditions for the vertical sorting constraint are: (1) the vertical offset of the center points of the two license plates is ∆y = |y1 - y2|, h_avg<∆y < 1.5 * h_avg, where h_avg is the average height of the two license plates; (2) the horizontal offset of the center points of the two license plates is ∆x < 0.2 * w_avg, where w_avg is the average width of the two license plates.

[0140] In this embodiment of the invention, the invention optimizes the spatial distribution of license plate key points through preset geometric constraint rules, enhances the robustness of multi-license plate detection, and can be applied to complex scenarios such as multi-line license plates and tilted license plates. It solves the problem that existing license plate detection methods based on object detection or semantic segmentation perform poorly in complex scenarios, such as multi-line license plates, occlusion, and changes in lighting, resulting in inaccurate license plate detection.

[0141] like Figure 2 As shown, this embodiment of the invention provides a multi-license plate key point detection device, which includes:

[0142] The acquisition module 201 is used to acquire the license plate image data of the target vehicle;

[0143] Prediction module 202 is used to predict key points of the license plate image data using a trained license plate key point detection model to obtain initial license plate key points;

[0144] The first processing module 203 is used to optimize the initial license plate key points through preset geometric constraint rules to obtain optimized license plate key points.

[0145] The second processing module 204 is used to perform affine transformation on the optimized license plate key points and output the corrected license plate key points.

[0146] Optionally, the device is further configured to acquire a sample training dataset and an untrained license plate key point detection model. The sample training dataset includes sample license plate image data, confidence label data of the license plate key points corresponding to the sample license plate image data, and coordinate label data of the license plate key points. The output of the untrained license plate key point detection model is the license plate key point prediction vector and the confidence data corresponding to the license plate key point prediction vector, as well as the coordinate data corresponding to the license plate key points. The untrained license plate key point detection model is fine-tuned and trained using the sample training dataset. After training is completed, a trained license plate key point detection model is obtained.

[0147] Optionally, the device is further configured to input the sample license plate image data into the untrained license plate key point detection model, output the confidence data and coordinate data of the license plate key points through the untrained license plate key point detection model; calculate a first error loss between the confidence data and the confidence label data of the license plate key points using a first loss function; and calculate a second error loss between the coordinate data and the coordinate label data of the license plate key points using a second loss function; obtain a total error loss by weighting the first error loss and the second error loss; and adjust the model parameters of the untrained license plate key point detection model using a backpropagation algorithm with the goal of minimizing the total error loss, iterating the adjustment process of the model parameters until the total loss is less than a preset value or the number of iterations reaches a preset number, then stop training to obtain a trained license plate key point detection model.

[0148] Optionally, the prediction module 202 is further configured to perform feature extraction processing on the license plate image data using a trained license plate key point detection model to obtain license plate image features; perform key point confidence prediction on the license plate image features to obtain license plate key point confidence; perform key point coordinate prediction on the license plate image features to obtain license plate key point coordinates; and obtain initial license plate key points based on the license plate key point confidence and the license plate key point coordinates.

[0149] Optionally, the second processing module 203 is further configured to perform edge alignment constraints on the initial license plate key points using preset geometric constraint rules to obtain edge-aligned license plate key points; and to perform multi-license plate relative position constraints on the edge-aligned license plate key points to obtain optimized license plate key points.

[0150] Optionally, the second processing module 203 is further configured to perform license plate direction detection on the license plate key points after edge alignment constraint to determine whether the license plate direction is consistent; if the license plate direction is consistent, then the spacing constraint is applied to the constrained license plate key points to obtain optimized license plate key points.

[0151] Optionally, the second processing module 203 is further configured to apply a first arrangement spacing constraint to the license plate key points after edge alignment constraint to obtain license plate key points after first arrangement spacing constraint; and apply a second arrangement spacing constraint to the license plate key points after edge constraint to obtain license plate key points after second arrangement spacing constraint.

[0152] like Figure 3 As shown, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-described multi-license plate key point detection methods.

[0153] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301 to execute the multi-license plate key point detection method, wherein:

[0154] The processor 301 executes the calculator program for the multi-license plate key point detection method stored in the runtime memory 302, and performs the following steps:

[0155] Obtain the license plate image data of the target vehicle;

[0156] The license plate key points are obtained by predicting key points in the license plate image data using a trained license plate key point detection model.

[0157] The initial license plate key points are optimized by using preset geometric constraint rules to obtain optimized license plate key points.

[0158] The optimized license plate key points are then subjected to affine transformation and post-processing to output the corrected license plate key points.

[0159] Optionally, before the initial license plate key points are obtained by predicting key points in the license plate image data using the trained license plate key point detection model, the method executed by the processor 301 further includes:

[0160] Obtain a sample training dataset and an untrained license plate key point detection model. The sample training dataset includes sample license plate image data, confidence label data of license plate key points corresponding to the sample license plate image data, and coordinate label data of license plate key points. The output of the untrained license plate key point detection model is the license plate key point prediction vector, the confidence data corresponding to the license plate key point prediction vector, and the coordinate data corresponding to the license plate key points.

[0161] The untrained license plate key point detection model is fine-tuned and trained using the sample training dataset. Once training is complete, a trained license plate key point detection model is obtained.

[0162] Optionally, the processor 301 executes the step of fine-tuning the untrained license plate key point detection model using the sample training dataset. Upon completion of training, a trained license plate key point detection model is obtained, including:

[0163] The sample license plate image data is input into the untrained license plate key point detection model, and the untrained license plate key point detection model outputs the confidence data of the license plate key points and the coordinate data of the license plate key points.

[0164] A first error loss is calculated between the confidence score data of the license plate key points and the confidence label data of the license plate key points using a first loss function; and a second error loss is calculated between the coordinate data of the license plate key points and the coordinate label data of the license plate key points using a second loss function.

[0165] The total error loss is obtained by weighting the first error loss and the second error loss.

[0166] With minimizing the total error loss as the optimization objective, the model parameters of the untrained license plate key point detection model are adjusted using the backpropagation algorithm. The adjustment process of the model parameters is iterated until the total loss is less than a preset value or the number of iterations reaches a preset number, at which point training stops, and a trained license plate key point detection model is obtained.

[0167] Optionally, the processor 301 executes the step of predicting key points in the license plate image data using a trained license plate key point detection model to obtain initial license plate key points, including:

[0168] The license plate image data is processed by feature extraction using a trained license plate key point detection model to obtain license plate image features;

[0169] The key point confidence scores of the license plate image features are predicted to obtain the license plate key point confidence scores.

[0170] The key point coordinates of the license plate image are predicted to obtain the key point coordinates of the license plate.

[0171] Based on the confidence level of the license plate key points and the coordinates of the license plate key points, the initial license plate key points are obtained.

[0172] Optionally, the processor 301 performs optimization processing on the initial license plate key points using preset geometric constraint rules to obtain optimized license plate key points, including:

[0173] The initial license plate key points are edge aligned by applying preset geometric constraint rules to obtain the license plate key points after edge alignment constraint.

[0174] The license plate key points after edge alignment constraint are subjected to multi-license plate relative position constraint to obtain optimized license plate key points.

[0175] Optionally, the processor 301 performs multi-license plate relative position constraints on the license plate key points after edge alignment constraints to obtain optimized license plate key points, including:

[0176] The license plate direction is detected for the key points of the license plate after the edge alignment constraint to determine whether the license plate direction is consistent.

[0177] If the license plate directions are consistent, then the spacing of the constrained license plate key points is constrained to obtain optimized license plate key points.

[0178] Optionally, the processor 301 performs the following step: applying spacing constraints to the license plate key points after edge alignment constraints to obtain optimized license plate key points, including:

[0179] Apply a first arrangement spacing constraint to the license plate key points after the edge alignment constraint to obtain the license plate key points after the first sorting spacing constraint.

[0180] Apply a second arrangement spacing constraint to the license plate key points after the edge constraint to obtain the license plate key points after the second arrangement spacing constraint.

[0181] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the multi-license plate key point detection method provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0182] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0183] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for detecting key points of multiple license plates, characterized in that, The method includes the following steps: Obtain the license plate image data of the target vehicle; The license plate key points are obtained by predicting key points in the license plate image data using a trained license plate key point detection model. The initial license plate key points are optimized by using preset geometric constraint rules to obtain optimized license plate key points. The optimized license plate key points are then subjected to affine transformation and post-processing to output the corrected license plate key points.

2. The multi-license plate key point detection method as described in claim 1, characterized in that, Before performing key point prediction on the license plate image data using the trained license plate key point detection model to obtain initial license plate key points, the method further includes: Obtain a sample training dataset and an untrained license plate key point detection model. The sample training dataset includes sample license plate image data, confidence label data of license plate key points corresponding to the sample license plate image data, and coordinate label data of license plate key points. The output of the untrained license plate key point detection model is the license plate key point prediction vector, the confidence data corresponding to the license plate key point prediction vector, and the coordinate data corresponding to the license plate key points. The untrained license plate key point detection model is fine-tuned and trained using the sample training dataset. Once training is complete, a trained license plate key point detection model is obtained.

3. The multi-license plate key point detection method as described in claim 2, characterized in that, The step involves fine-tuning the untrained license plate key point detection model using the sample training dataset. Once training is complete, a trained license plate key point detection model is obtained, including: The sample license plate image data is input into the untrained license plate key point detection model, and the untrained license plate key point detection model outputs the confidence data of the license plate key points and the coordinate data of the license plate key points. A first error loss is calculated between the confidence score data of the license plate key points and the confidence label data of the license plate key points using a first loss function; and a second error loss is calculated between the coordinate data of the license plate key points and the coordinate label data of the license plate key points using a second loss function. The total error loss is obtained by weighting the first error loss and the second error loss. With minimizing the total error loss as the optimization objective, the model parameters of the untrained license plate key point detection model are adjusted using the backpropagation algorithm. The adjustment process of the model parameters is iterated until the total loss is less than a preset value or the number of iterations reaches a preset number, at which point training stops, and a trained license plate key point detection model is obtained.

4. The multi-license plate key point detection method as described in claim 3, characterized in that, The step of predicting key points in the license plate image data using a trained license plate key point detection model to obtain initial license plate key points includes: The license plate image data is processed by feature extraction using a trained license plate key point detection model to obtain license plate image features; The key point confidence scores of the license plate image features are predicted to obtain the license plate key point confidence scores. The key point coordinates of the license plate image are predicted to obtain the key point coordinates of the license plate. Based on the confidence level of the license plate key points and the coordinates of the license plate key points, the initial license plate key points are obtained.

5. The multi-license plate key point detection method as described in claim 1, characterized in that, The optimization process, which optimizes the initial license plate key points using preset geometric constraint rules, yields optimized license plate key points, including: The initial license plate key points are edge aligned by applying preset geometric constraint rules to obtain the license plate key points after edge alignment constraint. The license plate key points after edge alignment constraint are subjected to multi-license plate relative position constraint to obtain optimized license plate key points.

6. The multi-license plate key point detection method as described in claim 5, characterized in that, The step of applying multi-license plate relative position constraints to the license plate key points after edge alignment constraints to obtain optimized license plate key points includes: The license plate direction is detected for the key points of the license plate after the edge alignment constraint to determine whether the license plate direction is consistent. If the license plate directions are consistent, then the spacing of the constrained license plate key points is constrained to obtain optimized license plate key points.

7. The multi-license plate key point detection method as described in claim 6, characterized in that, The step of applying spacing constraints to the license plate key points after edge alignment constraints to obtain optimized license plate key points includes: Apply a first arrangement spacing constraint to the license plate key points after the edge alignment constraint to obtain the license plate key points after the first sorting spacing constraint. Apply a second arrangement spacing constraint to the license plate key points after the edge constraint to obtain the license plate key points after the second arrangement spacing constraint.

8. A multi-license plate key point detection device, characterized in that, The multi-license plate key point detection device includes: The acquisition module is used to acquire the license plate image data of the target vehicle; The prediction module is used to predict key points in the license plate image data using a trained license plate key point detection model to obtain initial license plate key points. The first processing module is used to optimize the initial license plate key points according to preset geometric constraint rules to obtain optimized license plate key points. The second processing module is used to perform affine transformation on the optimized license plate key points and output the corrected license plate key points.

9. An electronic device, characterized in that, include: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the multi-license plate key point detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-license plate key point detection method as described in any one of claims 1 to 7.

Images