A method, system, device and storage medium for one-stage license plate image extraction

Abstract

This invention discloses a one-stage license plate image extraction method, system, device, and storage medium. The extraction method includes: acquiring image data; performing corner prediction on the image data based on a corner detection model to obtain output results, which include an association embedding prediction map, a corner prediction map, and a corner offset prediction map; obtaining the basic coordinates of the corners based on the corner prediction map; extracting the corner offset amount according to the corner offset prediction map to correct the basic coordinates of the corners; and combining the association embedding prediction map to perform association vector aggregation on the corners to form the license plate position. This invention directly extracts the license plate region using a single model, alleviating the speed reduction problem caused by combining multiple models; and directly obtains the pixel position of the license plate corner image through the corner prediction map output by the model, solving the accuracy and stability problems of regression affine transformation coefficients and perspective transformation coefficients.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation image processing, and in particular to a one-stage license plate image extraction method, system, device, and storage medium. Background Technology

[0002] License plate image extraction, which automatically obtains license plate information, is an important component of intelligent transportation. Currently, license plate image extraction methods are mainly divided into two categories: The first is traditional methods, which use manual features such as color, texture, and edge features to extract image information and locate the license plate. This method is greatly affected by external factors, such as lighting and complex backgrounds. The second category is deep learning methods, which mainly detect the license plate location by developing a model to learn effective features from a large amount of data. This method primarily uses a license plate detector to detect the approximate location of the license plate, and then accurately extracts the four corner points of the license plate from the detected image.

[0003] Currently, deep learning methods generally employ a combination of dual or multiple models to extract license plate images. However, combining multiple models reduces computational speed, resulting in no improvement in license plate extraction efficiency. Furthermore, if the camera captures the license plate image from a large angle, it can cause significant distortion. If existing regression affine coefficient methods are used to extract corner points, the correction results will be poor due to the large distortion. While existing regression perspective transformation methods can solve the problem of large license plate distortion, their stability is poor. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, one of the objectives of this invention is to provide a one-stage license plate image extraction method that improves the speed of license plate extraction while also improving the accuracy and stability of license plate extraction.

[0005] The second objective of this invention is to provide a one-stage license plate image extraction system that performs the aforementioned one-stage license plate image extraction method.

[0006] The third objective of this invention is to provide an electronic device.

[0007] The fourth objective of this invention is to provide a storage medium.

[0008] One of the objectives of this invention is achieved through the following technical solution:

[0009] A one-stage license plate image extraction method includes:

[0010] Image data is acquired, and corner prediction is performed on the image data based on a corner detection model to obtain output results, including an association embedding prediction map, a corner prediction map, and a corner offset prediction map.

[0011] The corner coordinates are obtained based on the corner prediction map. The corner offset is extracted according to the corner offset prediction map to correct the corner coordinates. The corner is then combined with the association embedding prediction map to perform association vector aggregation to form the license plate position.

[0012] Furthermore, after acquiring the image data, the process also includes:

[0013] The image data is converted into an image of a specified size using bilinear interpolation, and the converted image of the specified size is then subjected to mean removal and normalization.

[0014] Furthermore, the corner detection model is trained using a convolutional neural network, and the training method is as follows:

[0015] Obtain a training dataset, and perform feature extraction on the training dataset to obtain a feature map;

[0016] Feature maps of different sizes are fused to obtain the association embedding prediction map, the corner prediction map, and the corner offset prediction map;

[0017] The model loss function is determined by combining the association embedding prediction map, the corner prediction map, and the corner offset prediction map obtained during training. The model loss function is then optimized using a gradient-based optimization algorithm to obtain the optimized model parameters and thus the corner detection model.

[0018] Furthermore, the model loss function is:

[0019] L = L k +L o +L AE ;

[0020] Among them, L k L o L AE These refer to the license plate corner loss function, the bias loss function, and the association embedding loss function, respectively.

[0021] Furthermore, the method for obtaining the basic coordinates of the corner points based on the corner prediction map is as follows:

[0022] Max pooling is performed on the corner prediction map. The pooling result is subtracted from the corner prediction map element by element to obtain the position where the subtracted element value is zero. The coordinates corresponding to this position are used as the basic coordinates of the corner.

[0023] Furthermore, the method for correcting the basic coordinates of the corner points is as follows:

[0024] The corner offset corresponding to the corner base coordinates is extracted from the corner offset prediction map, and the corner offset is combined with the corner base coordinates to obtain the offset-corrected corner position.

[0025] The offset-corrected corner position is multiplied by the feature map downsampling factor to obtain the corrected corner coordinates.

[0026] Furthermore, the method for aggregating the association vectors of corner points using the aforementioned association embedding prediction graph is as follows:

[0027] Extract the association embedding vector of the corner point corresponding to the basic coordinates of the corner point from the association embedding prediction map;

[0028] Calculate the Euclidean distance between each associated embedding vector and take the point with the smallest Euclidean distance as the corner point of the same license plate.

[0029] The second objective of this invention is achieved by the following technical solution:

[0030] A one-stage license plate image extraction system, performing the one-stage license plate image extraction method described above; the system includes:

[0031] The corner prediction module is used to acquire image data and perform corner prediction on the image data based on the corner detection model to obtain output results, including an association embedding prediction map, a corner prediction map, and a corner offset prediction map.

[0032] The corner correction module is used to obtain the basic coordinates of the corner based on the corner prediction map, extract the corner offset according to the corner offset prediction map to correct the basic coordinates of the corner, and combine the associated embedding prediction map to perform associated vector aggregation on the corner to form the license plate position.

[0033] The third objective of this invention is achieved by the following technical solution:

[0034] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned one-stage license plate image extraction method.

[0035] The fourth objective of this invention is achieved by the following technical solution:

[0036] A storage medium storing a computer program that, when executed, implements the aforementioned one-stage license plate image extraction method.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention directly extracts the license plate region using a single model, alleviating the speed reduction problem caused by combining multiple models. It detects the license plate corner positions based on a corner detection model obtained through offline training, and corrects the license plate by combining the output results. The pixel positions of the license plate corner images are directly obtained through the corner prediction map output by the model, which solves the accuracy and stability problems of regression affine transformation coefficients and perspective transformation coefficients. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the one-stage license plate image extraction method of the present invention;

[0040] Figure 2 This is a schematic diagram of the training process of the corner detection model of the present invention;

[0041] Figure 3 This is a model structure diagram of the corner detection model of the present invention;

[0042] Figure 4 This is a schematic diagram illustrating the method for obtaining license plate positions based on a corner detection model according to the present invention.

[0043] Figure 5 This is a schematic diagram of the modules of the first-stage license plate image extraction system of the present invention. Detailed Implementation

[0044] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0045] Example 1

[0046] This embodiment provides a one-stage license plate image extraction method that integrates coarse and fine license plate region extraction. It directly extracts the license plate region using only a single neural network model, achieving faster detection speed compared to dual-model computation. Furthermore, this embodiment improves the detection method for license plate corner points by obtaining all corner point positions through finding local maxima in a heatmap and combining these with correlation vectors to aggregate corner points and form the license plate location. This improves the accuracy and stability of corner point coordinates, enhancing the overall detection performance.

[0047] refer to Figure 1 As shown, the one-stage license plate image extraction method in this embodiment specifically includes the following steps:

[0048] Step S1: Acquire image data, and perform corner prediction on the image data based on the corner detection model to obtain output results, including the association embedding prediction map, the corner prediction map, and the corner offset prediction map.

[0049] Step S2: Obtain the basic coordinates of the corner points based on the corner point prediction map, extract the corner point offset according to the corner point offset prediction map to correct the basic coordinates of the corner points, and combine the correlation embedding prediction map to perform correlation vector aggregation on the corner points to form the license plate position.

[0050] This embodiment uses a camera installed on the roadside or at the entrance / exit of a parking lot to capture images of the license plates of passing vehicles, thereby obtaining image data containing license plate information. After acquiring the image data, it needs to be preprocessed. The preprocessing method is as follows:

[0051] The image data is converted into an image of a specified size using bilinear interpolation. In this embodiment, the image data is converted into a 512x512 image. Subsequently, the converted image of the specified size is subjected to mean removal and normalization processing in order to standardize the image.

[0052] The corner detection model described in this embodiment is pre-trained offline using a convolutional neural network. The offline loop initializes the model parameters in the neural network, and once the model parameters are determined, offline training is no longer necessary. In this embodiment, the pre-processed image is imported into the pre-trained corner detection model for online prediction. In practical applications, the initialized model parameters are used to calculate and predict the result, and the four corners of the license plate are obtained after relevant post-processing.

[0053] refer to Figure 2 , Figure 3 As shown, the offline training method for the corner detection model in this embodiment is as follows:

[0054] Step S11: Obtain the training dataset and perform feature extraction on the training dataset to obtain a feature map;

[0055] In this embodiment, vehicle image data is collected, and the four corner points of the license plate are marked in the image to construct the training dataset. This training dataset is then used to train a neural network to obtain the corner detection model. In this embodiment, the corner detection model includes a feature extraction module and a feature fusion module. The feature extraction model is used to extract feature information from the image; specifically, it involves performing multiple iterative convolution operations and downsampling operations on the input to obtain feature maps of different sizes.

[0056] Step S12: Fuse feature maps of different sizes to obtain output results, which include association embedding prediction map, corner prediction map and corner offset prediction map;

[0057] In this embodiment, feature maps of different sizes obtained by the feature extraction module are fused layer by layer in the feature fusion module, and the model prediction result is output. Specifically:

[0058] Step S121: Sort the feature maps obtained by the feature extraction module according to their size, and select the feature map with the smallest size as the input;

[0059] Step S122: Apply deconvolution to the input to upsample the size of the feature map and convert it to the size of the next feature map. Then, add the upsampled feature map to the next size feature map pixel by pixel to obtain the fused feature map.

[0060] Step S123: Using the fused feature map as input, repeat step S122 until all feature maps have been fused. Then execute step S124.

[0061] Step S124: Apply a 1x1 convolution to the fused feature map to obtain the output results, which include the association embedding prediction map, the corner prediction map, and the corner offset prediction map.

[0062] Step S13: Determine the model loss function based on the output results, optimize the model loss function based on the gradient optimization algorithm, and obtain the optimized model parameters to obtain the corner detection model.

[0063] Since the license plate corner detection model is composed of a convolutional neural network, the parameters in the network need to be determined through training. The loss function for training is determined by comparing the predicted results of the detection model with the actual results pixel by pixel. The formula for the model's loss function is:

[0064] L = L k +L o +L AE ;

[0065] In the above formula, L k L o L AE These refer to the license plate corner loss function, the bias loss function, and the association embedding loss function, respectively.

[0066] Wherein, the license plate corner loss function L k The formula is:

[0067]

[0068] Y xyk This represents the confidence value of the corner points in the label, Y. xyk =1 is a corner point; This is the corner prediction map result from the model; α and β are hyperparameters used to adjust the training weight. The license plate corner loss function is used to optimize the confidence of the corner, so that the predicted value of the corner position is higher than that of the surrounding points.

[0069] Secondly, the bias loss function L o The expression is:

[0070]

[0071] Where p represents the actual keypoint locations within the original input image. To shrink the feature map to the original image The true location of the key points To predict the location of key points in the feature map; The bias is used to offset the predicted keypoints. The bias loss function is used to mitigate the keypoint position shift caused by image downsampling. When the model outputs the association embedding prediction map, corner prediction map, and corner offset prediction map are the same size as the input image, L2 optimization is not required. o .

[0072] Furthermore, the association embedding loss function L AE By L pull and L push It consists of two parts, and its expression is:

[0073] L AE =L push + pull ;

[0074]

[0075]

[0076] Where k is the number of corner points of the license plate, n is the number of license plates in the diagram, and h nk For the predicted association embedding vector, L is the average vector of the predicted embedding vectors for the same license plate corner point. pull This will reduce the Euclidean distance of the associative embedding vectors of corner points belonging to the same license plate, L push This increases the Euclidean distance between the embedding vectors of different license plates. Therefore, by calculating the Euclidean distance between the vectors, the detected corner points are distinguished into different license plates.

[0077] In this embodiment, after determining the model loss function, a gradient-based optimization algorithm is used to optimize the model loss function and obtain the optimized model parameters. The essence of the gradient-based optimization algorithm is to approximate the loss function using a first-order Taylor expansion, then find the minimum value of this approximated function, and use the minimum value as the value for the next iteration. This optimization algorithm is existing technology and will not be described in detail here.

[0078] In this embodiment, the model parameters of the corner detection model can be determined through steps S11 to S13. (Reference) Figure 4 As shown, when an image from a real-world application scenario is input into the corner detection model, three prediction results are obtained: an association embedding prediction map, a corner prediction map, and a corner offset prediction map.

[0079] The corner prediction map is used to obtain the basic coordinates of the corner points, specifically as follows:

[0080] The corner prediction map is pooled using a max pooling layer with a size of 3 and a step size of 1 to obtain the pooled result. The pooled result is then subtracted from the corner prediction map element by element. The position where the subtracted element value is 0 is the local maximum value, and the coordinates corresponding to this position are used as the basic coordinates of the corner.

[0081] In this embodiment, the corner offset prediction map is used to correct the corner base coordinates. Specifically, the corner offset corresponding to the corner base coordinates is extracted from the corner offset prediction map, and the corner offset is added to the corner base coordinates to obtain the offset-corrected corner position. Then, the offset-corrected corner position is multiplied by the feature map downsampling factor to obtain the corrected corner coordinates in the input image.

[0082] In this embodiment, the association embedding prediction map is used for corner point classification. Specifically, the association embedding vector of the corner point corresponding to the basic coordinates of the corner point is extracted from the association embedding prediction map; and by calculating the Euclidean distance between each association embedding vector, the point with the smallest Euclidean distance is taken as the corner point of the same license plate, so that the four corner points in the license plate image can be accurately identified.

[0083] This embodiment proposes to use a one-stage license plate region detection method to replace the two-stage license plate region detection, so as to reduce time consumption and improve the efficiency of license plate feature extraction; and proposes a new corner detection method to make up for the shortcomings of the current corner detection methods, improve the accuracy of license plate corner detection and the stability of corner coordinates, and improve the detection effect.

[0084] Example 2

[0085] This embodiment provides a one-stage license plate image extraction system, which executes the one-stage license plate image extraction method as described in Embodiment 1; see reference Figure 5 As shown, the license plate image extraction system includes:

[0086] The model training module is used to construct a training dataset and train a corner detection module based on the training dataset.

[0087] The corner prediction module is used to acquire image data and perform corner prediction on the image data based on the trained corner detection model to obtain output results, including an association embedding prediction map, a corner prediction map, and a corner offset prediction map.

[0088] The corner correction module is used to obtain the basic coordinates of the corner based on the corner prediction map, extract the corner offset according to the corner offset prediction map to correct the basic coordinates of the corner, and combine the associated embedding prediction map to perform associated vector aggregation on the corner to form the license plate position.

[0089] Example 3

[0090] This embodiment provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the one-stage license plate image extraction method in Embodiment 1. In addition, this embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed, it implements the above-mentioned one-stage license plate image extraction method.

[0091] The systems, devices, and storage media in Embodiments 2 and 3 are based on multiple aspects of the same inventive concept as the methods in the foregoing embodiments. The implementation process of the methods has been described in detail above, so those skilled in the art can clearly understand the structure and implementation process of the systems, devices, and storage media in this embodiment based on the foregoing description. For the sake of brevity, they will not be described again here.

[0092] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A one-stage license plate image extraction method, characterized in that, include: A training dataset is obtained, and feature extraction is performed on the training dataset to obtain a feature map; wherein, the training dataset is constructed by collecting vehicle image data and marking the four corner points of the license plate in the image; Feature maps of different sizes are fused to obtain an association embedding prediction map, a corner prediction map, and a corner offset prediction map; The model loss function is determined by combining the associated embedding prediction map, the corner prediction map, and the corner offset prediction map obtained during training. The model loss function is then optimized using a gradient-based optimization algorithm to obtain the optimized model parameters, thus acquiring the corner detection model. The model loss function is: L=L k +L o +L AE ; Among them, L k L o L AE These refer to the license plate corner loss function, the bias loss function, and the association embedding loss function, respectively; the expression for the association embedding loss function is: Where K is the number of corner points of the license plate, N is the number of license plates in the diagram, and h nk For the predicted association embedding vector, Lpull is the average vector of the predicted embedding vectors of the same license plate corner point; Lpush will decrease the Euclidean distance of the associated embedding vectors of corner points belonging to the same license plate, while Lpush will increase the Euclidean distance of the mean embedding vectors of different license plates. Image data is acquired, and corner prediction is performed on the image data based on a corner detection model to obtain the output result. The output includes the correlation embedding prediction map, the corner prediction map, and the corner offset prediction map; Based on the corner prediction map, the basic coordinates of the corner points are obtained, and the corner offset is extracted from the corner offset prediction map. The corner point base coordinates are corrected by extracting the corner point offset corresponding to the corner point base coordinates from the corner point offset prediction map, adding the corner point offset to the corner point base coordinates to obtain the offset-corrected corner point position, and then multiplying the offset-corrected corner point position by the feature map downsampling factor to obtain the corrected corner point coordinates in the input image. The association embedding vectors of the corner points corresponding to the basic coordinates of the corner points are extracted from the association embedding prediction map; and by calculating the Euclidean distance between each association embedding vector, the point with the smallest Euclidean distance is taken as the corner point of the same license plate, and finally the four corner points in the license plate image can be accurately identified.

2. The one-stage license plate image extraction method according to claim 1, characterized in that, After acquiring the image data Also includes: The image data is converted into an image of a specified size using bilinear interpolation. The image undergoes mean removal and normalization.

3. The one-stage license plate image extraction method according to claim 1, characterized in that, Based on the corner prediction map The method for obtaining the basic coordinates of a corner point is as follows: Max pooling is performed on the corner prediction map, and the pooling result is then compared element-wise with the corner prediction map. Subtract the values ​​to obtain the position where the subtracted element is zero, and use the coordinates corresponding to that position as the base coordinates of the corner point.

4. A one-stage license plate image extraction system, characterized in that, Perform as described in any one of claims 1 to 3 A one-stage license plate image extraction method; the system includes: The corner prediction module is used to acquire image data and perform corner prediction on the image data based on a corner detection model. The measurement is used to obtain the output results, which include the correlation embedding prediction map, the corner prediction map, and the corner offset prediction map. The corner correction module is used to obtain the basic coordinates of the corner points based on the corner prediction map, and to predict the corner offsets. The corner offset is extracted to correct the base coordinates of the corner points, and then combined with the associated embedded prediction map to further refine the corner point coordinates. Row-related vectors are aggregated to form the license plate location.

5. An electronic device, characterized in that, It includes a processor, a memory, and components stored in the memory that can be used in the application. A computer program running on the processor, wherein the processor, when executing the computer program, implements any one of claims 1 to 3. The one-stage license plate image extraction method described above.

6. A computer-readable storage medium, characterized in that, It stores a computer program, the computer program When executed, it implements the one-stage license plate image extraction method according to any one of claims 1 to 3.

Images