A license plate recognition method, system, and computing device

Abstract

This invention discloses a license plate recognition method, system, and computing device. The license plate recognition method includes the following steps: acquiring multiple first images, each first image including a vehicle image with at least one license plate, and labeling the coordinates of the center point of the main license plate in the first images; constructing a relational network based on a deep convolutional neural network, constructing a loss function, inputting the first images into the relational network for iterative training, mapping the first images and the coordinates of the center point of the main license plate until a preset termination condition is met to end the training, obtaining the trained relational network; inputting the vehicle image to be recognized into the trained relational network, outputting the predicted coordinates of the center point of the main license plate; acquiring the coordinates of the center point of the license plate in the vehicle image to be recognized, calculating the Euclidean distance between the coordinates of the center point of the license plate in the vehicle image to be recognized and the predicted coordinates of the center point of the main license plate, and identifying the final main license plate. This invention can effectively identify the main license plate, improving the efficiency and accuracy of license plate recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a license plate recognition method, system, and computing device. Background Technology

[0002] Current roadside parking systems use cameras to identify parked vehicles and charge based on time. However, a significant problem hinders the widespread adoption of these systems is low license plate recognition accuracy. Analysis of license plate recognition errors reveals that when a vehicle enters or exits a parking space, the image output by the vehicle frame detector simultaneously includes both the vehicle's license plate and the license plate of the vehicle in the adjacent parking space. When the smaller image cropped from this vehicle detection frame is sent to the license plate detector, it detects and identifies two license plates simultaneously. This results in one vehicle corresponding to two license plates, leading to low license plate recognition accuracy.

[0003] To address the above problems, existing solutions fall into two categories: (1) randomly selecting one of the two license plate recognition results as the license plate recognition result for this vehicle; (2) as in the method for license plate detection and matching of vehicle images and license plate recognition method disclosed in patent publication number CN114241465A, the license plate closer to the center of the vehicle detection box is selected as the license plate recognition result for this vehicle based on the relationship between the center positions of the two license plates and the vehicle detection box. However, both of these solutions have serious accuracy problems, and the second solution is highly rule-based, making it difficult to write a rule algorithm that satisfies all conditions. At the same time, the maintainability of the algorithm code is poor. Summary of the Invention

[0004] To overcome the defects and shortcomings of existing technologies, this invention provides a license plate recognition method. This invention constructs a relational network based on a deep convolutional neural network, maps the coordinates of the center point of the first image and the main license plate, calculates the Euclidean distance between the center point coordinates of the license plate in the image of the vehicle to be recognized and the predicted center point coordinates of the main license plate output by the relational network, and identifies the final main license plate. This effectively solves the problem of not being able to distinguish which license plate belongs to the main license plate when the input image contains two or more license plates for roadside parking, avoids cumbersome and difficult-to-maintain rule algorithms, and improves the efficiency and accuracy of license plate recognition.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a license plate recognition method, comprising the following steps:

[0007] Acquire multiple first images, each first image including a vehicle image with at least one license plate, and annotate the coordinates of the center point of the main license plate in the first image;

[0008] A relational network is constructed based on a deep convolutional neural network, and a loss function is constructed. The labeled first image is input into the relational network for iterative training. The coordinates of the first image and the center point of the main license plate are mapped until the preset termination condition is met to end the training and obtain the trained relational network.

[0009] Acquire an image of the vehicle to be identified, input the image into the trained relational network, and output the predicted coordinates of the center point of the main license plate.

[0010] The coordinates of the center point of the license plate of the vehicle image to be identified are obtained, the Euclidean distance between the center point of the license plate of the vehicle image to be identified and the predicted center point of the main license plate is calculated, and the final main license plate is identified. The Euclidean distance between the center point of the final main license plate and the predicted center point of the main license plate is minimized.

[0011] As a preferred technical solution, after the step of annotating the coordinates of the center point of the main license plate of the first image, a normalization step is also included to normalize the coordinates of the center point of the main license plate of the first image.

[0012] And / or,

[0013] After obtaining the coordinates of the center point of the license plate in the image of the vehicle to be identified, a normalization step is also included to normalize the coordinates of the center point of the license plate in the image of the vehicle to be identified.

[0014] As a preferred technical solution, the coordinates of the center point of the main license plate in the first image are normalized. Specific steps include:

[0015] Taking a corner of the first image as the origin of the pixel coordinate system, the x-axis coordinate value of the center point of the main license plate in the first image is divided by the width of the first image to obtain the normalized x-coordinate of the center point of the main license plate. The y-axis coordinate value of the center point of the main license plate in the pixel coordinate system is divided by the height of the first image to obtain the normalized y-coordinate of the center point of the main license plate.

[0016] The coordinates of the license plate center point of the vehicle image to be identified are normalized. The specific steps include:

[0017] Using a corner of the vehicle image to be identified as the origin of the pixel coordinate system, the x-axis coordinate of the license plate center point in the pixel coordinate system is divided by the width of the vehicle image to obtain the normalized x-coordinate of the license plate center point. The y-axis coordinate of the license plate center point in the pixel coordinate system is divided by the height of the vehicle image to obtain the normalized y-coordinate of the license plate center point.

[0018] As a preferred technical solution, obtaining the coordinates of the license plate center point of the vehicle image to be identified specifically includes:

[0019] The image of the vehicle to be identified is input into the license plate detector, which assigns a license plate detection box to the license plate in the image of the vehicle to be identified, and then outputs the second image.

[0020] Traverse each license plate detection box in the second image and calculate the coordinates of the center point of the license plate detection box as the coordinates of the center point of the license plate in the image of the vehicle to be identified.

[0021] As a preferred technical solution, obtaining the coordinates of the license plate center point of the vehicle image to be identified specifically includes:

[0022] The image of the vehicle to be identified is input into the license plate detector. If the license plate detector does not assign a license plate detection box, it is determined that the main license plate of the currently input vehicle image does not exist.

[0023] When the license plate detector assigns a license plate detection frame, it determines that the license plate corresponding to the current license plate detection frame is the final main license plate;

[0024] When the license plate detector assigns two or more license plate detection frames, it calculates the center point coordinates of each license plate detection frame as the center point coordinates of the license plate in the image of the vehicle to be identified.

[0025] As a preferred technical solution, the relational network sequentially includes: a first two-dimensional convolutional layer, a first BatchNorm2d layer, a first activation function layer, a first two-dimensional max pooling layer, a second two-dimensional convolutional layer, a second BatchNorm2d layer, a second activation function layer, a second two-dimensional max pooling layer, a third two-dimensional convolutional layer, a third BatchNorm2d layer, a third activation function layer, a third two-dimensional max pooling layer, a two-dimensional adaptive max pooling layer, a fully connected layer, and a fourth activation function layer.

[0026] As a preferred technical solution, the convolution kernels and strides of the first, second, and third two-dimensional convolutional layers are all the same;

[0027] And / or,

[0028] The convolution kernels of the first, second, and third two-dimensional max pooling layers are all the same.

[0029] As a preferred technical solution, the first activation function layer, the second activation function layer, and the third activation function layer adopt the ReLU activation function layer;

[0030] And / or,

[0031] The fourth activation function layer uses the Sigmoid activation function layer.

[0032] As a preferred technical solution, the loss function is MSE Loss, specifically expressed as follows:

[0033]

[0034] Where n is the number of the first images, x i Let y represent the first image used for training the relational network. i This represents the coordinates of the center point of the main license plate marked in the i-th first image.

[0035] As a preferred technical solution, the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate is calculated by traversing all vehicle images to be identified, and corresponding to 0, 1 or more Euclidean distances are obtained.

[0036] When a distance of 0 Euclidean distances is obtained, it is determined that the main license plate of the vehicle in the current image to be identified does not exist;

[0037] When one Euclidean distance is obtained, the coordinates of the center point of the license plate of the current vehicle image to be identified are determined as the final coordinates of the center point of the main license plate.

[0038] When multiple Euclidean distances are obtained, the license plate center point coordinate with the smallest Euclidean distance to the predicted main license plate center point coordinate is selected as the main license plate center point coordinate.

[0039] To achieve the second objective mentioned above, the present invention adopts the following technical solution:

[0040] A license plate recognition system includes: a first image acquisition module, an image annotation module, a relation network construction module, a loss function construction module, a network training module, a vehicle image acquisition module, a main license plate center point coordinate prediction module, a license plate center point coordinate acquisition module, an Euclidean distance calculation module, and a main license plate recognition module.

[0041] The first image acquisition module is used to acquire multiple first images, wherein the first image includes a vehicle image with at least one license plate;

[0042] The image annotation module is used to annotate the coordinates of the center point of the main license plate in the first image;

[0043] The relation network construction module is used to construct a relation network based on a deep convolutional neural network;

[0044] The loss function construction module is used to construct the loss function;

[0045] The network training module is used to input the labeled first image into the relation network for iterative training, map the coordinates of the first image and the center point of the main license plate, and end the training until the preset termination condition is met, thus obtaining the trained relation network.

[0046] The vehicle image acquisition module is used to acquire images of the vehicle to be identified.

[0047] The main license plate center point coordinate prediction module is used to input the image of the vehicle to be identified into the trained relational network and output the predicted main license plate center point coordinates.

[0048] The license plate center point coordinate acquisition module is used to acquire the center point coordinates of the license plate of the vehicle image to be identified;

[0049] The Euclidean distance calculation module is used to calculate the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate.

[0050] The main license plate recognition module is used to identify the final main license plate, and the Euclidean distance between the center point coordinates of the final main license plate and the predicted center point coordinates of the main license plate is minimized.

[0051] As a preferred technical solution, the license plate center point coordinate acquisition module includes a license plate detector, and the acquisition of the license plate center point coordinates of the image of the vehicle to be identified specifically includes:

[0052] The image of the vehicle to be identified is input into the license plate detector, which assigns a license plate detection box to the license plate in the image of the vehicle to be identified, and then outputs the second image.

[0053] Traverse each license plate detection box in the second image and calculate the coordinates of the center point of the license plate detection box as the coordinates of the center point of the license plate in the image of the vehicle to be identified.

[0054] To achieve the third objective mentioned above, the present invention adopts the following technical solution:

[0055] A computing device includes a processor and a memory for storing processor-executable programs, wherein when the processor executes the programs stored in the memory, it implements the license plate recognition method described above.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] (1) This invention constructs a relational network based on a deep convolutional neural network, maps the coordinates of the center point of the first image and the main license plate, calculates the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate output by the relational network, and identifies the final main license plate. This can effectively solve the problem that when the input image contains two or more license plates, it is impossible to distinguish which license plate belongs to the main license plate. It avoids cumbersome and difficult-to-maintain rule algorithms and improves the efficiency and accuracy of license plate recognition.

[0058] (2) After annotating the coordinates of the center point of the main license plate of the first image, the present invention normalizes the coordinates of the center point of the main license plate of the first image and inputs the normalized first image into the relation network for iterative training, which can improve the convergence speed of the relation network.

[0059] (3) The first image of the present invention includes a vehicle image with at least one license plate, that is, it may contain one license plate or two or more license plates. The first image is input into the relation network for iterative training, which can improve the generalization ability of the relation network and make the prediction results output by the relation network more accurate. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the license plate recognition method of the present invention.

[0061] Figure 2 This is a schematic diagram of the first image of the present invention;

[0062] Figure 3 This is a schematic diagram of the network architecture of the relational network of the present invention;

[0063] Figure 4 This is a schematic diagram showing the predicted center point coordinates of the main vehicle license plate output by the relational network of this invention.

[0064] Figure 5 This is a schematic diagram of the second image output by the license plate detector of the present invention;

[0065] Figure 6 This is a schematic diagram of the final main license plate image identified by the present invention;

[0066] Figure 7 This is a schematic diagram illustrating the determination process of calculating different numbers of Euclidean distances between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate in the present invention.

[0067] Figure 8 A schematic diagram illustrating the determination process for assigning different numbers of vehicle detection frames to the license plate detector of the present invention; Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] Example 1

[0070] like Figure 1 As shown, this embodiment provides a license plate recognition method, including the following steps:

[0071] S1: Obtain multiple first images, such as Figure 2 As shown, the first image includes a vehicle image with at least one license plate, and the coordinates of the center point of the main license plate in the first image are marked.

[0072] In this embodiment, when annotating the coordinates of the center point of the main license plate in the first image, the outer contour of the license plate is used as a rectangle, and the coordinates of the center point of the rectangle are used as the coordinates of the center point of the main license plate.

[0073] In this embodiment, one image corresponds to one annotation file. For example, 1.jpg corresponds to 1.txt. 1.jpg is a small image containing only a single vehicle, and the content of the 1.txt file is the center point coordinates "0.34 0.64". The larger the dataset, the higher the accuracy. To meet the minimum network training requirements for license plate recognition, this embodiment collected a total of 5000 images. The number of images containing two or more license plates in the dataset is 3800, and the number of images containing only one license plate is 1200.

[0074] In this embodiment, multiple roadside parking images can be acquired through the roadside parking system. The roadside parking images are input into the vehicle frame detector to obtain a small image containing only a single vehicle as the first image. At the same time, the main license plate in the small image is identified manually, and the center point coordinates of the main license plate in the image coordinate system are marked.

[0075] S2: Construct a relational network based on a deep convolutional neural network, construct a loss function, input the labeled first image into the relational network for iterative training, map the coordinates of the first image and its main license plate center point until the preset termination condition is met to end the training and obtain the trained relational network.

[0076] In this embodiment, multiple first images output by the vehicle frame detector are input into the relationship network. These multiple first images may contain one license plate or two license plates. In order for the relationship network to give reasonable prediction results when processing small images containing only one license plate, small images containing only one license plate are also included in the training set, which can improve the generalization ability of the relationship network.

[0077] In this embodiment, after the step of annotating the coordinates of the center point of the main license plate of the first image, a normalization step is also included. The coordinates of the center point of the main license plate of the first image are normalized by taking a corner of the first image as the origin of the pixel coordinate system, dividing the x-axis coordinate value of the center point of the main license plate of the first image in the pixel coordinate system by the width of the first image to obtain the abscissa of the normalized coordinates of the center point of the main license plate, and dividing the y-axis coordinate value of the center point of the main license plate in the pixel coordinate system by the height of the first image to obtain the ordinate of the normalized coordinates of the center point of the main license plate.

[0078] In this embodiment, after annotating the coordinates of the center point of the main license plate in the first image, the coordinates of the center point of the main license plate in the first image are normalized. The normalized first image is then input into the relation network for iterative training, which can improve the convergence speed of the relation network.

[0079] like Figure 3 As shown, the relational network consists of the following layers in sequence: a first 2D convolutional layer, a first BatchNorm2d layer, a first activation function layer, a first 2D max pooling layer, a second 2D convolutional layer, a second BatchNorm2d layer, a second activation function layer, a second 2D max pooling layer, a third 2D convolutional layer, a third BatchNorm2d layer, a third activation function layer, a third 2D max pooling layer, a 2D adaptive max pooling layer, a fully connected layer, and a fourth activation function layer.

[0080] In this embodiment, the kernels and strides of the first, second, and third two-dimensional convolutional layers are the same, preferably 3x3 two-dimensional convolutional layers with a stride of 1.

[0081] In this embodiment, the convolution kernels of the first, second, and third two-dimensional max pooling layers are all the same, preferably 2x2 two-dimensional max pooling layers.

[0082] In this embodiment, the first activation function layer, the second activation function layer, and the third activation function layer are ReLU activation function layers;

[0083] In this embodiment, the fourth activation function layer is a Sigmoid activation function layer;

[0084] In this embodiment, the two-dimensional adaptive max pooling layer is a two-dimensional adaptive max pooling layer with a height and width of 1 in the output feature map, and the fully connected layer is a fully connected layer with an output dimension of 2.

[0085] In this embodiment, the input image size is 320*320*3. After the first 2D convolutional layer extracts shallow information, the resulting feature map size is 320*320*6. After a 2*2 max pooling layer, the feature map size is 160*160*6. After the second 2D convolutional layer extracts semantic information, the feature map size is 160*160*12. After a second 2*2 max pooling layer, the feature map size is 80*80*12. Finally, an adaptive max pooling layer compresses the feature map size to 1*1*12. Since the final output of the relational network is a 2D geometric center coordinate, a fully connected layer with a 12-dimensional input and a 2-dimensional output is added after the adaptive max pooling layer. Finally, a sigmoid activation function layer is added to normalize the output of the fully connected layer.

[0086] In this embodiment, the relational network implements a functional relational mapping from the first image to the center point of the main license plate, avoiding redundant and difficult-to-maintain strategy code, and improving the efficiency and accuracy of license plate recognition.

[0087] In this embodiment, the loss function is MSE Loss, specifically expressed as:

[0088]

[0089] Where n is the number of the first images, x i Let y represent the first image used for training the relational network. i This represents the coordinates of the center point of the main license plate marked in the i-th first image.

[0090] In this embodiment, the relationship network is trained using the ADAM optimizer with a batch size of 32 and a learning rate of 0.01. The model is trained iteratively until the training loss is less than 0.06. The batch size can be adjusted according to the number of training sets. For example, if the number of training sets increases, the batch size can be increased, such as preferably 48 or 64. If there is a first image that the relationship network predicts incorrectly, the first image is simply added to the relationship network training set, allowing the relationship network to fit these samples. This also avoids designing complex and difficult-to-maintain rule codes.

[0091] S3: Obtain the image of the vehicle to be identified, input the image of the vehicle to be identified into the trained relational network, and output the predicted coordinates of the center point of the main license plate.

[0092] like Figure 4As shown, the trained relational network outputs the predicted center point coordinates of the main license plate. The loss function describes the difference between the prediction result of the relational network and the actual annotation result. The goal of training the relational network is to reduce this difference by adjusting the parameters in the relational network. When this difference becomes small enough, the prediction result of the relational network can be used to indicate which license plate is the main license plate of the first image.

[0093] S4: Obtain the center point coordinates of the license plate of the vehicle image to be identified, calculate the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate, and identify the final main license plate, wherein the Euclidean distance between the center point coordinates of the final main license plate and the predicted center point coordinates of the main license plate is minimized.

[0094] In this embodiment, obtaining the coordinates of the center point of the license plate of the vehicle image to be identified specifically includes:

[0095] Input the image of the vehicle to be identified into the license plate detector, such as Figure 5 As shown, the license plate detector assigns a license plate detection bounding box to the license plate in the image of the vehicle to be identified, obtains a second image, and outputs it.

[0096] Traverse each license plate detection box in the second image and calculate the coordinates of the center point of the license plate detection box as the coordinates of the center point of the license plate in the image of the vehicle to be identified;

[0097] Calculate the Euclidean distance between the center point coordinates of the license plate detection frame and the center point coordinates of the predicted main license plate, such as... Figure 6 As shown, the final main license plate is identified, and the Euclidean distance between the center point coordinates of the license plate detection box corresponding to the main license plate and the center point coordinates of the predicted main license plate is minimized.

[0098] In this embodiment, for the center point (x1, y1) of the license plate detection frame, the coordinates of the predicted center point of the main license plate by the relation network are (x2, y2), and the Euclidean algorithm between the two coordinates is as follows:

[0099]

[0100] The smaller the Euclidean distance, the greater the similarity.

[0101] In this embodiment, after obtaining the coordinates of the center point of the license plate of the vehicle image to be identified, a normalization step is also included. The coordinates of the center point of the license plate of the vehicle image to be identified are normalized. Taking a certain corner of the vehicle image to be identified as the origin of the pixel coordinate system, the x-axis coordinate value of the center point of the license plate in the pixel coordinate system is divided by the width of the vehicle image to be identified to obtain the abscissa of the normalized coordinates of the center point of the license plate. The y-axis coordinate value of the center point of the license plate in the pixel coordinate system is divided by the height of the vehicle image to be identified to obtain the ordinate of the normalized coordinates of the center point of the license plate.

[0102] In this embodiment, after obtaining the coordinates of the center point of the license plate in the image of the vehicle to be identified, the coordinates of the center point of the license plate in the image of the vehicle to be identified are normalized. This is to ensure that the Euclidean distance between the center point of the license plate in the image of the vehicle to be identified and the predicted center point of the main license plate has a unified calculation standard, thereby improving the accuracy of the Euclidean distance calculation.

[0103] In this embodiment, when calculating the center point coordinates of the license plate detection box, the vehicle detection box is a rectangle, and the center point of the matrix is ​​the center point coordinate of the license plate detection box. When calculating the center point of the license plate detection box, it is necessary to uniformly calculate it under the pixel coordinate system with a certain corner of the first image (such as the upper left corner) as the origin, and finally normalize it to correspond with the prediction result of the relation network that has also undergone normalization.

[0104] like Figure 7 As shown, the Euclidean distance between the center point coordinates of the license plate of the vehicle to be identified and the predicted center point coordinates of the main license plate is calculated. By traversing all vehicle images to be identified, 0, 1 or more Euclidean distances are obtained.

[0105] When a distance of 0 Euclidean distances is obtained, it is determined that the main license plate of the vehicle in the current image to be identified does not exist;

[0106] When one Euclidean distance is obtained, the coordinates of the center point of the license plate of the current vehicle image to be identified are determined as the final coordinates of the center point of the main license plate.

[0107] When multiple Euclidean distances are obtained, the license plate center point coordinate with the smallest Euclidean distance to the predicted main license plate center point coordinate is selected as the main license plate center point coordinate.

[0108] This embodiment constructs a relational network based on a deep convolutional neural network to map the coordinates of the center point of the first image and the main license plate. It calculates the Euclidean distance between the center point coordinates of the license plate and the predicted center point coordinates of the main license plate output by the relational network, thus identifying the final main license plate. This effectively solves the problem of current roadside parking systems where, when the input image contains two or more license plates, it is impossible to distinguish which license plate belongs to the main vehicle. It avoids cumbersome and difficult-to-maintain rule-based algorithms, improving the efficiency and accuracy of license plate recognition. After direct integration into the current roadside parking system, the license plate recognition accuracy increased from 93% to 97%, significantly improving the efficiency and accuracy of license plate recognition and enhancing the practicality of the roadside parking system.

[0109] Example 2

[0110] Except for the following technical solutions, the remaining technical solutions in this embodiment are the same as those in Embodiment 1;

[0111] This embodiment provides a license plate recognition method, including the following steps:

[0112] S1: Obtain multiple first images, each first image including a vehicle image with at least one license plate, and mark the coordinates of the center point of the main license plate in the first image;

[0113] S2: Construct a relational network based on a deep convolutional neural network, construct a loss function, input the labeled first image into the relational network for iterative training, map the coordinates of the first image and its main license plate center point until the preset termination condition is met to end the training and obtain the trained relational network.

[0114] S3: Obtain the image of the vehicle to be identified, input the image of the vehicle to be identified into the trained relational network, and output the predicted coordinates of the center point of the main license plate.

[0115] S4: Obtain the center point coordinates of the license plate of the vehicle image to be identified, calculate the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate, and identify the final main license plate, wherein the Euclidean distance between the center point coordinates of the final main license plate and the predicted center point coordinates of the main license plate is minimized.

[0116] In this embodiment, obtaining the coordinates of the center point of the license plate of the vehicle image to be identified specifically includes:

[0117] like Figure 8 As shown, the image of the vehicle to be identified is input into the license plate detector. When the license plate detector does not assign a license plate detection box, it is determined that the main license plate of the currently input vehicle image does not exist.

[0118] When the license plate detector assigns a license plate detection frame, it determines that the license plate corresponding to the current license plate detection frame is the final main license plate;

[0119] When the license plate detector assigns two or more license plate detection frames, it calculates the center point coordinates of each license plate detection frame as the center point coordinates of the license plate in the image of the vehicle to be identified.

[0120] Calculate the Euclidean distance between the center point coordinates of the license plate detection frame and the center point coordinates of the predicted main license plate to identify the final main license plate. The Euclidean distance between the center point coordinates of the license plate detection frame corresponding to the main license plate and the center point coordinates of the predicted main license plate is minimized.

[0121] Example 3

[0122] This embodiment provides a license plate recognition system, including: a first image acquisition module, an image annotation module, a relation network construction module, a loss function construction module, a network training module, a vehicle image acquisition module, a main license plate center point coordinate prediction module, a license plate center point coordinate acquisition module, an Euclidean distance calculation module, and a main license plate recognition module;

[0123] In this embodiment, the first image acquisition module is used to acquire multiple first images, the first images including vehicle images with at least one license plate;

[0124] In this embodiment, the image annotation module is used to annotate the coordinates of the center point of the main license plate in the first image;

[0125] In this embodiment, the relation network construction module is used to construct a relation network based on a deep convolutional neural network;

[0126] In this embodiment, the loss function construction module is used to construct the loss function;

[0127] In this embodiment, the network training module is used to input the labeled first image into the relation network for iterative training, map the coordinates of the first image and its main license plate center point, and end the training until the preset termination condition is met to obtain the trained relation network.

[0128] In this embodiment, the relational network sequentially includes: a first two-dimensional convolutional layer, a first BatchNorm2d layer, a first activation function layer, a first two-dimensional max pooling layer, a second two-dimensional convolutional layer, a second BatchNorm2d layer, a second activation function layer, a second two-dimensional max pooling layer, a third two-dimensional convolutional layer, a third BatchNorm2d layer, a third activation function layer, a third two-dimensional max pooling layer, a two-dimensional adaptive max pooling layer, a fully connected layer, and a fourth activation function layer.

[0129] In this embodiment, the kernels and strides of the first, second, and third two-dimensional convolutional layers are the same, preferably 3x3 two-dimensional convolutional layers with a stride of 1.

[0130] In this embodiment, the convolution kernels of the first, second, and third two-dimensional max pooling layers are all the same, preferably 2x2 two-dimensional max pooling layers.

[0131] In this embodiment, the first activation function layer, the second activation function layer, and the third activation function layer are ReLU activation function layers;

[0132] In this embodiment, the fourth activation function layer is a Sigmoid activation function layer;

[0133] In this embodiment, the two-dimensional adaptive max pooling layer is a two-dimensional adaptive max pooling layer with a height and width of 1 in the output feature map, and the fully connected layer is a fully connected layer with an output dimension of 2.

[0134] In this embodiment, the input image size is 320*320*3. After the first 2D convolutional layer extracts shallow information, the resulting feature map size is 320*320*6. After a 2*2 max pooling layer, the feature map size is 160*160*6. After the second 2D convolutional layer extracts semantic information, the feature map size is 160*160*12. After a second 2*2 max pooling layer, the feature map size is 80*80*12. Finally, an adaptive max pooling layer compresses the feature map size to 1*1*12. Since the final output of the relational network is a 2D geometric center coordinate, a fully connected layer with a 12-dimensional input and a 2-dimensional output is added after the adaptive max pooling layer. Finally, a sigmoid activation function layer is added to normalize the output of the fully connected layer.

[0135] In this embodiment, the relational network implements a functional relational mapping from the first image to the center point of the main license plate, avoiding redundant and difficult-to-maintain strategy code, and improving the efficiency and accuracy of license plate recognition.

[0136] In this embodiment, the loss function is MSE Loss, specifically expressed as:

[0137]

[0138] Where n is the number of the first images, x i Let y represent the first image used for training the relational network. i This represents the coordinates of the center point of the main license plate marked in the i-th first image.

[0139] In this embodiment, the vehicle image acquisition module is used to acquire images of the vehicle to be identified;

[0140] In this embodiment, the main license plate center point coordinate prediction module is used to input the image of the vehicle to be identified into the trained relational network and output the predicted coordinates of the main license plate center point.

[0141] In this embodiment, the license plate center point coordinate acquisition module is used to acquire the coordinates of the license plate center point of the image of the vehicle to be identified;

[0142] In this embodiment, the license plate center point coordinate acquisition module includes a license plate detector. Specifically, acquiring the license plate center point coordinates of the vehicle image to be identified includes:

[0143] The image of the vehicle to be identified is input into the license plate detector. The license plate detector assigns a license plate detection box to the license plate in the image of the vehicle to be identified, and then outputs the second image.

[0144] Traverse each license plate detection box in the second image and calculate the coordinates of the center point of the license plate detection box as the coordinates of the center point of the license plate in the image of the vehicle to be identified;

[0145] In this embodiment, the Euclidean distance calculation module is used to calculate the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate.

[0146] In this embodiment, for the center point (x1, y1) of the license plate detection frame, the coordinates of the predicted center point of the main license plate by the relation network are (x2, y2), and the Euclidean algorithm between the two coordinates is as follows:

[0147]

[0148] The smaller the Euclidean distance, the greater the similarity.

[0149] In this embodiment, the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate is calculated. All vehicle images to be identified are traversed to obtain 0, 1 or more Euclidean distances.

[0150] When a distance of 0 Euclidean distances is obtained, it is determined that the main license plate of the vehicle in the current image to be identified does not exist;

[0151] When one Euclidean distance is obtained, the coordinates of the center point of the license plate of the current vehicle image to be identified are determined as the final coordinates of the center point of the main license plate.

[0152] In this embodiment, the main license plate recognition module is used to identify the final main license plate, and the Euclidean distance between the center point coordinates of the final main license plate and the predicted center point coordinates of the main license plate is minimized.

[0153] In this embodiment, a normalization module is also included. After annotating the coordinates of the center point of the main license plate in the first image, the coordinates of the center point of the main license plate in the first image are normalized. Taking a corner of the first image as the origin of the pixel coordinate system, the x-axis coordinate value of the center point of the main license plate in the first image is divided by the width of the first image to obtain the abscissa of the normalized coordinates of the center point of the main license plate. The y-axis coordinate value of the center point of the main license plate in the first image is divided by the height of the first image to obtain the ordinate of the normalized coordinates of the center point of the main license plate. After annotating the coordinates of the center point of the main license plate in the first image, the coordinates of the center point of the main license plate in the first image are normalized. The normalized first image is input into the relation network for iterative training, which can improve the convergence speed of the relation network.

[0154] Example 4

[0155] Except for the following technical solutions, the remaining technical solutions in this embodiment are the same as those in Embodiment 3;

[0156] This embodiment provides a license plate recognition system, including: a first image acquisition module, an image annotation module, a relation network construction module, a loss function construction module, a network training module, a vehicle image acquisition module, a main license plate center point coordinate prediction module, a license plate center point coordinate acquisition module, an Euclidean distance calculation module, and a main license plate recognition module;

[0157] In this embodiment, the license plate center point coordinate acquisition module is used to acquire the coordinates of the license plate center point of the image of the vehicle to be identified;

[0158] In this embodiment, the license plate center point coordinate acquisition module includes a license plate detector and a license plate detection frame count calculation module. This embodiment acquires the license plate center point coordinates of the vehicle image to be identified, specifically including:

[0159] The image of the vehicle to be identified is input into the license plate detector. The license plate detection frame count calculation module is used to calculate the number of license plate detection frames assigned by the license plate detector. When the license plate detector does not assign any license plate detection frames, it is determined that the main license plate does not exist in the currently input image of the vehicle to be identified.

[0160] When the license plate detector assigns a license plate detection frame, it determines that the license plate corresponding to the current license plate detection frame is the final main license plate;

[0161] When the license plate detector assigns two or more license plate detection frames, it calculates the center point coordinates of each license plate detection frame as the center point coordinates of the license plate in the image of the vehicle to be identified.

[0162] Calculate the Euclidean distance between the center point coordinates of the license plate detection frame and the center point coordinates of the predicted main license plate to identify the final main license plate. The Euclidean distance between the center point coordinates of the license plate detection frame corresponding to the main license plate and the center point coordinates of the predicted main license plate is minimized.

[0163] Example 5

[0164] This embodiment provides a computing device, which may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer or other terminal device with display function. The computing device includes a processor and a memory. The memory stores one or more programs. When the processor executes the program stored in the memory, it implements the license plate recognition method of Embodiment 1 or Embodiment 2.

[0165] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A license plate recognition method, characterized in that, Includes the following steps: Acquire multiple first images, each first image including a vehicle image with at least one license plate, and annotate the coordinates of the center point of the main license plate in the first image; A relational network is constructed based on a deep convolutional neural network, and a loss function is constructed. The labeled first image is input into the relational network for iterative training. The coordinates of the first image and the center point of the main license plate are mapped until the preset termination condition is met to end the training and obtain the trained relational network. The relational network comprises, in sequence: a first two-dimensional convolutional layer, a first BatchNorm2d layer, a first activation function layer, a first two-dimensional max pooling layer, a second two-dimensional convolutional layer, a second BatchNorm2d layer, a second activation function layer, a second two-dimensional max pooling layer, a third two-dimensional convolutional layer, a third BatchNorm2d layer, a third activation function layer, a third two-dimensional max pooling layer, a two-dimensional adaptive max pooling layer, a fully connected layer, and a fourth activation function layer. Acquire an image of the vehicle to be identified, input the image into the trained relational network, and output the predicted coordinates of the center point of the main license plate. The coordinates of the center point of the license plate of the vehicle image to be identified are obtained, the Euclidean distance between the center point of the license plate of the vehicle image to be identified and the predicted center point of the main license plate is calculated, and the final main license plate is identified. The Euclidean distance between the center point of the final main license plate and the predicted center point of the main license plate is minimized.

2. The license plate recognition method according to claim 1, characterized in that, After the step of annotating the coordinates of the center point of the main license plate in the first image, a normalization step is also included to normalize the coordinates of the center point of the main license plate in the first image. And / or, After obtaining the coordinates of the center point of the license plate in the image of the vehicle to be identified, a normalization step is also included to normalize the coordinates of the center point of the license plate in the image of the vehicle to be identified.

3. The license plate recognition method according to claim 2, characterized in that, The coordinates of the center point of the main license plate in the first image are normalized. The specific steps include: Taking a corner of the first image as the origin of the pixel coordinate system, the x-axis coordinate value of the center point of the main license plate in the first image is divided by the width of the first image to obtain the normalized x-coordinate of the center point of the main license plate. The y-axis coordinate value of the center point of the main license plate in the pixel coordinate system is divided by the height of the first image to obtain the normalized y-coordinate of the center point of the main license plate. The coordinates of the license plate center point of the vehicle image to be identified are normalized. The specific steps include: Using a corner of the vehicle image to be identified as the origin of the pixel coordinate system, the x-axis coordinate of the license plate center point in the pixel coordinate system is divided by the width of the vehicle image to obtain the normalized x-coordinate of the license plate center point. The y-axis coordinate of the license plate center point in the pixel coordinate system is divided by the height of the vehicle image to obtain the normalized y-coordinate of the license plate center point.

4. The license plate recognition method according to claim 1, characterized in that, The process of obtaining the coordinates of the center point of the license plate in the image of the vehicle to be identified specifically includes: The image of the vehicle to be identified is input into the license plate detector, which assigns a license plate detection box to the license plate in the image of the vehicle to be identified, and then outputs the second image. Traverse each license plate detection box in the second image and calculate the coordinates of the center point of the license plate detection box as the coordinates of the center point of the license plate in the image of the vehicle to be identified.

5. The license plate recognition method according to claim 1, characterized in that, The process of obtaining the coordinates of the center point of the license plate in the image of the vehicle to be identified specifically includes: The image of the vehicle to be identified is input into the license plate detector. If the license plate detector does not assign a license plate detection box, it is determined that the currently input image of the vehicle to be identified does not have a main license plate. When the license plate detector assigns a license plate detection frame, it determines that the license plate corresponding to the current license plate detection frame is the final main license plate; When the license plate detector assigns two or more license plate detection frames, it calculates the center point coordinates of each license plate detection frame as the center point coordinates of the license plate in the image of the vehicle to be identified.

6. The license plate recognition method according to claim 1, characterized in that, The first, second, and third two-dimensional convolutional layers all have the same convolutional kernel and stride. And / or, The convolution kernels of the first, second, and third two-dimensional max pooling layers are all the same.

7. The license plate recognition method according to claim 1, characterized in that, The first activation function layer, the second activation function layer, and the third activation function layer all use ReLU activation function layers. And / or, The fourth activation function layer uses the Sigmoid activation function layer.

8. The license plate recognition method according to claim 1, characterized in that, The loss function adopted is MSE Loss, specifically expressed as: Where n is the number of the first images, x i Let y represent the first image used for training the relational network. i This represents the coordinates of the center point of the main license plate marked in the i-th first image.

9. The license plate recognition method according to claim 1, characterized in that, The calculation of the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate is performed by traversing all vehicle images to be identified, and obtaining 0, 1 or more Euclidean distances accordingly. When a distance of 0 Euclidean distances is obtained, it is determined that the main license plate of the vehicle in the current image to be identified does not exist; When one Euclidean distance is obtained, the coordinates of the center point of the license plate of the current vehicle image to be identified are determined as the final center point coordinates of the main license plate. When multiple Euclidean distances are obtained, the license plate center point coordinate with the smallest Euclidean distance to the predicted main license plate center point coordinate is selected as the main license plate center point coordinate.

10. A license plate recognition system, characterized in that, include: The system comprises the following modules: first image acquisition module, image annotation module, relation network construction module, loss function construction module, network training module, vehicle image acquisition module, main license plate center point coordinate prediction module, license plate center point coordinate acquisition module, Euclidean distance calculation module, and main license plate recognition module. The first image acquisition module is used to acquire multiple first images, wherein the first image includes a vehicle image with at least one license plate; The image annotation module is used to annotate the coordinates of the center point of the main license plate in the first image; The relation network construction module is used to construct a relation network based on a deep convolutional neural network; The relational network comprises, in sequence: a first two-dimensional convolutional layer, a first BatchNorm2d layer, a first activation function layer, a first two-dimensional max pooling layer, a second two-dimensional convolutional layer, a second BatchNorm2d layer, a second activation function layer, a second two-dimensional max pooling layer, a third two-dimensional convolutional layer, a third BatchNorm2d layer, a third activation function layer, a third two-dimensional max pooling layer, a two-dimensional adaptive max pooling layer, a fully connected layer, and a fourth activation function layer. The loss function construction module is used to construct the loss function; The network training module is used to input the labeled first image into the relation network for iterative training, map the coordinates of the first image and the center point of the main license plate, and end the training until the preset termination condition is met, thus obtaining the trained relation network. The vehicle image acquisition module is used to acquire images of the vehicle to be identified. The main license plate center point coordinate prediction module is used to input the image of the vehicle to be identified into the trained relational network and output the predicted main license plate center point coordinates. The license plate center point coordinate acquisition module is used to acquire the center point coordinates of the license plate of the vehicle image to be identified; The Euclidean distance calculation module is used to calculate the Euclidean distance between the center point coordinates of the license plate of the vehicle image to be identified and the predicted center point coordinates of the main license plate. The main license plate recognition module is used to identify the final main license plate, and the Euclidean distance between the center point coordinates of the final main license plate and the predicted center point coordinates of the main license plate is minimized.

11. The license plate recognition system according to claim 10, characterized in that, The license plate center point coordinate acquisition module includes a license plate detector. The acquisition of the license plate center point coordinates of the vehicle image to be identified specifically includes: The image of the vehicle to be identified is input into the license plate detector, which assigns a license plate detection box to the license plate in the image of the vehicle to be identified, and then outputs the second image. Traverse each license plate detection box in the second image and calculate the coordinates of the center point of the license plate detection box as the coordinates of the center point of the license plate in the image of the vehicle to be identified.

12. A computing device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the license plate recognition method as described in any one of claims 1-9.

Images