License plate recognition model training method, recognition method, device, equipment and medium

Abstract

The present invention discloses a method for training a license plate recognition model. The license plate recognition model includes a final character detection submodel, a final license plate determination submodel, and a final character recognition submodel. The method includes the steps of training an initial license plate determination submodel to obtain a final license plate determination submodel. Training the initial license plate determination submodel includes: training the initial license plate determination submodel based on positive license plate image samples, negative license plate image samples, prior information of the negative license plate image samples, and prior information of the positive license plate image samples. The prior information includes image aspect ratio, image pixel information, and the number of characters in the image. The prior information is used to train the license plate determination fully connected layer of the initial license plate determination submodel. This can effectively improve the accuracy of the final license plate determination submodel in determining license plate images, thereby improving the accuracy of license plate recognition. The present invention also discloses a license plate recognition method, electronic device, and computer-readable medium.

Description

Technical Field

[0001] The present invention relates to the technical field of license plate recognition, and in particular to a license plate recognition model training method, a license plate recognition method, a license plate recognition model training device, an electronic device and a computer-readable medium. Background Art

[0002] With the continuous development of my country's economy and the gradual improvement of people's living standards, motor vehicles have become one of the most important modes of transportation. License plate recognition technology is a widely used application of computer vision and image processing technologies in the field of intelligent transportation. Currently, existing license plate recognition solutions perform poorly in complex scenarios. Complex scenarios refer to situations where license plate image quality is affected by factors such as tilted license plate angle, blurred license plate images, rain, snow, fog, and strong or weak lighting conditions. Consequently, license plate recognition performance in complex scenarios is generally poor.

[0003] Therefore, a new license plate recognition solution is urgently needed to improve the license plate recognition effect. Summary of the Invention

[0004] To this end, the present invention provides a license plate recognition model training method, a license plate recognition method, a license plate recognition model training device, an electronic device and a computer-readable medium to address the deficiencies in the prior art.

[0005] To achieve the above-mentioned object, the first aspect of the present invention provides a method for training a license plate recognition model, wherein the license plate recognition model includes a final character detection sub-model, a final license plate determination sub-model, and a final character recognition sub-model. The method includes the steps of training an initial license plate determination sub-model to obtain the final license plate determination sub-model, wherein the training of the initial license plate determination sub-model includes:

[0006] An initial license plate determination sub-model is trained based on positive license plate image samples, negative license plate image samples, prior information of the negative license plate image samples, and prior information of the positive license plate image samples, wherein the prior information of the positive license plate image samples and the prior information of the negative license plate image samples both include image aspect ratio, image pixel information, and the number of image characters, and the prior information of the positive license plate image samples and the prior information of the negative license plate image samples are both used to train the license plate determination fully connected layer of the initial license plate determination sub-model.

[0007] In some embodiments, before the step of training the initial license plate determination sub-model to obtain the final license plate determination sub-model, the method further includes:

[0008] Determining labeled data for training image data;

[0009] The license plate image positive sample and the license plate image negative sample are determined according to the labeled data and the training image data.

[0010] In some embodiments, the annotated data includes license plate position coordinate data, and determining the license plate image positive sample and the license plate image negative sample based on the annotated data and the training image data includes:

[0011] Determine non-license plate position coordinate data and the license plate image positive sample according to the license plate position coordinate data and the training image data;

[0012] The license plate image negative sample is determined according to the non-license plate position coordinate data and the training image data.

[0013] In some embodiments, after determining the labeled data of the training image data, the method further includes the step of training an initial character detection sub-model to obtain the final character detection sub-model, wherein the training of the initial character detection sub-model includes:

[0014] An initial character detection sub-model is trained according to the license plate position coordinate data and the training image data, wherein the initial character detection sub-model includes a cascaded character detection convolution layer, a character detection pooling layer and a feature fusion layer.

[0015] In some embodiments, the annotated data further includes license plate character content data. After determining the image positive sample and the image negative sample based on the annotated data and the training image data, the method further includes the step of training an initial character recognition sub-model to obtain the final character recognition sub-model, wherein the training of the initial character recognition sub-model includes:

[0016] An initial character recognition sub-model is trained based on the license plate character content data and the license plate image positive sample, wherein the initial character recognition sub-model includes a cascaded feature extraction layer, a character recognition pooling layer, a semantic learning layer and a character recognition fully connected layer.

[0017] In some embodiments, the initial license plate determination submodel includes a cascaded preprocessing module, a first license plate determination convolutional layer, a first license plate determination pooling layer, a second license plate determination convolutional layer, a second license plate determination pooling layer, and a license plate determination fully connected layer; the image pixel information includes the red channel pixel mean, the green channel pixel mean, and the blue channel pixel mean. Before the step of training the initial license plate determination submodel to obtain the final license plate determination submodel, the method further includes the step of obtaining prior information, and the obtaining of prior information includes:

[0018] Obtaining the image aspect ratio of the positive sample of the license plate image and the negative sample of the license plate image;

[0019] Obtaining the red channel pixel mean, the green channel pixel mean, and the blue channel pixel mean of the license plate image positive sample and the license plate image negative sample according to the three-channel matrix;

[0020] The number of image characters of the positive sample of the license plate image and the negative sample of the license plate image are obtained according to the watershed algorithm.

[0021] A second aspect of the present invention provides a license plate recognition method, the method comprising:

[0022] The image to be recognized is input into the license plate recognition model obtained by the license plate recognition model training method described in any of the previous embodiments, so that:

[0023] The final character detection sub-model determines the license plate position coordinate data of the image to be recognized;

[0024] The final license plate determination sub-model determines whether the image to be identified is a license plate image based on the image to be identified and the license plate position coordinate data;

[0025] The final character recognition sub-model identifies license plate character content in the image to be recognized when the final license plate determination sub-model determines that the image to be recognized is a license plate image.

[0026] A third aspect of the present invention provides a license plate recognition model training device, wherein the license plate recognition model includes a final character detection sub-model, a final license plate determination sub-model, and a final character recognition sub-model, and the device includes:

[0027] The first training module is used to train the initial license plate determination sub-model based on the positive sample of the license plate image, the negative sample of the license plate image, the prior information of the negative sample of the license plate image and the prior information of the positive sample of the license plate image to obtain the final license plate determination sub-model, wherein the prior information of the positive sample of the license plate image and the prior information of the negative sample of the license plate image both include the image aspect ratio, image pixel information and the number of image characters, and the prior information of the positive sample of the license plate image and the prior information of the negative sample of the license plate image are both used to train the license plate determination fully connected layer of the initial license plate determination sub-model.

[0028] A fourth aspect of the present invention provides an electronic device, comprising:

[0029] one or more processors;

[0030] A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the following:

[0031] The license plate recognition model training method as described in any of the previous embodiments;

[0032] The license plate recognition method as described in the previous embodiment;

[0033] One or more I / O interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

[0034] A fifth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, any of the following is achieved:

[0035] The license plate recognition model training method as described in any of the previous embodiments;

[0036] The license plate recognition method as described in the previous embodiment.

[0037] The image aspect ratio, image pixel information and number of image characters of the positive license plate image samples, negative license plate image samples and negative license plate image samples are used to train the final license plate determination sub-model in the license plate recognition model. When training the license plate determination fully connected layer of the final license plate determination sub-model, the image aspect ratio, image pixel information and number of image characters of the positive license plate image samples and the negative license plate image samples are integrated, which can effectively improve the accuracy of the final license plate determination sub-model in determining the license plate image, so that the license plate image can be detected normally and can smoothly enter the character recognition process, thereby improving the accuracy of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0039] Figure 1 A schematic diagram of the process of training the initial license plate determination sub-model provided by an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the process of determining positive and negative license plate image samples provided by the embodiment of the present invention Figure 1 ;

[0041] Figure 3 Schematic diagram of the process of determining positive and negative license plate image samples provided by the embodiment of the present invention Figure 2 ;

[0042] Figure 4 A schematic diagram of the main structure of the initial character detection sub-model provided by an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of the structure of the initial character recognition sub-model provided by an embodiment of the present invention;

[0044] Figure 6 A schematic diagram of the structure of the initial license plate determination sub-model provided by an embodiment of the present invention;

[0045] Figure 7 A schematic diagram of the training of the license plate recognition model provided by an embodiment of the present invention;

[0046] Figure 8 A license plate recognition flow chart provided by an embodiment of the present invention;

[0047] Figure 9 A schematic diagram of the modules of the license plate recognition model training device provided by an embodiment of the present invention;

[0048] Figure 10 A schematic diagram of a module of a license plate recognition device provided by an embodiment of the present invention;

[0049] Figure 11 A schematic diagram of a module of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0051] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0052] The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. As used in the present invention, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0053] When the terms "comprising" and / or "made of..." are used in the present invention, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not preclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0054] The embodiments of the present invention may be described with reference to plan views and / or cross-sectional views by way of ideal schematic views of the present invention. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0055] Unless otherwise defined, all terms (including technical and scientific terms) used in the present invention have the same meaning as commonly understood by those skilled in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined in the present invention.

[0056] With the continuous development of my country's economy and the gradual improvement of people's living standards, motor vehicles have become one of the most important modes of transportation. License plate recognition technology is a widely used application of computer vision and image processing technologies in the field of intelligent transportation. It is mainly divided into two parts: license plate detection and character recognition. License plate detection determines whether an image is a license plate image, while character recognition identifies the characters in an image that is determined to be a license plate image.

[0057] In the field of license plate detection, the speed and accuracy of license plate detection algorithms using traditional models such as structural feature methods and statistical feature methods can meet actual needs in certain specific scenarios (such as parking lot access control). However, the performance in some complex scenarios is not satisfactory. Complex scenarios refer to situations where the license plate is tilted at an angle, the license plate image is blurred, rainy, snowy, foggy, strong or weak lighting conditions affect the quality of the license plate image. Therefore, the license plate detection effect in complex scenes is generally poor. In the field of character recognition, the traditional character recognition method is to extract individual characters by segmentation from the detection area. Commonly used segmentation methods include projection-based and character contour-based methods. Template matching is then performed on the individual characters extracted after segmentation to obtain the final recognition result. This method also has the problem of poor character recognition effect on license plate images taken in complex scenes. Therefore, it is crucial to explore license plate recognition methods that meet the conditions of complex scenes.

[0058] Since the 21st century, with the development of computer technology, artificial intelligence and deep learning techniques have made significant progress, and they have gradually been applied to license plate recognition algorithms. License plate recognition algorithms based on deep learning can also be divided into two modules: the license plate detection module and the character recognition module. In the license plate detection component, target detection methods are currently mostly used. Convolutional neural networks are used to locate the license plate in the image. Threshold screening is performed to obtain the license plate coordinates. Perspective transformation is then used to correct tilted license plates. However, most algorithms suffer from long computation times and poor detection performance for tilted license plates. In the license plate character recognition component, a network structure consisting of convolutional layers plus fully connected layers is often used to predict license plate characters. However, the accuracy of recognizing tilted and blurred license plate characters is also low. Throughout the license plate recognition process, the license plate detection module is responsible for first detecting and segmenting the area in the image containing the license plate. The segmented image is then normalized using methods such as interpolation. Finally, the normalized result is fed into the character detection module for character recognition, which outputs the final license plate character recognition result. This solution has a good recognition effect for license plate images taken under ideal conditions. However, for license plate images taken in complex scenes such as occlusion, blur, distortion and special lighting effects, the license plate will not be detected normally because the result score of the license plate detection is lower than the threshold, resulting in the license plate character recognition process no longer being carried out, that is, the license plate recognition for the image will be declared a failure.

[0059] In view of this, the embodiments of the present invention propose that the main reason why the existing license plate recognition methods are less effective in complex scenes is that the license plate detection link cannot accurately determine the license plate image. Information such as the image size, pixels and number of characters in the license plate image can effectively assist license plate detection, but most of the existing license plate detection methods do not utilize this image information. Therefore, a new license plate recognition model can be trained in combination with information such as the image size, pixels and number of characters of the license plate image, so that the new license plate recognition model can be used to recognize the image to be recognized, thereby improving the accuracy of license plate recognition.

[0060] Accordingly, in a first aspect, an embodiment of the present invention provides a method for training a license plate recognition model, wherein the license plate recognition model includes a final character detection sub-model, a final license plate determination sub-model, and a final character recognition sub-model. The method includes the steps of training an initial license plate determination sub-model to obtain a final license plate determination sub-model, wherein, Figure 1 As shown, the training of the initial license plate determination sub-model includes:

[0061] In step S11, the initial license plate determination sub-model is trained based on the positive license plate image samples, the negative license plate image samples, the prior information of the negative license plate image samples and the prior information of the positive license plate image samples, wherein the prior information of the positive license plate image samples and the prior information of the negative license plate image samples both include the image aspect ratio, image pixel information and the number of image characters, and the prior information of the positive license plate image samples and the prior information of the negative license plate image samples are both used to train the license plate determination fully connected layer of the initial license plate determination sub-model.

[0062] Among them, the positive sample of the license plate image is the image whose image content includes the license plate, the negative sample of the license plate image is the image whose image content does not include the license plate, the image aspect ratio is the aspect ratio of the license plate image obtained by photographing the license plate from the front, the image pixel information includes the pixel information of the three color channels of red (R), green (G), and blue (B) in the image, and the number of image characters is the total number of characters in the image content.

[0063] By inputting the positive samples of license plate images, negative samples of license plate images, prior information of negative samples of license plate images and positive samples of license plate images into the initial license plate determination sub-model, the output result of the initial license plate determination sub-model can be obtained. The initial license plate determination sub-model adjusts the parameters and hyperparameters within the model according to the relationship between the input and output until the convergence conditions of the initial license plate determination sub-model are reached to obtain the final license plate determination sub-model. The initial license plate determination sub-model includes at least a license plate determination fully connected layer, and the prior information is used to train the license plate determination fully connected layer.

[0064] It can be seen from the above step S11 that the license plate recognition model training method provided by the embodiment of the present invention uses the image aspect ratio, image pixel information and number of image characters of the license plate image positive sample, license plate image negative sample and license plate image negative sample for training the final license plate determination sub-model in the license plate recognition model. When training the license plate determination fully connected layer of the final license plate determination sub-model, the image aspect ratio, image pixel information and number of image characters of the license plate image positive sample and the license plate image negative sample are integrated, which can effectively improve the accuracy of the final license plate determination sub-model in determining the license plate image, so that the license plate image can be detected normally and can smoothly enter the character recognition process, thereby improving the accuracy of license plate recognition.

[0065] Before officially starting to train the license plate recognition model, the training image data, i.e., the original image data, needs to be processed to obtain samples (including positive samples and negative samples) that can be used to train the model and annotated data for auxiliary parameter adjustment. Accordingly, in some embodiments, before the step of training the initial license plate determination sub-model to obtain the final license plate determination sub-model, as shown in FIG. Figure 2 As shown, the method may further include the following steps:

[0066] In step S21 , the labeling data of the training image data is determined.

[0067] In step S22, the positive sample and the negative sample of the license plate image are determined according to the labeled data and the training image data.

[0068] The training image data all contain images containing license plates, but the training image data also contains other image content besides license plates. The training image data is annotated and the annotated content is saved as the annotated data of the corresponding training image data for use in training the license plate recognition model.

[0069] The initial license plate recognition model provided by the embodiments of the present invention is actually a binary classification model, requiring two types of samples for training: positive and negative. Positive samples are images whose content includes license plates, while negative samples are images whose content does not. Therefore, for the training image data, the image must first be segmented based on the license plate position coordinate data to obtain positive license plate image samples. Secondly, based on the size of the positive sample images, images are randomly captured at other non-license plate locations within the training image data as negative license plate image samples. Finally, both positive and negative samples are input as training data into the initial license plate recognition sub-model. Prior information from the training data is then incorporated into the fully connected license plate recognition layer of the initial license plate recognition sub-model to complete model training.

[0070] Accordingly, in some embodiments, Figure 3As shown, the annotated data includes license plate position coordinate data, and determining the license plate image positive sample and the license plate image negative sample based on the annotated data and the training image data (i.e., step S22) may include the following steps:

[0071] In step S221 , non-license plate position coordinate data and license plate image positive samples are determined based on the license plate position coordinate data and the training image data.

[0072] In step S222 , a license plate image negative sample is determined based on the non-license plate position coordinate data and the training image data.

[0073] It should be noted that the preprocessing module of the initial license plate determination sub-model can also have the function of segmenting the image according to the license plate position coordinate data to obtain the license plate image, that is, it can determine the positive sample and negative sample of the license plate image according to the license plate position coordinate data and the training image data.

[0074] In some embodiments, after determining the annotation data of the training image data, the method also includes the step of training an initial character detection sub-model to obtain the final character detection sub-model, wherein the training of the initial character detection sub-model may include the following steps: training the initial character detection sub-model based on the license plate position coordinate data and the training image data, wherein the initial character detection sub-model includes a cascaded character detection convolution layer, a character detection pooling layer and a feature fusion layer.

[0075] The license plate position coordinate data is the position coordinates of the license plate in the image, including the coordinates of the upper left corner and the lower right corner of the rectangular frame.

[0076] When training the initial character detection sub-model, first load the pre-trained model parameters to enable the model to initially have the ability to recognize characters. Then, fine-tune the model parameters using the license plate position coordinate data to obtain the final trained character detection sub-model.

[0077] like Figure 4As shown, it is a schematic diagram of the main structure of the initial character detection sub-model provided by an embodiment of the present invention. The initial character detection sub-model can be understood as a "U"-shaped network. It inputs an image of size 3xHxW (dimensions, H is the image height, W is the image width), and can output a character Gaussian heat map of size 1x(H / 2)x(W / 2). Based on the Vgg_16 network (Visual geometry group network), the different levels of feature information of the image are obtained through each character detection convolution layer and the character detection pooling layer. Then, in the feature fusion process, the different levels of feature information of the image are spliced ​​and fused to obtain a single character Gaussian heat map of the entire image. According to the single character Gaussian heat map, it is subjected to image morphological processing such as binarization, finding connected domains, corrosion and expansion, and the position coordinates of the possible license plate in the image are obtained. The single character detection algorithm used for text recognition is applied to the license plate recognition algorithm. By processing the single character Gaussian heat map, the position coordinates of the license plate in the image are obtained, which has a better effect on the extraction of tilted license plates and incomplete license plates.

[0078] In some embodiments, the annotated data also includes license plate character content data. After determining the image positive sample and the image negative sample based on the annotated data and the training image data, the method also includes the step of training the initial character recognition sub-model to obtain the final character recognition sub-model, wherein the training of the initial character recognition sub-model may include the following steps: training the initial character recognition sub-model based on the license plate character content data and the license plate image positive sample, wherein the initial character recognition sub-model includes a cascaded feature extraction layer, a character recognition pooling layer, a semantic learning layer and a character recognition fully connected layer.

[0079] The training of the character recognition sub-model is relatively simple. For the positive samples of license plate images obtained by segmenting the training image data according to the license plate position coordinate data, they can be input into the initial character recognition sub-model together with the labeled license plate character content to complete the training of the model.

[0080] In an embodiment of the present invention, the training image data is processed uniformly, and the license plate position coordinate data and the license plate character content data are obtained simultaneously through a single data annotation, so that the training of the three models of the initial character detection sub-model, the initial license plate determination sub-model and the initial character recognition sub-model can be realized, thereby realizing end-to-end intelligent license plate recognition model training without the need for manual intervention in the model training process.

[0081] like Figure 5As shown, it is a schematic diagram of the structure of the initial character recognition sub-model provided by an embodiment of the present invention. The initial character recognition model is composed of a feature extraction layer, a character recognition pooling layer, a semantic learning layer and a character recognition fully connected layer. An image of size 1xHxW is input, and the image information is abstracted into a matrix of size 512x(H / 16)x(W / 4) through the feature extraction layer. The image information is then abstracted into a matrix of size 1x(W / 4)x512 through the character recognition pooling layer. It is then input into the semantic learning layer based on LSTM (Long short-term memory, long short-term memory network), making full use of the contextual information of the license plate characters, and finally obtaining a one-dimensional vector 1x the total number of recognizable characters, which can be mapped as ABCD...WXYZ. The value of each position in the vector represents the probability of the mapped character existing at that position in the image. The characters are screened by a threshold value, and the license plate character recognition is finally completed.

[0082] In some embodiments, the initial license plate determination submodel includes a cascaded preprocessing module, a first license plate determination convolutional layer, a first license plate determination pooling layer, a second license plate determination convolutional layer, a second license plate determination pooling layer and a license plate determination fully connected layer; the image pixel information includes the red channel pixel mean, the green channel pixel mean and the blue channel pixel mean. Before the step of training the initial license plate determination submodel to obtain the final license plate determination submodel, the method also includes the step of obtaining prior information, and the obtaining of prior information may include the following steps: obtaining the image aspect ratio of the positive sample of the license plate image and the negative sample of the license plate image; obtaining the red channel pixel mean, green channel pixel mean and blue channel pixel mean of the positive sample of the license plate image and the negative sample of the license plate image according to the three-channel matrix; obtaining the number of image characters of the positive sample of the license plate image and the negative sample of the license plate image according to the watershed algorithm.

[0083] like Figure 6As shown, it is a schematic diagram of the structure of the initial license plate determination submodel provided by an embodiment of the present invention. The initial license plate determination submodel consists of a cascaded preprocessing module, a first license plate determination convolutional layer, a first license plate determination pooling layer, a second license plate determination convolutional layer, a second license plate determination pooling layer and a license plate determination fully connected layer. First, the input image is converted into a grayscale image through preprocessing and then converted into a 64*128 (dimension, image height*width) grayscale image through normalization and other processing. After the first determination convolutional layer, an image of 16x64x128 size is obtained, after the first license plate determination pooling layer, an image of 16x16x32 size is obtained, after the second license plate determination convolutional layer, an image of 32x16x32 size is obtained, after the second license plate determination pooling layer, an image of 32x4x8 size is obtained, and a feature vector of 1x1024 size (dimension, 1024 is the number of image features) size is obtained by expansion, and a one-dimensional feature vector of 1x256 (dimension, 256 is the number of image features) is obtained by fusion through the fully connected layer. At this point, the 1x5 feature vector (the prior information) is concatenated onto this feature vector, resulting in a 1x261 feature vector (261 is the number of image features), enabling the model to more accurately identify license plates. Finally, after two fully connected layers, a 1x2 two-dimensional vector is obtained, which can be used to determine whether the input image (or the image content) is a license plate.

[0084] Among them, the prior information includes the image aspect ratio, the red channel pixel mean, the green channel pixel mean, the blue channel pixel mean and the estimated number of characters. The image aspect ratio, the red channel pixel mean, the green channel pixel mean and the blue channel pixel mean can be obtained through the original image information, and the estimated number of characters can be obtained through the watershed algorithm. Specifically: (1) The size of my country's automobile license plate is 44 cm wide and 14 cm high, so the aspect ratio of the license plate image taken from the front should be around 3.143. Through geometric operations, it can be calculated that the aspect ratio of the license plate image with the shooting pitch angle and horizontal angle both within the range of 45° is between 2.223 and 4.444; (2) the pixel mean of the three RGB channels (red channel, green channel, blue channel) can be obtained by calculating the three-channel matrix of the input image; (3) the number of characters can be estimated by the watershed algorithm. First, the input three-channel color image is converted into a single-channel grayscale image. Secondly, the grayscale image is binarized, and the license plate text part is set to 1 and the background part is set to 0. Finally, each column of the image is summed to obtain a one-dimensional series. By counting the number of peaks in the series, the number of characters contained in the image can be roughly obtained.

[0085] like Figure 7As shown, it is a training diagram of the license plate recognition model provided by an embodiment of the present invention. First, prepare the data: training image data, license plate position coordinate data and license plate character content data, and then input these data into the initial character detection sub-model, the initial license plate determination sub-model and the initial character recognition sub-model for training respectively to obtain the final character detection sub-model, the final license plate determination sub-model and the final character recognition sub-model.

[0086] After the license plate recognition model is trained, the trained license plate recognition model can be used to perform license plate recognition on an image to be recognized. Accordingly, an embodiment of the present invention further provides a license plate recognition method, which may include: inputting the image to be recognized into the license plate recognition model obtained by the license plate recognition model training method described in the previous embodiments, so that:

[0087] Finally, the character detection sub-model determines the license plate position coordinate data of the image to be recognized;

[0088] The final license plate determination sub-model determines whether the image to be identified is a license plate image based on the image to be identified and its license plate position coordinate data;

[0089] The final character recognition sub-model identifies the license plate character content in the image to be recognized when the final license plate determination sub-model determines that the image to be recognized is a license plate image.

[0090] Specifically, the image to be recognized is input into the final character detection sub-model to obtain the license plate position coordinate data of the image to be recognized, the image to be recognized and the license plate position coordinate data are input into the final license plate determination sub-model, the final license plate determination sub-model segments the image to be recognized according to the license plate position coordinate data to obtain the image to be determined, and then determines whether the image to be determined is a license plate image, discards the image to be determined that is determined to be a non-license plate image, and inputs the image to be determined that is determined to be a license plate image into the final character recognition sub-model to obtain the license plate character content recognition result.

[0091] like Figure 8As shown, the license plate recognition flow chart provided by the embodiment of the present invention is mainly divided into two stages: model training and license plate recognition. In the model training stage, the training image data can be first labeled, and then the initial character detection sub-model, the initial license plate determination sub-model and the initial character recognition sub-model are trained until the accuracy of the three sub-models meets the standards, and the final character detection sub-model, the final license plate determination sub-model and the final character recognition sub-model are saved. In the license plate recognition stage, the image to be recognized is input into the final character detection sub-model for character detection. The final license plate image determination sub-model performs license plate image segmentation on the image to be recognized based on the license plate position coordinate data output by the final character detection sub-model to obtain the image to be determined. The image to be determined is then subjected to license plate image determination. If it is determined to be a license plate image, it is input into the final character recognition model for character recognition and outputs the recognition result. Otherwise, the process ends.

[0092] The steps of the various methods above are divided only for the purpose of clear description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0093] Based on the same technical concept, an embodiment of the present invention also provides a license plate recognition model training device, wherein the license plate recognition model includes a final character detection sub-model, a final license plate determination sub-model and a final character recognition sub-model. Figure 9 As shown, the device includes:

[0094] The first training module 101 is used to train the initial license plate determination sub-model based on the positive license plate image sample, the negative license plate image sample, the prior information of the negative license plate image sample and the prior information of the positive license plate image sample to obtain the final license plate determination sub-model, wherein the prior information of the positive license plate image sample and the prior information of the negative license plate image sample both include the image aspect ratio, image pixel information and the number of image characters, and the prior information of the positive license plate image sample and the prior information of the negative license plate image sample are both used to train the fully connected layer of the initial license plate determination sub-model.

[0095] In some embodiments, the first training module 101 may also be used to:

[0096] Determining labeled data for training image data;

[0097] The license plate image positive sample and the license plate image negative sample are determined according to the labeled data and the training image data.

[0098] In some embodiments, the labeled data includes license plate position coordinate data, and the first training module 101 is used to:

[0099] Determine non-license plate position coordinate data and the license plate image positive sample according to the license plate position coordinate data and the training image data;

[0100] The license plate image negative sample is determined according to the non-license plate position coordinate data and the training image data.

[0101] In some embodiments, the license plate recognition model may further include a second training module, which is used to: train an initial character detection sub-model based on the license plate position coordinate data and the training image data to obtain a final character detection sub-model, wherein the initial character detection sub-model includes a cascaded character detection convolution layer, a character detection pooling layer and a feature fusion layer.

[0102] In some embodiments, the annotated data also includes license plate character content data, and the license plate recognition model may further include a third training module, which is used to: train an initial character recognition sub-model based on the license plate character content data and the license plate image positive sample, wherein the initial character recognition sub-model includes a cascaded feature extraction layer, a character recognition pooling layer, a semantic learning layer and a character recognition fully connected layer.

[0103] In some embodiments, the initial license plate determination submodel includes a cascaded preprocessing module, a first license plate determination convolutional layer, a first license plate determination pooling layer, a second license plate determination convolutional layer, a second license plate determination pooling layer, and a license plate determination fully connected layer; the image pixel information includes the mean of red channel pixels, the mean of green channel pixels, and the mean of blue channel pixels. The license plate recognition model training device may further include an acquisition module, which is used to:

[0104] Obtaining the image aspect ratio of the positive sample of the license plate image and the negative sample of the license plate image;

[0105] Obtaining the red channel pixel mean, the green channel pixel mean, and the blue channel pixel mean of the license plate image positive sample and the license plate image negative sample according to the three-channel matrix;

[0106] The number of image characters of the positive sample of the license plate image and the negative sample of the license plate image are obtained according to the watershed algorithm.

[0107] Based on the same technical concept, the embodiment of the present invention also provides a license plate recognition device, such as Figure 10 As shown, the license plate recognition device includes:

[0108] The license plate recognition module 201 is used to input the image to be recognized into the license plate recognition model obtained by the license plate recognition model training method described in any of the previous embodiments, so that:

[0109] The final character detection sub-model determines the license plate position coordinate data of the image to be recognized;

[0110] The final license plate determination sub-model determines whether the image to be identified is a license plate image based on the image to be identified and the license plate position coordinate data;

[0111] The final character recognition sub-model identifies license plate character content in the image to be recognized when the final license plate determination sub-model determines that the image to be recognized is a license plate image.

[0112] It should be understood that the present invention is not limited to the specific configurations and processes described in the above embodiments and illustrated in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here. The specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] In addition, an embodiment of the present invention provides an electronic device, such as Figure 11 As shown, it includes:

[0114] One or more processors 301;

[0115] The memory 302 stores one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement any of the following:

[0116] The license plate recognition model training method as described in any of the previous embodiments;

[0117] The license plate recognition method as described in any of the previous embodiments;

[0118] One or more I / O interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

[0119] One or more I / O interfaces 303 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0120] Among them, the processor 201 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 202 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 203 is connected between the processor 201 and the memory 202, and can realize information interaction between the processor 201 and the memory 202, including but not limited to a data bus (Bus), etc.

[0121] In some embodiments, the processor 201 , the memory 202 , and the I / O interface 203 are connected to each other via a bus, and further connected to other components of the computing device.

[0122] This embodiment further provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements any of the following:

[0123] The license plate recognition model training method as described in any of the previous embodiments;

[0124] The license plate recognition method as described in any of the previous embodiments;

[0125] To avoid repeated description, the specific steps of the license plate recognition model training method and the license plate recognition method are not described here.

[0126] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods invented above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0127] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0128] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0129] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A license plate recognition model training method, characterized in that: The license plate recognition model includes a final character detection sub-model, a final license plate determination sub-model and a final character recognition sub-model. The method includes the steps of training an initial license plate determination sub-model to obtain the final license plate determination sub-model, wherein the training of the initial license plate determination sub-model includes: An initial license plate determination sub-model is trained based on a positive license plate image sample, a negative license plate image sample, prior information of the negative license plate image sample, and prior information of the positive license plate image sample, wherein the prior information of the positive license plate image sample and the prior information of the negative license plate image sample both include image aspect ratio, image pixel information, and the number of image characters, and the prior information of the positive license plate image sample and the prior information of the negative license plate image sample are both used to train a license plate determination fully connected layer of the initial license plate determination sub-model; the number of image characters is a number estimated and determined based on the number of characters in the positive license plate image sample and the negative license plate image sample according to a watershed algorithm; The annotated data also includes license plate character content data. After determining the image positive sample and the image negative sample based on the annotated data and the training image data, the method further includes the step of training an initial character recognition sub-model to obtain the final character recognition sub-model, wherein the training of the initial character recognition sub-model includes: Training an initial character recognition sub-model based on the license plate character content data and the license plate image positive sample, wherein the initial character recognition sub-model includes a cascaded feature extraction layer, a character recognition pooling layer, a semantic learning layer, and a character recognition fully connected layer; The semantic learning layer is a data processing layer implemented based on the long short-term memory network.

2. The method according to claim 1, wherein Before the step of training the initial license plate determination sub-model to obtain the final license plate determination sub-model, the method further includes: Determining labeled data for training image data; The license plate image positive sample and the license plate image negative sample are determined according to the labeled data and the training image data.

3. The method according to claim 2, wherein: The annotated data includes license plate position coordinate data, and determining the license plate image positive sample and the license plate image negative sample based on the annotated data and the training image data includes: Determine non-license plate position coordinate data and the license plate image positive sample according to the license plate position coordinate data and the training image data; The license plate image negative sample is determined according to the non-license plate position coordinate data and the training image data.

4. The method according to claim 3, wherein: After determining the annotated data of the training image data, the method further includes the step of training an initial character detection sub-model to obtain the final character detection sub-model, wherein the training of the initial character detection sub-model includes: An initial character detection sub-model is trained according to the license plate position coordinate data and the training image data, wherein the initial character detection sub-model includes a cascaded character detection convolution layer, a character detection pooling layer and a feature fusion layer.

5. The method according to claim 1, wherein The initial license plate determination submodel includes a cascaded preprocessing module, a first license plate determination convolutional layer, a first license plate determination pooling layer, a second license plate determination convolutional layer, a second license plate determination pooling layer, and a license plate determination fully connected layer; the image pixel information includes a red channel pixel mean, a green channel pixel mean, and a blue channel pixel mean. Before the step of training the initial license plate determination submodel to obtain the final license plate determination submodel, the method further includes a step of obtaining prior information, and the obtaining of prior information includes: Obtaining the image aspect ratio of the positive sample of the license plate image and the negative sample of the license plate image; Obtaining the red channel pixel mean, the green channel pixel mean, and the blue channel pixel mean of the license plate image positive sample and the license plate image negative sample according to the three-channel matrix; The number of image characters of the positive sample of the license plate image and the negative sample of the license plate image are obtained according to the watershed algorithm.

6. A license plate recognition method, characterized in that: The method comprises: The image to be recognized is input into the license plate recognition model obtained by the license plate recognition model training method according to any one of claims 1 to 5, so that: The final character detection sub-model determines the license plate position coordinate data of the image to be recognized; The final license plate determination sub-model determines whether the image to be identified is a license plate image based on the image to be identified and the license plate position coordinate data; The final character recognition sub-model identifies license plate character content in the image to be recognized when the final license plate determination sub-model determines that the image to be recognized is a license plate image.

7. A license plate recognition model training device, characterized in that: The license plate recognition model includes a final character detection sub-model, a final license plate determination sub-model and a final character recognition sub-model, and the device includes: a first training module, for training an initial license plate determination sub-model based on a positive license plate image sample, a negative license plate image sample, prior information of the negative license plate image sample, and prior information of the positive license plate image sample to obtain the final license plate determination sub-model, wherein the prior information of the positive license plate image sample and the prior information of the negative license plate image sample both include image aspect ratio, image pixel information, and the number of image characters, and the prior information of the positive license plate image sample and the prior information of the negative license plate image sample are both used to train the license plate determination fully connected layer of the initial license plate determination sub-model; the number of image characters is a number estimated and determined by the number of characters in the positive license plate image sample and the negative license plate image sample based on a watershed algorithm; The annotated data also includes license plate character content data, and the license plate recognition model also includes a third training module, which is used to: train an initial character recognition sub-model based on the license plate character content data and the license plate image positive sample, wherein the initial character recognition sub-model includes a cascaded feature extraction layer, a character recognition pooling layer, a semantic learning layer and a character recognition fully connected layer; the semantic learning layer is a data processing layer implemented based on a long short-term memory network.

8. An electronic device, characterized in that: include: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the following: The license plate recognition model training method according to any one of claims 1 to 5; The license plate recognition method according to claim 6; One or more I / O interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

9. A computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the computer program performs any of the following: The license plate recognition model training method according to any one of claims 1 to 5; The license plate recognition method according to claim 6.

Images