Character non-segmentation mode license plate recognition method based on convolutional neural network model

Abstract

The application discloses a character non-segmentation mode license plate recognition method based on a convolutional neural network model, which comprises the following steps: detecting a license plate region and constructing a highway license plate image set based on a high-speed whole vehicle image under abnormal weather conditions; researching a character non-segmentation mode license plate recognition framework, and constructing a model by optimizing a multi-classification character non-segmentation mode license plate recognition framework based on a CB loss function; and constructing a convolutional neural network model to extract features of the license plate, and preferably selecting InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SE-Net-LPR-CB convolutional neural network models to complete the recognition of the vehicle license plate. The application has the beneficial effects that a license plate image set suitable for neural network learning and conforming to a real highway scene is constructed, a vehicle license plate recognition framework under a non-segmentation mode is provided, a convolutional neural network model with reasonable structure is selected, automatic recognition of the vehicle license plate is completed, the accuracy and speed of the license plate recognition are improved, and an effective new idea is provided for the vehicle license plate recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent transportation and wisdom highway information perception, and particularly relates to a character non-segmentation mode license plate recognition method based on a convolutional neural network model. BACKGROUND

[0002] In order to improve the efficiency of highway transportation, the non-stop charging system of the highway has become a key part of the realization of the wisdom and informatization of the highway, and the perception of the holographic information of the vehicle in the highway scene, especially the recognition of the license plate of the vehicle, is one of the important contents of the construction of the non-stop charging system of the wisdom highway. However, due to the single recognition mode of the license plate of the vehicle, the existing license plate recognition model has poor effect under the open-air scene characteristics of some highways, especially in special weather conditions such as rain, snow and fog, the light changes significantly, and the image noise is more, which greatly reduces the accuracy of the recognition of the license plate of the vehicle. The non-stop charging system of the wisdom highway urgently needs to optimize and innovate the perception of the license plate of the vehicle, and cope with the complex and changeable open-air environment through various license plate recognition methods. Therefore, based on the highway vehicle image under abnormal weather conditions, a multi-class character non-segmentation mode license plate recognition framework is constructed, and an InceptionV3-LPR-CB convolutional neural network is used as the core structure of the model, which provides an effective new idea for the recognition of the license plate of the vehicle. SUMMARY

[0003] The purpose of the application is to overcome the shortcomings in the existing vehicle license plate recognition technology, and to provide a character non-segmentation mode license plate recognition method based on a convolutional neural network model, which realizes the recognition of the license plate in the character non-segmentation mode under the framework of the character non-segmentation mode license plate recognition based on the CB (effective sample number) loss function, and combines the InceptionV3-LPR-CB convolutional neural network model, which provides new vehicle perception technology support for the non-stop charging system in the highway scene.

[0004] The technical scheme is that in order to achieve the above purpose, the character non-segmentation mode license plate recognition method based on the convolutional neural network model provided by the present application comprises the following steps:

[0005] S1: based on the highway vehicle image under abnormal weather conditions, constructing a highway license plate image set;

[0006] S2: based on the CB (effective sample number) loss function, constructing a multi-class character non-segmentation mode license plate recognition framework;

[0007] S3: constructing a convolutional neural network model for feature extraction of the license plate, and optimizing the InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SENet-LPR-CB convolutional neural network models to complete the recognition of the license plate of the vehicle.

[0008] Further, the construction method of the expressway license plate image set based on the high-speed vehicle image under abnormal weather conditions in step S1 is as follows: the YOLOv3-based expressway vehicle holographic detection method is used to detect the constructed vehicle data set to obtain vehicle holographic detection results under various weather conditions. The detection content includes vehicle windows, vehicle faces and license plates. Then, the obtained license plate images are screened and sorted, and the license plate labels are corrected. Finally, 13733 pieces of license plate data sets under abnormal weather conditions are constructed, including 2144 pieces of rain day license plate images, 2505 pieces of snow day license plate images, 2151 pieces of fog day license plate images, 2256 pieces of night license plate images, and 4677 pieces of license plate images under normal weather conditions.

[0009] Further, the specific construction steps of constructing a multi-class character non-segmentation mode license plate recognition framework based on the CB (effective sample number) loss function in step S2 are as follows:

[0010] ①The number of effective samples of various classes is counted to determine the calculation method of the number weight part in the CB loss function;

[0011] ②The difficulty of various samples is calculated to determine the calculation method of the difficulty weight part in the CB loss function;

[0012] ③The specific expression of the CB loss function is formed, and the model framework is built according to the loss function.

[0013] Further, the step S3 of constructing the convolutional neural network model for feature extraction of license plates, preferably the InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SENet-LPR-CB convolutional neural network model complete the vehicle license plate recognition method: construct the InceptionV3-LPR-CB convolutional neural network model composed of Block1, Block2, Block3, Block4, Block5, the loss function adopts the CB (effective sample number) based loss function, and the main network structure adopts the basic network structure characteristics of InceptionV3. Construct the ResNet50-LPR-CB convolutional neural network composed of 5 convolution modules (conv2d), the loss function adopts the CB (effective sample number) based loss function, and the main network structure adopts the basic network structure characteristics of ResNet50. Construct the ResNeXt-LPR-CB convolutional neural network composed of 5 convolution modules (conv2d), the loss function adopts the CB (effective sample number) based loss function, and the main network structure is based on ResNet, keeping the first convolution module unchanged, and increasing the number of convolution kernel of the second to fourth convolution modules by two times. Construct the SENet-LPR-CB convolutional neural network added with SE module on the ResNeXt network, the SE module adopts two fully connected layers to learn the importance of features, the module first uses global average pooling to change the original HxWxC feature map to 1x1xC, where C is the channel number, secondly uses a fully connected layer to change the channel size of the feature map to one sixteenth of the original, and then uses a fully connected layer with ReLU activation function to change the channel size to the original size, and finally uses a Sigmoid activation function to change the feature size to between 0 and 1, that is, the weight. The loss function of the convolutional neural network adopts the CB (effective sample number) based loss function, and the main network structure adds the SE module in the middle of each residual module of the ResNeXt network.

[0014] After the vehicle license plate image feature extraction is completed by the InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SENet-LPR-CB models, an 8x8 maximum pooling layer, a full connection layer and a Dropout layer are connected, followed by a reshape layer, a full connection layer and a softmax function to realize the recognition of the vehicle license plate. When recognizing, the input image should be normalized to 299x299x3, and after the convolution operation of Block1, Block2, Block3, Block4 and Block5, an 8x8 kernel global maximum pooling layer is used to operate to obtain a 2048-dimensional feature vector, a full connection layer is used to reduce the 2048-dimensional feature vector to a 910-dimensional vector, the 910-dimensional vector is reshaped into 7 130-dimensional vectors, and a full connection layer is used to reduce the 7 130-dimensional vectors to 7 65-dimensional vectors, and finally a softmax classification operation is used on the 65 corresponding dimensions to obtain the distribution probability of the 7 characters of the image on the 65 character classes. The one with the highest probability is the final license plate recognition result.

[0015] The application combines the target detection model of YOLOv3, the multi-classification character non-segmentation mode license plate recognition framework based on the CB (effective sample number) loss function and the InceptionV3-LPR-CB convolutional neural network model, and completes the license plate recognition task without segmenting the license plate characters. The method first acquires the license plate region from the whole vehicle image through the YOLOv3 target detection algorithm, then introduces the CB (effective sample number) based loss function into the neural network optimization algorithm, and finally completes the license plate recognition based on the InceptionV3-LPR-CB convolutional neural network model.

[0016] The application has the beneficial effects that the license plate region can be automatically recognized in the whole vehicle image, a reasonable multi-classification character non-segmentation mode license plate recognition framework is used, a corresponding loss function is set, a highway license plate image set for convolutional neural network training is constructed, the vehicle license plate recognition is realized, and technical support is provided for the vehicle holographic information perception in the highway scene. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a clear license plate image example for use.

[0018] Figure 2 It is a CB loss function class weighting parameter based on the effective sample number.

[0019] Figure 3 It is the InceptionV3-LPR-CB network model structure and parameters.

[0020] Figure 4 ResNet50-LPR-CB network model structure and parameters.

[0021] Figure 5 ResNeXt-LPR-CB network model structure and parameters.

[0022] Figure 6 SENet-LPR-CB network model structure and parameters.

[0023] Figure 7 InceptionV3-LPR, ResNeXt-LPR, ResNet50-LPR, and SENet-LPR four networks under the model accuracy of CE and CB different loss functions. DETAILED DESCRIPTION

[0024] The application provides a character non-segmentation mode license plate recognition method based on a convolutional neural network model. The following specific embodiments are only used for describing the application and are not used for limiting the scope of the application.

[0025] S1: based on the high-speed whole vehicle image under abnormal weather conditions, a highway license plate image set is constructed.

[0026] Specifically, it is:

[0027] First, the actual vehicle image collected on the highway at high speed is taken as the benchmark to construct the vehicle data set under abnormal weather conditions, a total of 13988 vehicle images, of which 2372 are foggy, 2645 are snowy, 2248 are at night, 2112 are rainy, and 4611 are normal weather. Second, the YOLOv3 target detection model is used to detect the vehicle images collected on the highway under abnormal weather conditions to obtain the holographic detection results of the vehicle under various weather conditions, including the vehicle window, vehicle face and license plate. Then the obtained license plate images are screened and sorted, and the license plate label is corrected to construct the highway license plate data set under abnormal weather conditions, a total of 13733, of which 2144 are rainy, 2505 are snowy, 2151 are foggy, 2256 are at night, and 4677 are normal weather. Finally, the license plate data is sorted and the data label is confirmed, and the license plate is divided into license plates that can determine the license plate number and license plates that cannot determine the license plate number, of which 10316 can determine the license plate number and 3417 cannot determine the license plate number. For images that can determine the license plate number, although they have problems such as blur, rain and snow obstruction, low contrast, etc. to some extent, they can clearly determine the license plate number information, retain and prepare to input the neural network for training; for images that cannot determine the license plate number, their resolution is too low or too blurred, and they cannot determine the complete license plate number information, especially the Chinese characters in the license plate information, so they cannot determine the label, this part of the license plate should not be used for subsequent experiments, should be deleted, and the clear license plate image available for use is shown as Figure 1

[0028] S2: Construct a multi-class character non-segmentation mode license plate recognition framework based on CB (effective sample number) loss function;

[0029] Specifically,

[0030] ① The number of effective samples of various types of samples is counted to determine the calculation method of the number weight part of the CB loss function.

[0031] ​For the unbalanced classification problem, there are both classes with a large number of samples and classes with a small number of samples. In theory, the more samples, the higher the coverage of the feature space of the class, and the better the learning effect of the class. However, due to the information overlap between data, the more samples, the less marginal effect the network model can get from the data. In order to better cover the feature space, assume that the set of all possible sample data in the feature space is S, the volume of S is N, and N >= 1. Assume that the volume of each sample in S is 1, and there is a possibility of sample overlap. Now sample from S to cover the set S. The more sampling data, the better the coverage of S, and the expected total amount of sampling data increases with the number of samples. Define the effective sample number as the expected volume of the sample. Assume that N is the total number of samples, E n is the effective number of samples. The probability of a newly sampled data point in the already sampled data set is P, and the probability of not being in the already sampled data set is 1-P. Assume that there is beta, and beta = (N-1) / N. When n = 1, there is no overlap, and E1 = 1. Write E1 as:

[0032]

[0033] Assume that there are n-1 samples, and sample the nth sample. The expected volume of the previously sampled sample is E n-1 , and the overlap probability p of the nth sample and the previously sampled sample is:

[0034]

[0035] The expected volume of the nth sample after sampling can be expressed as:

[0036]

[0037] Where E n-1 can be expressed as:

[0038]

[0039] Then we can get:

[0040]

[0041] Then the class balance loss function based on the effective sample number can be written as:

[0042]

[0043] According to the statistical results of each sample in the data set, the number weight value is as shown in Figure 2 , which can be used to complete the calculation of the number loss function part.

[0044] ②Calculate the difficulty of various samples to determine the calculation method of the difficulty weight part in the CB loss function.

[0045] Since a large number of simple samples have small contribution to the model, more complex sampling needs to be taken for difficult samples, and the samples are reweighted. The focal loss function calculation method is as follows. For samples with high probability tends to 0, and the loss value can be reduced, and the difficult sample that is difficult to distinguish has a low true probability, has less impact on loss, so that the loss value of simple samples can be reduced while the contribution of difficult samples to the model is improved.

[0046]

[0047] The class-balanced focal loss function formula can be written as:

[0048]

[0049] The difficulty weight is brought into the above formula to complete the calculation of the difficulty loss function part.

[0050] ③Form the specific expression of the CB loss function, and build the model framework according to the loss function.

[0051] S3: Construct a convolutional neural network model to extract features of the license plate, and preferably use InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SE- Net-LPR-CB models to complete the recognition of vehicle license plates.

[0052] Specifically,

[0053] The InceptionV3-LPR-CB convolutional neural network model is composed of Block1, Block2, Block3, Block4 and Block5, and its loss function adopts the CB (effective sample number) loss function. The main network structure adopts the basic network structure characteristics of InceptionV3, and the InceptionV3-LPR-CB network structure and parameters are as shown in Figure 3 The ResNet50-LPR-CB convolutional neural network is composed of five convolution modules (conv2d), and its loss function adopts the CB (effective sample number) loss function. The main network structure adopts the basic network structure characteristics of ResNet50, and the ResNet50-LPR-CB network structure and parameters are as shown in Figure 4The ResNeXt-LPR-CB convolutional neural network is composed of 5 convolution modules (conv2d), and the loss function adopts the CB (effective sample number) based loss function. The main network structure is based on ResNet, the first convolution module is kept unchanged, the number of convolution kernel of the second to fourth convolution module is doubled, and the ResNeXt-LPR-CB network structure and parameters are as shown in Table 2. Figure 5 The SENet-LPR-CB convolutional neural network adds an SE module to the ResNeXt network. The SE module uses two fully connected layers to learn the importance of features. The module first uses global average pooling to change the feature map from HxWxC to 1x1xC, where C is the channel number. Then, the fully connected layer changes the channel number of the feature map to one-sixteenth of the original size. Then, the fully connected layer with ReLU activation function changes the channel number to the original size. Finally, the Sigmoid activation function changes the feature size to between 0 and 1, which is the weight. The loss function of the SENet-LPR-CB convolutional neural network adopts the CB (effective sample number) based loss function. The main network structure adds an SE module in the middle of each residual module of the ResNeXt network. The SENet-LPR-CB network structure and parameters are as shown in Table 3. Figure 6

[0054] During recognition, the input image should be normalized to 299x299x3. After extracting the features through the corresponding network, the global maximum pooling layer is used for operation, and a 2048-dimensional feature vector can be obtained. Then, the vector is reduced to 910-dimensional through a fully connected layer. The 910-dimensional vector is reshaped into 7 130-dimensional vectors, and the 7 130-dimensional vectors are reduced to 7 65-dimensional vectors through a fully connected layer. Finally, the softmax classification operation is performed on the corresponding 65-dimensional dimensions, and the distribution probability of the 7 characters in the image on the 65 characters can be obtained. The one with the highest probability is the final license plate recognition result.

[0055] ​In this example, InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SE- Net-LPR-CB four multi-classification models based on CE loss function and InceptionV3-LPR- CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SE-Net-LPR-CB multi-classification models based on CB loss function are compared. 80% of the image set is used for training, 10% is used for verification, and 10% is used for testing. The batch size is 24, the learning rate is 0.0001, and the results show that the test accuracy of the four models for the whole license plate (7 characters in total) is 93.553%, 95.132%, 91.316% and 91.776% respectively. Obviously, the multi-classification model framework based on CB loss function has higher recognition accuracy, which proves that in the two character non-segmentation mode license plate recognition framework, the character non-segmentation mode license plate recognition framework based on CB (effective sample number) loss function is the best. At the same time, under the condition of selecting CB multi-classification model as the character non-segmentation mode recognition framework, ResNeXt-LPR-CB, ResNet50-LPR-CB, InceptionV3-LPR-CB and SE-Net-LPR-CB different convolutional neural network models are compared. 80% of the image set is used for training, 10% is used for verification, and 10% is used for testing. The batch size is 24, the learning rate is 0.0001, and the results show that the InceptionV3-LPR-CB network model performs best on the test set and the validation set, with an identification accuracy of 94.455% and 95.132% for the whole license plate (7 characters in total), which proves that among the four different convolutional neural network models, InceptionV3-LPR-CB is the best. The experimental results are shown in Figure 7 Fig. 6.

Claims

1. A character non-segmentation mode license plate recognition method based on a convolutional neural network model, characterized in that, Comprise the following steps: S1: based on the abnormal weather conditions under the high-speed vehicle image, construct highway license plate image set; S2: based on CB loss function, construct multi-class character non-segmentation mode license plate recognition framework; S3: construct convolutional neural network model for feature extraction of license plate, complete the recognition of vehicle license plate through InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SENet-LPR-CB convolutional neural network model; wherein, the InceptionV3-LPR-CB convolutional neural network model is composed of Block1, Block2, Block3, Block4, Block5, its loss function adopts CB loss function, and the main network structure adopts the basic network structure characteristics of InceptionV3; the ResNet50-LPR-CB convolutional neural network model is composed of 5 convolution modules conv2d, its loss function adopts CB loss function, and the main network structure adopts the basic network structure characteristics of ResNet50; the ResNeXt-LPR-CB convolutional neural network model is composed of 5 convolution modules conv2d, its loss function adopts CB loss function, and the main network structure is based on ResNet50, keeps the first convolution module unchanged, and increases the number of convolution kernel of the second to fifth convolution module by two times; the SENet-LPR-CB convolutional neural network model adds SE module on ResNeXt network, SE module adopts two fully connected layers to learn the importance of features, which firstly adopts global average pooling to change the feature map from HxWxC to 1x1xC, wherein C is the channel number, secondly adopts fully connected layer to change the channel number of feature map to one sixteenth of the original, and then adopts fully connected layer with ReLU activation function to change the channel number to the original size, finally adopts Sigmoid activation function to change the feature size to between 0 and 1, that is, the weight; the loss function of the SENet-LPR-CB convolutional neural network model adopts CB loss function, and the main network structure adds SE module in the middle of each residual module of ResNeXt network; The normalized license plate image is input into the four convolutional neural network models, i.e., InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SE-Net-LPR-CB, to obtain the distribution probability of each of the seven characters, and the class corresponding to the maximum probability value is the license plate class predicted by the vector, i.e., the corresponding classes of the seven characters are obtained simultaneously, realizing the recognition of the vehicle license plate in the character non-segmentation mode; finally, the recognition accuracies of the four convolutional neural network models, i.e., InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SE-Net-LPR-CB, are compared, and the optimal license plate recognition method in the character non-segmentation mode can be selected. 2.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 1, wherein, The method for constructing the expressway license plate image set in the step S1 based on the whole vehicle image under abnormal weather conditions is as follows: the constructed vehicle data set is detected by using the YOLOv3-based abnormal weather condition expressway vehicle holographic detection method to obtain the vehicle holographic detection result under various weather conditions. 3.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 2, wherein, The whole vehicle data set under abnormal weather conditions on the expressway has a total of 13988 vehicle images, including 2372 foggy vehicle images, 2645 snowy vehicle images, 2248 night vehicle images, 2112 rainy vehicle images and 4611 normal weather vehicle images. After using the YOLOv3 target detection model to perform license plate region perception and screening and sorting the license plate images, the abnormal weather condition expressway license plate data set is obtained, including 13733 images, of which 2144 are rainy license plate images, 2505 are snowy license plate images, 2151 are foggy license plate images, 2256 are night license plate images and 4677 are normal weather condition license plate images. 4.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 1, wherein, The specific steps for constructing the multi-class character non-segmentation mode license plate recognition framework based on the CB loss function in the step S2 are as follows: ①The effective number of various types of samples is counted to determine the calculation method of the number weight part in the CB loss function; ②The difficulty of various samples is calculated to determine the calculation method of the difficulty weight part in the CB loss function; ③The specific expression of the CB loss function is formed, and the model framework is built according to the loss function. 5.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 1, wherein, The method for constructing the InceptionV3-LPR-CB convolutional neural network model to perform feature extraction of the license plate in the step S3 is as follows: the size of the license plate image input into the InceptionV3-LPR-CB convolutional neural network model is 299*299*3, and after the convolution operation of Block1, Block2, Block3, Block4 and Block5, an 8*8*2048-dimensional feature map is obtained, and after the operation of the 8*8 kernel global maximum pooling layer, a 2048-dimensional feature vector is obtained. 6.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 1, wherein, The method for constructing the ResNet50-LPR-CB convolutional neural network model in step S3 to extract features of the license plate is as follows: the size of the expressway license plate image under abnormal weather conditions input into the ResNet50-LPR-CB convolutional neural network model is 224x224x3, and after convolution operations of conv1, conv2, conv3, conv4 and conv5, a feature map with a dimension of 7x7x2048 is obtained, and after operation of a global average pooling layer with a 7x7 kernel, a feature vector with a dimension of 2048 is obtained. 7.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 1, wherein, The method for constructing the ResNeXt-LPR-CB convolutional neural network model in step S3 to extract features of the license plate is as follows: the size of the expressway license plate image under abnormal weather conditions input into the ResNeXt-LPR-CB convolutional neural network model is 224x224x3, and after convolution operations of conv1, conv2, conv3, conv4 and conv5, a feature map with a dimension of 7x7x2048 is obtained, and after operation of a global average pooling layer with a 7x7 kernel, a feature vector with a dimension of 2048 is obtained. 8.The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 1, wherein, The method for constructing the ResNeXt-LPR-CB convolutional neural network model in step S3 to extract features of the license plate is as follows: the size of the expressway license plate image under abnormal weather conditions input into the ResNeXt-LPR-CB convolutional neural network model is 224x224x3, and after convolution operations of conv1, conv2, conv3, conv4 and conv5, a feature map with a dimension of 7x7x2048 is obtained, and after operation of a global average pooling layer with a 7x7 kernel, a feature vector with a dimension of 2048 is obtained.

9. The character non-segmentation mode license plate recognition method based on a convolutional neural network model according to claim 5 or 6 or 7 or 8, characterized in that, The 2048-dimensional feature vectors extracted by the InceptionV3-LPR-CB, ResNet50-LPR-CB, ResNeXt-LPR-CB and SENet-LPR-CB convolutional neural network models are all reduced to 910-dimensional vectors by a fully connected layer, the 910-dimensional vectors are reshaped into 7 vectors with a dimension of 130, and the 7 vectors with a dimension of 130 are reduced to 7 vectors with a dimension of 65 by a fully connected layer, and a softmax classification operation is performed on the 65 corresponding dimensions, so that the distribution probability of the 7 characters of the image on 65 types of characters is obtained.

Images