License plate recognition methods, devices, computer equipment and storage media

Abstract

This application relates to artificial intelligence, specifically to a license plate recognition method, apparatus, computer device, and storage medium. By constructing a license plate recognition model, simple images are predicted using a shallow network, while complex images are predicted using a deep network. This allows for prediction of license plate images at different stages of the model based on their complexity. For simple license plate images, prediction results can be obtained at the shallow layers of the model, without requiring the entire model to process the correct result, thus improving the overall efficiency of license plate recognition.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and image recognition technology, and in particular to a license plate recognition method, device, computer equipment, and storage medium. Background Technology

[0002] With the development of image recognition technology, intelligent license plate recognition is being used more and more widely in actual business scenarios. For example, the application of intelligent license plate recognition in access control systems has greatly improved access control processing efficiency.

[0003] To improve license plate recognition accuracy, neural network license plate recognition models are typically used. In real-world business scenarios, these trained models are utilized for recognition. Commonly used neural network structures include transformers. It is generally believed that the deeper the model structure, the richer the feature extraction information, and the higher the recognition accuracy. Therefore, in practical applications, multi-layered network structures are often trained for license plate recognition. However, correspondingly, the deeper the model, the lower the processing efficiency, resulting in the low recognition efficiency of existing license plate recognition models. Summary of the Invention

[0004] Therefore, it is necessary to provide a license plate recognition method, device, computer equipment, and storage medium that can improve recognition efficiency in response to the above-mentioned technical problems.

[0005] A license plate recognition method, the method comprising:

[0006] Obtain license plate image;

[0007] The license plate image is input into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the feature extraction layers of the backbone network that are not the last layer.

[0008] According to the order of depth of the feature extraction layers of the license plate recognition model from shallow to deep, the feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain prediction results based on the license plate features.

[0009] When the prediction result of the stage classifier meets the requirements, the prediction result of the stage classifier is used as the license plate recognition result. When the prediction result of the stage classifier does not meet the requirements, the license plate features of the stage feature extraction layer at this depth are input into the feature extraction layer at the next depth for processing. The feature extraction layer at the next depth and the corresponding classifier continue to perform feature extraction and prediction processing respectively until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is used as the license plate recognition result.

[0010] A license plate recognition device, the device comprising:

[0011] The image acquisition module is used to acquire license plate images;

[0012] The input module is used to input the license plate image into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the non-last feature extraction layer of the backbone network.

[0013] The prediction module is used to perform feature extraction and prediction processing according to the depth of the feature extraction layers of the license plate recognition model from shallow to deep. The feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain the prediction results based on the license plate features.

[0014] The output module is used to take the prediction result of the stage classifier as the license plate recognition result when the prediction result of the stage classifier meets the requirements, and to input the license plate features of the stage feature extraction layer of the current depth into the feature extraction layer of the next depth for processing when the prediction result of the stage classifier does not meet the requirements. The feature extraction and prediction processes are then carried out by the feature extraction layer of the next depth and the corresponding classifier, until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is taken as the license plate recognition result.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0016] Obtain license plate image;

[0017] The license plate image is input into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the feature extraction layers of the backbone network that are not the last layer.

[0018] According to the order of depth of the feature extraction layers of the license plate recognition model from shallow to deep, the feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain prediction results based on the license plate features.

[0019] When the prediction result of the stage classifier meets the requirements, the prediction result of the stage classifier is used as the license plate recognition result. When the prediction result of the stage classifier does not meet the requirements, the license plate features of the stage feature extraction layer at this depth are input into the feature extraction layer at the next depth for processing. The feature extraction layer at the next depth and the corresponding classifier continue to perform feature extraction and prediction processing respectively until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is used as the license plate recognition result.

[0020] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0021] Obtain license plate image;

[0022] The license plate image is input into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the feature extraction layers of the backbone network that are not the last layer.

[0023] According to the order of depth of the feature extraction layers of the license plate recognition model from shallow to deep, the feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain prediction results based on the license plate features.

[0024] When the prediction result of the stage classifier meets the requirements, the prediction result of the stage classifier is used as the license plate recognition result. When the prediction result of the stage classifier does not meet the requirements, the license plate features of the stage feature extraction layer at that depth are input into the feature extraction layer at the next depth for processing. The feature extraction layer at the next depth and the corresponding classifier at the same depth continue to perform feature extraction and prediction processing respectively, until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is used as the license plate recognition result.

[0025] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the following steps:

[0026] Obtain license plate image;

[0027] The license plate image is input into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the feature extraction layers of the backbone network that are not the last layer.

[0028] According to the order of depth of the feature extraction layers of the license plate recognition model from shallow to deep, the feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain prediction results based on the license plate features.

[0029] When the prediction result of the stage classifier meets the requirements, the prediction result of the stage classifier is used as the license plate recognition result. When the prediction result of the stage classifier does not meet the requirements, the license plate features of the stage feature extraction layer at this depth are input into the feature extraction layer at the next depth for processing. The feature extraction layer at the next depth and the corresponding classifier continue to perform feature extraction and prediction processing respectively until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is used as the license plate recognition result.

[0030] The aforementioned license plate recognition method, device, computer equipment, and storage medium construct a license plate recognition model. This model enables simple images to be predicted using a shallow network, while complex images are predicted using a deep network. Consequently, the model can predict license plate images at different stages based on their complexity. For simple license plate images, prediction results can be obtained at the shallow layer of the model, without needing to go through the entire model to obtain the correct result, thus improving the overall efficiency of license plate recognition. Attached Figure Description

[0031] Figure 1 This is an application environment diagram of the license plate recognition method in one embodiment;

[0032] Figure 2 This is a flowchart illustrating a license plate recognition method in one embodiment;

[0033] Figure 3 This is a schematic diagram of the license plate recognition model in one embodiment;

[0034] Figure 4 This is a schematic diagram of the license plate recognition model in another embodiment;

[0035] Figure 5 This is a flowchart illustrating the license plate recognition model training process in one embodiment;

[0036] Figure 6 This is a schematic diagram of the license plate recognition model training process in one embodiment;

[0037] Figure 7 This is a schematic diagram illustrating the application of a license plate recognition model in one embodiment;

[0038] Figure 8 This is a structural block diagram of a license plate recognition device in one embodiment;

[0039] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0044] The solutions provided in this application involve technologies such as artificial intelligence and computer vision, which are specifically illustrated through the following embodiments:

[0045] The license plate recognition method provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. The terminal is installed in the license plate recognition application environment. The terminal captures a license plate image and sends it to the server. Server 104 runs a license plate recognition model. Server 104 calls the license plate recognition model to recognize the license plate image and obtain the license plate recognition result. Server 104 returns the license plate recognition result to terminal 102. Terminal 102 is a device with camera functionality, and can be, but is not limited to, various personal computers, laptops, smartphones, tablets, portable wearable devices, and cameras installed in fixed locations. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0046] In one implementation, the terminal 102 can further query vehicle owner information based on the returned license plate recognition results. For example, traffic police can take a photo using a portable terminal, obtain the license plate through recognition, and retrieve the vehicle owner information based on the license plate, avoiding tedious data entry and improving efficiency. Similarly, community security guards can take a photo using a portable terminal, obtain the license plate through recognition, and quickly query whether the vehicle belongs to their community to find the vehicle owner information.

[0047] In one embodiment, the terminal 102 may run an access control system. The terminal 102 records vehicle information based on the license plate recognition results returned by the server and controls the vehicle to pass through the access control system.

[0048] In one embodiment, such as Figure 2 As shown, a license plate recognition method is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0049] Step 202: Obtain the license plate image.

[0050] The process involves capturing license plate images using cameras installed in the license plate recognition environment and sending them to a server for recognition. License plate recognition applications are typically located at checkpoints, such as parking lots, highway toll stations, and residential areas.

[0051] Step S204: Input the license plate image into the pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected in sequence, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the non-last feature extraction layer of the backbone network.

[0052] Specifically, such as Figure 3As shown, the backbone network 300 includes multiple feature extraction layers 3011 and a deep classifier 3022. The multiple feature extraction layers 3011 are connected sequentially, and the deep classifier 3022 is connected after the last feature extraction layer 3011. The feature extraction layer is the feature extraction network structure located before the deep classifier in the license plate recognition model. In one embodiment, the backbone network is a Transformer, and the feature extraction layers are deep neural networks with a Transformer structure. The feature extraction layers have multiple layers with different depths; as the depth increases, the details of the extracted features become richer. Therefore, in terms of depth, the depth of the deep classifier is greater than the depth of the stage classifier.

[0053] Specifically, the structure of the license plate recognition model is as follows: Figure 3 As shown, the system includes a backbone network 300, which comprises multiple feature extraction layers 3011 and a deep classifier 3022. The feature extraction layers 3011 are connected sequentially, and the deep classifier 3022 is connected after the last feature extraction layer 3011. The license plate recognition model also includes a stage classifier 3021 connected to a non-last feature extraction layer 3011 of the backbone network 300.

[0054] In this design, a stage classifier can be connected after each feature extraction layer (3011). The feature extraction layers are multi-layered, with each layer connected to a classifier of the same depth. The classifier's function is to predict license plate information based on the license plate features extracted by the connected feature extraction layers.

[0055] Step S206: In order of increasing depth of the feature extraction layers of the license plate recognition model, the feature extraction layers and their corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain prediction results based on the license plate features.

[0056] Specifically, according to the depth of the feature extraction layers of the license plate recognition model from shallow to deep, the shallow feature extraction layers and their corresponding classifiers first perform feature extraction and prediction processing, and then, according to the depth order, the next depth feature extraction layers and their corresponding classifiers perform feature extraction and prediction processing.

[0057] Specifically, a stage transition layer is connected between the stage classifier and the feature extraction layer. The stage transition layer includes at least one fully connected layer and an activation function.

[0058] Step S208: When the prediction result of the stage classifier meets the requirements, the prediction result of the stage classifier is used as the license plate recognition result. When the prediction result of the stage classifier does not meet the requirements, the license plate features of the stage feature extraction layer at this depth are input into the feature extraction layer at the next depth for processing. The feature extraction layer at the next depth and the corresponding classifier continue to perform feature extraction and prediction processing respectively until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is used as the license plate recognition result.

[0059] It is understandable that the processing of the feature extraction layer and the classifier is carried out in depth, with the previous stage feature extraction layer and the stage classifier handling the processing first.

[0060] When the prediction result of the stage classifier meets the requirements, it indicates that the prediction result obtained by the shallow stage classifier is satisfactory, and this prediction result is directly used as the license plate recognition result. When the prediction result of the stage classifier does not meet the requirements, it indicates that the shallow stage classifier cannot recognize the license plate, and the license plate features of this stage feature extraction layer are input into the next feature extraction layer for further feature extraction, and the next layer classifier continues to predict. In other words, when a shallow classifier cannot recognize the license plate, it continues to be processed by a deeper feature extraction layer and classifier. The deeper feature extraction layer performs further calculations to extract deeper features, and then the classifier of this layer is used for prediction. The effect of this is that simple license plate images can be recognized by a shallower feature extraction layer and classifier, while complex license plate images, when the shallower feature extraction layer and classifier cannot predict them, continue to be processed by the next feature extraction layer until a satisfactory prediction result is obtained on a stage classifier, or until a deeper classifier is reached, and the prediction result of the deeper classifier is used as the license plate recognition result. That is, simple images are predicted by shallow networks, and complex images are predicted by deep networks.

[0061] Whether the prediction results meet the requirements can be judged based on the confidence level of the prediction results.

[0062] In this embodiment, a license plate recognition model is constructed, which is used to predict simple images in a shallow network and complex images in a deep network. This allows the model to predict license plate images at different stages based on their complexity. For simple license plate images, the prediction result can be obtained at the shallow layer of the model without going through the entire model, thus improving the overall efficiency of license plate recognition.

[0063] In another embodiment, the stage feature extraction layer includes a shallow feature extraction layer and a middle feature extraction layer, and the stage classifier includes a shallow classifier connected to the shallow feature extraction layer and a middle classifier connected to the middle feature extraction layer.

[0064] Specifically, the license plate recognition model in this embodiment is as follows: Figure 4 As shown, there are three depth-level feature extraction layers: a shallow feature extraction layer 4011, a middle feature extraction layer 4012, and a deep feature extraction layer 4013. The shallow feature extraction layer is connected to a shallow classifier 4021, the middle feature extraction layer is connected to a middle classifier 4022, and the deep feature extraction layer is connected to a deep classifier 4023.

[0065] When processing the license plate recognition model, it proceeds from shallow to deep. First, it is processed by the shallow feature extraction layer 4011 and the shallow classifier 4021. The shallow feature extraction layer 4011 extracts the shallow features of the license plate image, and the shallow classifier 4021 obtains the shallow prediction result based on the shallow license plate features.

[0066] If the confidence level of the shallow prediction result of the shallow classifier is greater than the threshold, the shallow prediction result is used as the license plate recognition result. If the confidence level of the shallow prediction result of the shallow classifier is not greater than the threshold, the shallow features extracted by the shallow feature extraction layer are input into the middle feature extraction layer, which extracts the middle license plate features. The middle classifier then obtains the middle prediction result based on the middle license plate features.

[0067] The process begins with a shallow network in the model. When the confidence level of the shallow classifier's prediction is low, the shallow license plate features are further input into a mid-level feature extraction layer, where they are processed by a deeper mid-level feature extraction layer and a mid-level classifier. The shallow network can identify license plates that are relatively easy to recognize, while the mid-level feature extraction layer performs further identification for license plate images that the shallow network cannot recognize.

[0068] If the confidence level of the intermediate prediction result is greater than the threshold, then the intermediate prediction result is used as the license plate recognition result. If the confidence level of the intermediate prediction result is not greater than the threshold, then the intermediate features extracted by the intermediate feature extraction layer are input into the deep feature extraction layer, which extracts deep license plate features. The deep classifier then obtains the license plate recognition result based on the deep license plate features.

[0069] The intermediate layer network can recognize license plates of moderate difficulty. For license plate images that the intermediate layer network cannot recognize, the deep feature extraction layer can further perform recognition.

[0070] In this embodiment, by connecting classifiers to feature extraction layers at different depths, samples of varying difficulty are output at different stages, thereby improving the overall efficiency of license plate recognition.

[0071] The shallow classifier 4021 is connected to the shallow feature extraction layer 4011 via a shallow transition layer 4031, and the mid-level classifier 4022 is connected to the mid-level feature extraction layer 4012 via a mid-level transition layer. The shallow transition layer comprises a fully connected layer and an activation function, and the mid-level transition layer comprises a fully connected layer and an activation function. The structures of the shallow classifier, the mid-level classifier, and the deep classifier are identical.

[0072] Generally speaking, for transition layers, increasing the number of fully connected layers is beneficial to improving classification accuracy, but the inference throughput of the backbone network is relatively large. Considering the trade-off between additional computational cost and classification accuracy, the above-mentioned lightweight structure is designed.

[0073] To ensure the accuracy of the license plate recognition model, the training process of the license plate recognition model is as follows: Figure 5 As shown, it includes the following steps:

[0074] S502, Obtain the backbone network of the pre-trained license plate recognition model. The backbone network includes a deep feature extraction layer and a deep classifier.

[0075] In this embodiment, the backbone network is pre-trained. That is, the license plate recognition model in this embodiment includes a two-stage training process. The first stage is the pre-training of the backbone network. In one approach, the backbone network can use a LeViT pre-trained model, that is, utilizing an existing pre-trained model as the final backbone network of the license plate recognition model.

[0076] Alternatively, for the license plate recognition model to be trained, the parameters of the stage classifier can be fixed, and the backbone network can be trained.

[0077] S504 fixes the parameters of the backbone network, extracts license plate features from samples in the training sample set through the feature extraction layers of the license plate recognition model to be trained, and the classifiers corresponding to the feature extraction layers make predictions based on the license plate features, thus obtaining the prediction results output by classifiers of different depths.

[0078] The second stage of training for the license plate recognition model involves fixing the parameters of the backbone network and training the feature classifier. The structure of the stage classifier is the same as that of the deep classifier. The shallow transition layer consists of a fully connected layer and an activation function layer, with the input and output dimensions of the fully connected layer set to the same dimensions as the shallow and deep feature extraction layers of the model, respectively. The intermediate feature transition layer also consists of a fully connected layer and an activation function layer, with the input and output dimensions of the fully connected layer set to the same dimensions as the intermediate and deep feature extraction layers of the model, thus ensuring consistency in the dimensions of the access points connecting to feature extraction layers of different depths.

[0079] In the second stage of training, the parameters of the backbone network are fixed, and samples from the training sample set are input into the license plate recognition model to be trained. Starting from shallow to deep layers, each feature extraction layer extracts license plate features, and the classifier corresponding to that depth obtains a prediction result based on these features. Simultaneously, the license plate features from the shallower feature extraction layers are also input into the next deeper feature extraction layer, where they are processed by the next deeper feature extraction layer and its corresponding classifier. Thus, each depth's classifier outputs a prediction result for the license plate image.

[0080] S506. Based on the prediction results output by classifiers of different depths, calculate the training loss of the stage classifier, and train the parameters of the stage classifier of the license plate recognition model based on the training loss.

[0081] In the second training phase, the backbone network is fixed, and the parameters of the stage classifiers and transition classifiers of the license plate recognition model are trained based on the training loss. The training loss includes the training stages of each stage classifier, such as the loss of the shallow classifier and the loss of the mid-level classifier.

[0082] Specifically, based on the prediction results output by classifiers of different depths, the loss of each stage of the classifier is calculated. This loss can include cross-entropy loss. Cross-entropy can be used as a loss function in neural networks (machine learning), where p represents the distribution of the true labels and q represents the distribution of the predicted labels of the trained model. The cross-entropy loss function measures the similarity between p and q. Another advantage of using cross-entropy as a loss function is that using the sigmoid function during gradient descent avoids the problem of reduced learning rate associated with the mean squared error loss function, because the learning rate can be controlled by the output error.

[0083] S508, when the training reaches the termination condition, obtains the trained license plate recognition model based on the parameters of the backbone network and the stage classifier.

[0084] The training termination condition includes reaching the maximum number of training iterations. When the training reaches the termination condition, the trained license plate recognition model is obtained based on the parameters of the backbone network and the stage classifier.

[0085] In this embodiment, the license plate recognition model is trained using a two-stage training method: the first stage trains the main model, and the second stage trains the stage transitioner and stage classifier, which can improve training efficiency.

[0086] In other embodiments, a third stage of training may also be included. Specifically, a two-stage training method is employed. In the second stage of training, the backbone network remains unchanged, resulting in limited features that it can learn at different stages. After the second stage of training, the entire license plate extraction model can be trained again using a small number of samples for fine-tuning to improve the model's recognition accuracy.

[0087] In one embodiment, the training loss includes the cross-entropy loss of the labels of all stage classifiers, the distillation loss, and the entropy constraint loss; wherein the distillation loss is used to align the prediction results of the stage classifiers with the prediction results of the deep classifiers.

[0088] Specifically, the second phase of training process is as follows: Figure 6 As shown, the training loss includes the cross-entropy loss of the labels of all stage classifiers, the distillation loss, and the entropy constraint loss. Among them, the distillation loss aligns the prediction results of the stage classifiers with the prediction results of the deep classifiers, thereby making the features output by the intermediate feature extraction layers closer to the features input to the deep classifiers, and improving the prediction accuracy of the stage classifiers.

[0089] For label cross-entropy loss, cross-entropy can be used as a loss function in neural networks (machine learning). p represents the distribution of the true labels, and q is the distribution of the predicted labels from the trained model. The cross-entropy loss function measures the similarity between p and q. Another advantage of using cross-entropy as a loss function is that it avoids the problem of reduced learning rate caused by mean squared error loss functions during gradient descent, as the learning rate can be controlled by the output error. Specifically, for each stage classifier, which includes two stages (e.g., a shallow classifier corresponding to the shallow feature extraction layer and a middle classifier corresponding to the middle feature extraction layer), the features obtained from the stage feature extraction layer... The prediction results obtained by the stage classifier Here, 'i' represents the index number at different depths, with index numbers 1 and 2 indicating intermediate stages, and '3' representing the deep classifier in the backbone network. For each stage classifier, cross-entropy classification loss is used to compare its predictions with the actual image results. Alignment is performed, and cross-entropy loss is calculated, including the cross-entropy loss of shallow classifiers and the cross-entropy loss of mid-level classifiers. ,in , This represents the number of samples.

[0090] To narrow the gap between features learned at different stages and features classified by deep classifiers, and to improve the prediction accuracy of shallow and mid-level classifiers, self-distillation can be performed on the prediction results of the pre-trained backbone network (i.e., the prediction results of deep classifiers) for different stage classes. Specifically, the prediction results of the stage classifiers during training are compared with the prediction results of the deep classifiers, and the distillation loss is determined based on the differences between the deep classifier prediction results and the prediction results of each stage classifier.

[0091] Specifically, the features obtained from the stage feature extraction layer Prediction results obtained through the stage classifier Compared with the prediction results of deep classifiers For comparison, distillation losses were calculated using hard distillation, including distillation losses from the shallow classifier and the middle classifier. In one implementation, a soft distillation method was also attempted, comparing the consistency between the distribution of predictions obtained by the stage classifier and the original distribution of predictions, and evaluating this consistency using KL divergence. Training revealed that hard distillation makes it easier for shallow and deep layers to learn to conform to a consistent classification.

[0092] The classifier determines whether to output samples based on the actual prediction distribution at each stage. When the prediction distribution is steep (i.e., the score of the correct label is much larger than the score of the incorrect label) and the prediction confidence is high, the prediction at that stage is considered reliable. Therefore, it is proposed to use entropy to constrain the entropy of the prediction distribution of the samples output from the current layer to be as small as possible, thereby increasing the prediction accuracy. The goal is to maximize the discriminative power of the classifier, ensuring the correct label is distinguishable from others. In other words, entropy constraints mean that the entropy of each stage of the classifier is inversely proportional to the prediction accuracy; the lower the entropy, the higher the prediction accuracy. This embodiment proposes a calculation... Entropy is used as another training loss for the model, namely entropy loss, which includes the entropy loss of shallow classifiers and the entropy loss of mid-level classifiers. ,in .

[0093] The final training loss of the model is Where λ currently takes the value of 0.1. In summary, the training of the license plate recognition model includes two stages: the first stage fixes the parameters of the transitioner and stage classifier, and trains the backbone network; the second stage fixes all parameters of the backbone network and trains the model based on the aforementioned training loss. The parameters of the training phase transitioner and phase classifier are adjusted to ensure that the inference overhead of the shallow and medium-level classifiers is small, while the predicted distribution is as close as possible to that of the deep layers, thus ensuring a certain level of prediction accuracy.

[0094] In this application, based on the LeViT network framework, classifiers at different model stages are incorporated to select samples based on their difficulty, thus achieving different stage outputs for different samples. Furthermore, self-distillation is added to further supervise the classifiers at different stages, making the features output at each stage closer to the features input to the final classifier. In addition, entropy constraints are added to the outputs of the classifiers at different stages to promote more distinct sample features and drive the samples towards the outputs of the classifiers at previous stages. During the inference stage, the outputs of the classifiers at different stages are calculated and compared with a threshold, thereby addressing the issue of different stage outputs for samples of varying difficulty.

[0095] The license plate recognition model in this application, once trained, can directly classify images. For example... Figure 7 As shown, the model input is of size [size missing]. For RGB images, the algorithm framework analyzes the classifier outputs at different stages to determine the difficulty of image classification, and then decides which stage the image should be output from based on the judgment result. In short, easily classifiable samples are output at shallower levels of the model, while difficult samples are output at higher levels, thus reducing the model's inference time. This technique can be widely applied to various classification tasks, such as vehicle model classification, pedestrian ID classification, and license plate attribute classification, significantly improving the performance of massive image classification and search. Figure 7 The vehicle license plate attribute classification task requires distinguishing between clear license plates, occluded license plates, blurred license plates, and non-license plate images. Most of the massive license plate data consists of clear license plates, non-license plates, and simple occluded license plates. These data can all be judged by the shallow classifier. Only a small number of blurry and tilted view samples, which are more difficult to distinguish, are output by the middle and high layers, thus greatly reducing the inference cost.

[0096] After running the license plate recognition model for a period of time, the throughput of the model at different stages is shown in Table 1.

[0097] Table 1: Throughput of the current method at different stages

[0098]

[0099] Table 2 shows a comparison of the accuracy of the two-stage classifiers with that of the deep classifier after training.

[0100] Table 2: Classification accuracy of classifiers at different stages

[0101]

[0102] In the derivation phase, the prediction results obtained using the stage classifier are utilized. Maximum confidence The difficulty level of the samples is assessed. When a sample inference reaches the shallow layer, the prediction results of the shallow layer classifier are used as the basis for the inference. Maximum confidence To determine whether the sample can be output in a shallow layer, i.e., when When the model considers that the sample can be output from the shallow layer, if it does not meet the conditions, it continues to the middle layer to judge whether it can be output. If it still does not meet the conditions in the middle layer, it outputs the result in the deepest layer. This inference framework can use a simple threshold setting to make output decisions because entropy constraints are used in the training process to ensure that the prediction distribution remains steep while the prediction is accurate. When θ is 0.8, the accuracy of the output of the model at different stages is shown in Table 3. In the results of 50,000 test images in ImageNet[5], the first stage outputs 10,417 samples, the second stage outputs 10,056 samples, and the last stage outputs 29,527 samples. The average throughput of the model inference is 6,711.712 samples / sec, compared with the throughput of the original model of 6,123.46 samples / sec, the inference speed is improved by 9%, while the average top-1 classification accuracy is reduced by about 0.9%.

[0103] Table 3: Accuracy of Output Samples at Different Stages and Classification Accuracy of the Final Model

[0104]

[0105] For massive image classification problems, this method can significantly reduce unnecessary computational overhead. By effectively dividing samples into easy and difficult ones, it allows samples to be output from different stages of the model, thereby improving the model's inference speed without excessively sacrificing inference accuracy.

[0106] It should be understood that, although Figure 2 , Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , Figure 5 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0107] In one embodiment, such as Figure 8 As shown, a license plate recognition device is provided. This device can employ software modules, hardware modules, or a combination of both as part of a computer device. Specifically, the device includes: an image acquisition module, an input module, a prediction module, and an output module, wherein:

[0108] Image acquisition module 802 is used to acquire license plate images;

[0109] The input module 804 is used to input the license plate image into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to the non-last feature extraction layer of the backbone network.

[0110] The prediction module 806 is used to perform feature extraction and prediction processing according to the depth of the feature extraction layer of the license plate recognition model from shallow to deep. The feature extraction layer and the corresponding classifier perform feature extraction and prediction processing respectively. The feature extraction layer extracts the license plate features of the license plate image, and the classifier corresponding to the feature extraction layer obtains the prediction result based on the license plate features.

[0111] The output module 808 is used to take the prediction result of the stage classifier as the license plate recognition result when the prediction result of the stage classifier meets the requirements, and to input the license plate features of the stage feature extraction layer of that depth into the feature extraction layer of the next depth for processing when the prediction result of the stage classifier does not meet the requirements. The feature extraction layer of the next depth and the corresponding classifier of the same depth continue to perform feature extraction and prediction processing until a license plate recognition result is obtained in one of the stage classifiers, or when a prediction result is obtained in the deep classifier, the prediction result of the deep classifier is taken as the license plate recognition result.

[0112] The aforementioned license plate recognition device constructs a license plate recognition model, which enables simple images to be predicted in a shallow network and complex images in a deep network. This allows the model to predict license plate images at different stages based on their complexity. For simple license plate images, the prediction result can be obtained at the shallow layer of the model without going through the entire model, thus improving the overall efficiency of license plate recognition.

[0113] In another embodiment, the stage feature extraction layer includes a shallow feature extraction layer and a middle feature extraction layer, and the stage classifier includes a shallow classifier connected to the shallow feature extraction layer and a middle classifier connected to the middle feature extraction layer.

[0114] The output module is configured to: if the confidence level of the shallow prediction result of the shallow classifier is greater than a threshold, then use the shallow prediction result as the license plate recognition result; if the confidence level of the shallow prediction result of the shallow classifier is not greater than the threshold, then input the shallow features extracted by the shallow feature extraction layer into the middle feature extraction layer, which extracts the middle license plate features, and the middle classifier obtains the middle prediction result based on the middle license plate features; if the confidence level of the middle prediction result is greater than the threshold, then use the middle prediction result as the license plate recognition result; if the confidence level of the middle prediction result is not greater than the threshold, then input the middle license plate features extracted by the middle feature extraction layer into the deep feature extraction layer, which extracts the deep license plate features, and the deep classifier obtains the license plate recognition result based on the deep license plate features, wherein the deep feature extraction layer is the last feature extraction layer.

[0115] In another embodiment, a shallow transition layer is connected between the shallow classifier and the shallow feature extraction layer, and a middle transition layer is connected between the middle classifier and the middle feature extraction layer; the shallow feature extraction layer includes a fully connected layer and an activation function, and the middle feature extraction layer includes a fully connected layer and an activation function; the structures of the shallow classifier, the middle classifier, and the deep classifier are the same.

[0116] In another embodiment, the license plate recognition device further includes:

[0117] The backbone network acquisition module is used to acquire the backbone network of the pre-trained license plate recognition model;

[0118] The first-stage training module is used to fix the parameters of the backbone network, extract license plate features from the samples in the training sample set through each feature extraction layer of the license plate recognition model to be trained, and make predictions based on the license plate features by the classifiers corresponding to the feature extraction layers, thereby obtaining prediction results output by classifiers of different depths.

[0119] The second-stage training module is used to calculate the training loss of the stage classifier based on the prediction results output by classifiers of different depths, and to train the parameters of the stage feature extraction layer and the stage classifier of the license plate recognition model based on the training loss.

[0120] The model acquisition module is used to obtain the trained license plate recognition model based on the parameters of the backbone network and the stage classifier when the training reaches the termination condition.

[0121] The training loss includes the cross-entropy loss of the labels of all stage classifiers, the distillation loss, and the entropy constraint loss; wherein, the distillation loss is used to align the prediction results of the stage classifiers with the prediction results of the deep classifiers.

[0122] In another embodiment, determining the distillation loss includes:

[0123] The prediction results of the stage classifiers during training are compared with the prediction results of the deep classifiers. The distillation loss is determined based on the difference between the prediction results of the deep classifiers and the prediction results of each stage classifier.

[0124] In another embodiment, the method for determining the entropy constraint loss includes:

[0125] Based on the prediction results of the classifiers at each stage, the entropy of each classifier is calculated to obtain the entropy-constrained loss. The entropy constraint includes the fact that the entropy of each classifier is inversely proportional to the prediction accuracy.

[0126] Specific limitations regarding license plate recognition devices can be found in the limitations of license plate recognition methods described above, and will not be repeated here. Each module in the aforementioned license plate recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0127] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores license plate recognition model data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a license plate recognition model method.

[0128] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0129] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0130] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0131] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A license plate recognition method, characterized in that, The method includes: Obtain license plate image; The license plate image is input into a pre-trained license plate recognition model. The model includes a backbone network comprising multiple feature extraction layers and a deep classifier. The feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The model also includes a stage classifier connected to a feature extraction layer other than the last layer of the backbone network. The stage classifier includes a shallow classifier connected to a shallow feature extraction layer and a mid-level classifier connected to a mid-level feature extraction layer. A shallow transition layer connects the shallow classifier to the shallow feature extraction layer, and a mid-level transition layer connects the mid-level classifier to the mid-level feature extraction layer. The shallow transition layer includes a fully connected layer and an activation function, and the mid-level transition layer includes a fully connected layer and an activation function. The structures of the shallow classifier, the mid-level classifier, and the deep classifier are identical. According to the order of depth of the feature extraction layers of the license plate recognition model from shallow to deep, the feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain prediction results based on the license plate features. If the confidence level of the shallow prediction result of the shallow classifier is greater than the threshold, then the shallow prediction result is used as the license plate recognition result; if the confidence level of the shallow prediction result of the shallow classifier is not greater than the threshold, then the shallow features extracted by the shallow feature extraction layer are input into the middle feature extraction layer, and the middle feature extraction layer extracts the middle license plate features. The middle classifier obtains the middle prediction result based on the middle license plate features; if the confidence level of the middle prediction result is greater than the threshold, then the middle prediction result is used as the license plate recognition result; if the confidence level of the middle prediction result is not greater than the threshold, then the middle license plate features extracted by the middle feature extraction layer are input into the deep feature extraction layer, and the deep feature extraction layer extracts the deep license plate features. The deep classifier obtains the license plate recognition result based on the deep license plate features. The deep feature extraction layer is the last feature extraction layer. The training process of the license plate recognition model includes: In the first stage, the parameters of the transitioner and stage classifier are fixed, and the backbone network is trained. In the second stage, the parameters of the backbone network are fixed, based on the training loss. The parameters of the training phase transitioner and the phase classifier; where, The cross-entropy loss of the labels for all stages of the classifier; The distillation loss is used to align the prediction results of the stage classifier with the prediction results of the deep classifier. The distillation loss is determined by comparing the prediction results of the stage classifier with the prediction results of the deep classifier during training, based on the differences between the prediction results of the deep classifier and the prediction results of each stage classifier. As a preset value, The entropy-constrained loss is calculated based on the prediction results of the classifiers at each stage. The entropy constraint includes the fact that the entropy of each classifier is inversely proportional to the prediction accuracy. In the third stage, the entire license plate recognition model is trained using samples and then fine-tuned.

2. The method according to claim 1, characterized in that, In the first stage, the parameters of the transitioner and stage classifier are fixed, and the backbone network is trained, including: Obtain the backbone network of the pre-trained license plate recognition model; With the parameters of the backbone network fixed, the samples in the training sample set are processed by extracting license plate features through each feature extraction layer of the license plate recognition model to be trained. The classifier corresponding to the feature extraction layer makes predictions based on the license plate features, and the prediction results output by classifiers of different depths are obtained. The method further includes: When the training reaches the termination condition, the trained license plate recognition model is obtained based on the parameters of the backbone network and the stage classifier.

3. A license plate recognition device, characterized in that, The device includes: The image acquisition module is used to acquire license plate images; An input module is used to input the license plate image into a pre-trained license plate recognition model. The license plate recognition model includes a backbone network, which includes multiple feature extraction layers and a deep classifier. The multiple feature extraction layers are connected sequentially, and the deep classifier is connected after the last feature extraction layer. The license plate recognition model also includes a stage classifier connected to a feature extraction layer other than the last layer of the backbone network. The stage classifier includes a shallow classifier connected to a shallow feature extraction layer and a middle classifier connected to a middle feature extraction layer. A shallow transition layer connects the shallow classifier to the shallow feature extraction layer, and a middle transition layer connects the middle classifier to the middle feature extraction layer. The shallow transition layer includes a fully connected layer and an activation function, and the middle transition layer includes a fully connected layer and an activation function. The structures of the shallow classifier, the middle classifier, and the deep classifier are identical. The prediction module is used to perform feature extraction and prediction processing according to the depth of the feature extraction layers of the license plate recognition model from shallow to deep. The feature extraction layers and the corresponding classifiers perform feature extraction and prediction processing respectively. The feature extraction layers extract the license plate features of the license plate image, and the classifiers corresponding to the feature extraction layers obtain the prediction results based on the license plate features. The output module is configured to: if the confidence level of the shallow prediction result of the shallow classifier is greater than a threshold, then use the shallow prediction result as the license plate recognition result; if the confidence level of the shallow prediction result of the shallow classifier is not greater than the threshold, then input the shallow features extracted by the shallow feature extraction layer into the middle feature extraction layer, which extracts the middle license plate features, and the middle classifier obtains the middle prediction result based on the middle license plate features; if the confidence level of the middle prediction result is greater than the threshold, then use the middle prediction result as the license plate recognition result; if the confidence level of the middle prediction result is not greater than the threshold, then input the middle license plate features extracted by the middle feature extraction layer into the deep feature extraction layer, which extracts the deep license plate features, and the deep classifier obtains the license plate recognition result based on the deep license plate features. The deep feature extraction layer is the last feature extraction layer. The first-stage training module is used for the first stage, fixing the parameters of the transitioner and stage classifier, and training the backbone network. The second-stage training module is used in the second stage to fix the parameters of the backbone network and apply the training loss. The parameters of the training phase transitioner and the phase classifier; where, The cross-entropy loss of the labels for all stages of the classifier; The distillation loss is used to align the prediction results of the stage classifier with the prediction results of the deep classifier. The distillation loss is determined by comparing the prediction results of the stage classifier with the prediction results of the deep classifier during training, based on the differences between the prediction results of the deep classifier and the prediction results of each stage classifier. As a preset value, The entropy-constrained loss is calculated based on the prediction results of the classifiers at each stage. The entropy constraint includes the fact that the entropy of each classifier is inversely proportional to the prediction accuracy. This module is used in the third stage to train the entire license plate recognition model using samples and then fine-tunes it.

4. The license plate recognition device according to claim 3, characterized in that, The device further includes: The backbone network acquisition module is used to acquire the backbone network of the pre-trained license plate recognition model; The first-stage training module is also used to fix the parameters of the backbone network, extract license plate features from the samples in the training sample set through each feature extraction layer of the license plate recognition model to be trained, and make predictions based on the license plate features by the classifiers corresponding to the feature extraction layers to obtain prediction results output by classifiers of different depths. The model acquisition module is used to obtain the trained license plate recognition model based on the parameters of the backbone network and the stage classifier when the training reaches the termination condition.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of claim 1 or 2.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of claim 1 or 2.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of claim 1 or 2.

Images