License plate recognition method, device, equipment and storage medium

Abstract

This application discloses a license plate recognition method, apparatus, device, and storage medium. The method comprises: obtaining license plate image samples as a training set; constructing a license plate recognition model based on a multi-task network structure; using the multi-task network structure to simultaneously extract various license plate information, including the number, color, and number of layers, through a deep CNN network; training the license plate recognition model using the training set; and using the trained license plate recognition model to recognize the license plate image to be tested. This multi-task network structure enables simultaneous extraction of all license plate information within the same network, fully utilizing computing resources with minimal computational effort. The constructed model is simple in structure, and all license plate recognition objectives can be achieved in one step without any additional processing. This addresses the low robustness issue of traditional methods and achieves high-precision license plate recognition.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to a license plate recognition method, device, equipment and storage medium. Background Art

[0002] Automatic license plate recognition is a very important issue in the field of intelligent transportation. Smart parking fee collection systems, vehicle search systems, traffic checkpoints’ illegal behavior capture, mobile illegal inspection equipment, etc. all require automatic license plate recognition to confirm the identity of the vehicle.

[0003] Currently, there are two main approaches to license plate recognition. The first is based on traditional machine learning. It first extracts each character through image segmentation, then classifies each character using template matching or SVM (support vector machine) to identify the license plate number. Simultaneously, the position of the cutout characters is used to determine the number of layers in the license plate, and traditional color analysis is used to extract the color. This approach suffers from poor robustness and is generally used at parking entrances. The second approach, using deep learning, decomposes the license plate recognition task into object detection and classification. It first detects each character in the license plate, then classifies each character to identify the license plate number. The number of layers is determined based on the position of the detection box, and color analysis or a classification model is used to identify the license plate color. This approach can handle complex environments, but splitting the license plate recognition task into two stages increases system complexity, complicates deployment, and significantly increases processing time, making real-time performance difficult to guarantee.

[0004] Therefore, how to achieve fast and low-resource license plate recognition is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a license plate recognition method, device, equipment and storage medium that can fully utilize computing resources to ensure the accuracy and robustness of license plate recognition. The specific scheme is as follows:

[0006] A license plate recognition method, comprising:

[0007] Get license plate image samples as training set;

[0008] Constructing a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including the number, color, and number of layers, through a deep CNN network;

[0009] Using the training set to train the license plate recognition model;

[0010] The license plate recognition model that has been trained is used to recognize the license plate image to be tested.

[0011] Preferably, in the above-mentioned license plate recognition method provided by an embodiment of the present invention, different information levels are used to process different tasks in the multi-task network structure, and the tasks include license plate number recognition tasks, color classification tasks and layer classification tasks.

[0012] Preferably, in the above-mentioned license plate recognition method provided by an embodiment of the present invention, the color classification task is introduced into the shallow features of the multi-task network structure;

[0013] Introducing the layer classification task into the middle-layer features of the multi-task network structure;

[0014] The license plate number recognition task is introduced into the final output task of the multi-task network structure.

[0015] Preferably, in the above-mentioned license plate recognition method provided in an embodiment of the present invention, two pooling layers are connected after the last convolutional layer of the multi-task network structure, so that the license plate number recognition task is changed into two branches, the first branch is used to process the upper license plate number recognition of the double-layer license plate, and the second branch is used to process the lower license plate number recognition of the double-layer license plate or to recognize the single-layer license plate number recognition.

[0016] Preferably, in the above-mentioned license plate recognition method provided by an embodiment of the present invention, training the license plate recognition model includes:

[0017] For a double-layer license plate, the first branch is labeled with the characters of the upper layer license plate, and the second branch is labeled with the characters of the lower layer license plate;

[0018] For a single-layer license plate, the first branch is labeled with an empty string, and the second branch is labeled with the single-layer license plate characters.

[0019] Preferably, in the above-mentioned license plate recognition method provided by the embodiment of the present invention, training the license plate recognition model further includes:

[0020] In the color classification task, training is performed using softmax output plus crossentrophy loss;

[0021] In the layer classification task, training is performed using softmax output plus crossentrophy loss;

[0022] In the license plate number recognition task, CTC connection temporal classification loss is used for training.

[0023] Preferably, in the above-mentioned license plate recognition method provided by the embodiment of the present invention, using the trained license plate recognition model to recognize the license plate image to be tested includes:

[0024] The softmax output of the corresponding layer of the trained license plate recognition model is used to identify the color and number of layers of the license plate image to be tested, and the output of the last convolutional layer is directly decoded to obtain the license plate number information of the license plate image to be tested.

[0025] An embodiment of the present invention further provides a license plate recognition device, comprising:

[0026] The sample acquisition module is used to obtain license plate image samples as training sets;

[0027] A model building module for building a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including the number, color, and number of layers, through a deep CNN network;

[0028] A model training module, configured to train the license plate recognition model using the training set;

[0029] The model inference module is used to use the trained license plate recognition model to identify the license plate image to be tested.

[0030] An embodiment of the present invention further provides a license plate recognition device, comprising a processor and a memory, wherein the processor implements the above-mentioned license plate recognition method provided in the embodiment of the present invention when executing a computer program stored in the memory.

[0031] An embodiment of the present invention further provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the license plate recognition method as provided in the embodiment of the present invention is implemented.

[0032] It can be seen from the above technical solution that the license plate recognition method provided by the present invention includes: obtaining license plate image samples as a training set; constructing a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including the number, color and number of layers, through a deep CNN network; using the training set to train the license plate recognition model; and using the trained license plate recognition model to recognize the license plate image to be tested.

[0033] The present invention takes advantage of the advantages of neural networks and can extract all license plate information simultaneously in the same network through a multi-task network structure, making full use of computing resources, with very small computational complexity. The constructed model structure is simple and all license plate recognition objectives can be obtained in one step without any additional processing, thus solving the problem of low robustness of traditional methods and achieving high-precision license plate recognition.

[0034] In addition, the present invention also provides corresponding devices, equipment and computer-readable storage media for the license plate recognition method, further making the above method more practical, and the device, equipment and computer-readable storage medium have corresponding advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0036] Figure 1 A flowchart of a license plate recognition method provided by an embodiment of the present invention;

[0037] Figure 2 One of the structural diagrams of the multi-task network structure provided by an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the principle of the spatial attention mechanism provided by an embodiment of the present invention;

[0039] Figure 4 The second structural diagram of the multi-task network structure provided by an embodiment of the present invention;

[0040] Figure 5 This is a schematic structural diagram of a license plate recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] The present invention provides a license plate recognition method, such as Figure 1 As shown, the following steps are included:

[0043] S101, obtaining license plate image samples as a training set;

[0044] In practical applications, license plate image samples can include license plate images captured by cameras, existing high-definition license plate images, etc. License plate recognition generally requires automatic extraction of the license plate number and identification of the license plate type. The license plate type includes the number of layers and the color of the license plate. The number of layers includes double-layer license plates (such as the first layer is Xiang J and the second layer is S2668) and single-layer license plates (such as a single layer: Xiang J S2668). License plate colors include yellow, blue, green, white, and yellow-green gradient license plates.

[0045] S102. Construct a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including the number, color, and number of layers, through a deep CNN network;

[0046] It should be noted that the model constructed by the present invention has a simple structure and can simultaneously realize license plate number recognition, license plate layer recognition, and license plate color recognition through a compact multi-task network structure with very small computational complexity.

[0047] S103, using the training set to train the license plate recognition model;

[0048] S104: Use the trained license plate recognition model to recognize the license plate image to be tested.

[0049] In the above-mentioned license plate recognition method provided in the embodiment of the present invention, all license plate information can be extracted simultaneously in the same network through a multi-task network structure, which makes full use of computing resources, has a small amount of calculation, and the constructed model structure is simple. All the goals of license plate recognition can be obtained in one step without any additional processing, which solves the problem of low robustness of traditional methods and achieves high-precision license plate recognition.

[0050] In specific implementation, in the above-mentioned license plate recognition method provided in an embodiment of the present invention, different information levels are used to process different tasks in a multi-task network structure, and the tasks may include license plate number recognition tasks, color classification tasks and layer classification tasks.

[0051] It is understandable that different deep learning tasks require different features to be extracted. For example, image classification tasks require extracting global information of the entire image, while semantic segmentation tasks require extracting semantic information of each pixel. Therefore, most current deep networks are designed to implement a fixed type of task. For example, an animal classification network can only implement image classification tasks and output the type of animal in the image. A pedestrian target detection network can only output the location information of each person in the image. However, the information of the neural network is transmitted layer by layer, and the features extracted at different layers of the network are gradually deepened. In the shallower layers, it may be simple extraction of shallow features such as edge information and color information, while in the deeper layers, deeper semantic information is extracted. Based on this feature, the present invention can implement different tasks at different layers, making it possible to solve the three elements in license plate recognition in the same network.

[0052] By analyzing the prior information transmitted by the network, the present invention meticulously designs the access locations of different tasks in the network. For the three tasks in license plate recognition, each task requires different information. For the color of the license plate, only relatively rough features are needed for judgment. For the number of layers of the license plate, more precise location information is needed for deduction, and color information is not required. For the license plate number recognition task, more abstract features are required to represent it. Therefore, in the specific implementation, in the above-mentioned license plate recognition method provided by the embodiment of the present invention, the color classification task is introduced in the shallow features of the deep network, the layer classification task is introduced in the middle-layer features, and the license plate number recognition task is introduced in the final output task.

[0053] In addition, if Figure 2 As shown in the figure, a recurrent neural network module (RNN) can be connected after the convolutional layer. RNN can learn the association between different characters and alleviate the problem of poor recognition effect of the two-layer network to a certain extent.

[0054] In specific implementation, in the above-mentioned license plate recognition method provided by an embodiment of the present invention, in the process of executing step S103 to train the license plate recognition model, it can also include: using softmax output plus crossentrophy loss for training in the color classification task; using softmax output plus crossentrophy loss for training in the layer classification task; using CTC connection temporal classification loss (CTCLOSS) for training in the license plate number recognition task to solve the license plate number recognition problem.

[0055] It should be noted that since CTCLOSS (Connected Temporal Classification) is generally used to deal with sequence recognition problems in a single dimension, directly using CTCLOSS (Connected Temporal Classification) has a natural disadvantage in dealing with the recognition of double-layer license plates. Figure 2 The use of a CNN+RNN structure in the neural network can alleviate this problem to a certain extent. The main purpose of RNN is to learn the association information between elements in a sequence. When there are a large number of double-layer license plates as a data set, RNN can also learn this kind of positional association information. However, there are usually relatively few double-layer license plates in the data set, so the improvement produced by RNN is also very limited. On the other hand, in addition to the position information of the double-layer license plate, there is no strong correlation between the characters on the license plate, unlike ordinary text recognition tasks, which have a grammatical context. Therefore, the present invention proposes a spatial attention mechanism to solve the problem of poor recognition of double-layer license plates, and at the same time removes the RNN module, thereby reducing the amount of calculation while improving the recognition effect of double-layer license plates.

[0056] In specific implementation, in the above license plate recognition method provided by the embodiment of the present invention, if Figure 3 and Figure 4 As shown in the figure, two pooling layers are added after the last convolutional layer in the multi-task network structure, splitting the license plate recognition task into two branches. The first branch handles the upper license plate number recognition of a double-layer license plate, while the second branch handles the lower license plate number recognition of a double-layer license plate or the recognition of a single-layer license plate number. This removes the RNN module from the CNN+RNN network while still maintaining high accuracy. This ensures that the model can be quickly deployed on all embedded devices.

[0057] Furthermore, in specific implementation, in the above-mentioned license plate recognition method provided in an embodiment of the present invention, in the process of executing step S103 to train the license plate recognition model, it can specifically include: for a double-layer license plate (such as the first layer is Xiang J and the second layer is S2668), the first branch is labeled with the upper-layer license plate characters (such as two characters of Xiang J), and the second branch is labeled with the lower-layer license plate characters (such as five characters of S2668); for a single-layer license plate (such as only one layer: Xiang JS2668), the first branch is labeled with an empty string, and the second branch is labeled with the single-layer license plate characters (such as seven characters of Xiang JS2668).

[0058] By training in this way, different branches can focus on different positions of the license plate. The upper branch focuses on identifying the upper license plate, and the lower branch focuses on the lower license plate. This design fundamentally overcomes the problems brought about by training using CTCLOSS.

[0059] In practice, in the license plate recognition method provided in the embodiments of the present invention, step S104 utilizes the trained license plate recognition model to identify the license plate image to be tested. This may include: utilizing the softmax output of the corresponding layer of the trained license plate recognition model to identify the color and layer number of the license plate image to be tested, and directly decoding the output of the last convolutional layer to obtain the license plate number information of the license plate image to be tested. In the case of a two-layer license plate, the upper and lower layers are separately inferred and decoded, and the decoded data are then concatenated to obtain the final license plate output, thereby achieving single-network multi-tasking functionality.

[0060] It should be pointed out that the multi-task network design of the present invention can extract all license plate information in the same network, making full use of computing resources; processing different tasks at different information levels, making full use of prior information to allocate different semantic information can ensure that each task is more accurate; and the present invention proposes a space to solve the effect of double-layer license plate recognition, which can further improve the license plate recognition accuracy without adding additional operations.

[0061] Based on the same inventive concept, an embodiment of the present invention also provides a license plate recognition device. Since the principle of solving the problem of this device is similar to that of the aforementioned license plate recognition method, the implementation of this device can refer to the implementation of the license plate recognition method, and the repeated parts will not be repeated.

[0062] In specific implementation, the license plate recognition device provided by the embodiment of the present invention is as follows: Figure 5 As shown, specifically including:

[0063] The sample acquisition module 11 is used to obtain license plate image samples as a training set;

[0064] The model building module 12 is used to build a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including the number, color, and number of layers, through a deep CNN network;

[0065] A model training module 13 is used to train a license plate recognition model using a training set;

[0066] The model inference module 14 is used to use the trained license plate recognition model to recognize the license plate image to be tested.

[0067] In the above-mentioned license plate recognition device provided in the embodiment of the present invention, the interaction of the above-mentioned four modules can fully utilize computing resources, the amount of calculation is very small, the model structure is simple, and all the goals of license plate recognition can be obtained in one step without any additional processing, thereby solving the problem of low robustness of traditional methods and achieving high-precision license plate recognition.

[0068] For more specific working processes of the above modules, please refer to the corresponding contents disclosed in the above embodiments, which will not be repeated here.

[0069] Correspondingly, an embodiment of the present invention further discloses a license plate recognition device, comprising a processor and a memory; wherein the license plate recognition method disclosed in the aforementioned embodiment is implemented when the processor executes a computer program stored in the memory.

[0070] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0071] Furthermore, the present invention also discloses a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the aforementioned license plate recognition method is implemented.

[0072] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0073] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar portions of the various embodiments will be sufficient. The devices, equipment, and storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the method descriptions.

[0074] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0075] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0076] In summary, an embodiment of the present invention provides a license plate recognition method, comprising: obtaining license plate image samples as a training set; constructing a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including the number, color, and number of layers, through a deep CNN network; using the training set to train the license plate recognition model; and using the trained license plate recognition model to recognize the license plate image to be tested. The present invention utilizes the advantages of neural networks and can simultaneously extract all license plate information in the same network through a multi-task network structure, making full use of computing resources, and the amount of calculation is very small. The constructed model structure is simple, and all license plate recognition targets can be obtained in one step without any additional processing, solving the problem of low robustness of traditional methods and achieving high-precision license plate recognition. In addition, the present invention also provides corresponding devices, equipment, and computer-readable storage media for the license plate recognition method, further making the above method more practical. The device, equipment, and computer-readable storage medium have corresponding advantages.

[0077] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0078] The above is a detailed introduction to the license plate recognition method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A license plate recognition method, characterized in that: include: Get license plate image samples as training set; Construct a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including number, color and number of layers, through a deep CNN network; different information levels are used in the multi-task network structure to process different tasks, including license plate number recognition task, color classification task and layer classification task; the color classification task is introduced into the shallow features of the multi-task network structure; the layer classification task is introduced into the middle-level features of the multi-task network structure; the license plate number recognition task is introduced into the last output task of the multi-task network structure; a recurrent neural network module is connected after the convolution layer of the multi-task network structure to learn the association relationship between different characters; two pooling layers are connected after the last convolution layer of the multi-task network structure, and the recurrent neural network module is removed at the same time, so that the license plate number recognition task becomes two branches, the first branch is used to process the upper license plate number recognition of a double-layer license plate, and the second branch is used to process the lower license plate number recognition of a double-layer license plate or recognize the single-layer license plate number recognition; Using the training set to train the license plate recognition model; The softmax output of the corresponding layer of the trained license plate recognition model is used to identify the color and number of layers of the license plate image to be tested, and the output of the last convolutional layer is directly decoded to obtain the license plate number information of the license plate image to be tested.

2. The license plate recognition method according to claim 1, characterized in that: Training the license plate recognition model includes: For a double-layer license plate, the first branch is labeled with the characters of the upper layer license plate, and the second branch is labeled with the characters of the lower layer license plate; For a single-layer license plate, the first branch is labeled with an empty string, and the second branch is labeled with the single-layer license plate characters.

3. The license plate recognition method according to claim 2, characterized in that: Training the license plate recognition model further includes: In the color classification task, training is performed using softmax output plus crossentrophy loss; In the layer classification task, training is performed using softmax output plus crossentrophy loss; In the license plate number recognition task, CTC connection temporal classification loss is used for training.

4. A license plate recognition device, characterized in that: include: The sample acquisition module is used to obtain license plate image samples as training sets; A model building module is used to build a license plate recognition model based on a multi-task network structure; the multi-task network structure is used to simultaneously extract various information from the license plate, including number, color and number of layers, through a deep CNN network; different information levels are used in the multi-task network structure to process different tasks, including license plate number recognition task, color classification task and layer classification task; the color classification task is introduced into the shallow features of the multi-task network structure; the layer classification task is introduced into the middle-level features of the multi-task network structure; the license plate number recognition task is introduced into the last output task of the multi-task network structure; a recurrent neural network module is connected after the convolution layer of the multi-task network structure to learn the association relationship between different characters; two pooling layers are connected after the last convolution layer of the multi-task network structure, and the recurrent neural network module is removed at the same time, so that the license plate number recognition task becomes two branches, the first branch is used to process the upper license plate number recognition of the double-layer license plate, and the second branch is used to process the lower license plate number recognition of the double-layer license plate or recognize the single-layer license plate number recognition; A model training module, configured to train the license plate recognition model using the training set; The model inference module is used to use the softmax output of the corresponding layer of the trained license plate recognition model to identify the color and number of layers of the license plate image to be tested, and directly decode the output of the last convolutional layer to obtain the license plate number information of the license plate image to be tested.

5. A license plate recognition device, characterized in that: The method comprises a processor and a memory, wherein the processor implements the license plate recognition method according to any one of claims 1 to 3 when executing the computer program stored in the memory.

6. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the license plate recognition method according to any one of claims 1 to 3 is implemented.

Images