License plate detection method, electronic device and computer readable storage medium

Abstract

The present application provides a license plate detection method, an electronic device and a computer-readable storage medium. The license plate detection method is implemented through a license plate detection model. The license plate detection model includes: a feature module, a spatial variation module and a recovery and recognition module. A spatial variation module (spatialtransform network, STN) is added to the license plate detection model. The STN can learn the parameters of the affine transformation, thereby restoring the image with low quality such as image distortion, rotation, uneven lighting and blur to a standard image, thereby improving the recognition accuracy. For the overexposure and blur caused by night, the license plate detection model of this embodiment can realize the restoration and calibration of the distorted image due to the addition of STN, which greatly improves the detection accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of license plate detection, and in particular to a license plate detection method, an electronic device and a computer-readable storage medium. Background Art

[0002] License plate detection and recognition are essential in the image field and have been widely used in many real-world scenarios, such as surveillance systems.

[0003] Existing license plate detection methods use convolutional neural networks. Because license plate detection and license plate recognition are strongly correlated, that is, accurate border prediction helps improve recognition accuracy, and accurate recognition results can be used to eliminate positive examples. The proposed model can be trained end-to-end without the need for additional heuristic rules (license plate color, character space), avoiding the intermediate character segmentation process. This detection method is end-to-end when making predictions, which has a great advantage in speed. It only needs to increase the input of character information during training. However, due to some extreme scenarios, the captured license plate images have image distortion, rotation, uneven lighting, blur and other low image quality. At this time, the existing detection methods will become extremely difficult to detect license plate images captured by the existing detection methods, and the recognition accuracy is low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a license plate detection method, an electronic device and a computer-readable storage medium to solve the problem that in some extreme scenarios, the captured license plate image has image distortion, rotation, uneven lighting and blur, etc., and the image quality is low. At this time, the existing detection method will become extremely difficult to perform license plate detection on the captured license plate image, and the recognition accuracy is low.

[0005] The embodiment of the present application provides a license plate detection method, which is applied to a license plate detection model, and the license plate detection model includes: a feature module, a spatial variation module and a recovery recognition module;

[0006] The license plate detection method includes:

[0007] The feature module extracts features from the license plate image to be identified, obtains a feature map and selects the license plate frame;

[0008] The spatial variation module corrects the license plate frame to obtain a corrected feature map; and

[0009] The restoration and recognition module dedistorts the license plate frame into a standard license plate frame based on the corrected feature map, and recognizes the license plate characters based on the standard license plate frame.

[0010] In the above technical solution, the license plate detection method is implemented through the license plate detection model, and a spatial transform network (STN) is added to the license plate detection model. The STN can learn the parameters of the affine transformation, thereby restoring the image with poor quality such as image distortion, rotation, uneven lighting and blur to a standard image, thereby improving the recognition accuracy. For the overexposure and blur caused by night, the license plate detection model of this embodiment can realize the restoration and calibration of the distorted image due to the addition of STN, which greatly improves the detection accuracy.

[0011] In some optional embodiments, wherein the feature module includes a backbone network of the SSD;

[0012] Extract features of the license plate image to be recognized, obtain the feature map and select the license plate frame, including:

[0013] The SSD backbone network uses a one-stage target detection algorithm to select the license plate frame.

[0014] In the above technical solution, the feature module is mainly composed of convolutional layers, which mainly plays the role of extracting features and selecting the license plate frame. The feature module of this embodiment adopts the SSD backbone network, which is a one-stage algorithm that can quickly select the license plate frame.

[0015] In some optional implementations, the spatial variation module includes a local network, a grid generator, and a sampler; correcting the license plate frame to obtain a corrected feature map includes:

[0016] The local network calculates the affine transformation parameters based on the license plate frame;

[0017] The grid generator constructs a sampling grid according to the affine transformation parameters;

[0018] The sampler obtains a corrected feature map based on the sampling grid and the feature map.

[0019] In the above technical solution, the local network is a network used to regress the transformation parameter θ. Its input is the license plate frame selected by the feature module frame, and the output θ is the output of a 6-dimensional (2x3) vector. These parameters are the parameters of the affine transformation. The local network consists of a convolutional layer and a regression layer. The grid generator constructs a sampling grid based on the predicted transformation parameters. It is the output obtained after a set of points in the input image are sampled and transformed. The sampler uses the sampling grid and the input feature map as input to generate output, and obtains the result after the feature map is transformed, that is, the corrected feature map.

[0020] In some optional embodiments, the recovery recognition module includes an instance normalization layer; before dedistorting the license plate frame to a standard license plate frame according to the corrected feature map and recognizing the license plate characters according to the standard license plate frame, it also includes:

[0021] The input data is normalized through the instance normalization layer.

[0022] In some optional implementations, the license plate detection model is trained according to a loss function, where the loss function is:

[0023]

[0024] Among them, L D is the detection loss, β and γ are weight parameters, and N is the size of a batch in training;

[0025] L rec (R, S) is the mean square error between the restored image R and the standard license plate S:

[0026]

[0027] γ cls (p,g) is the cross entropy:

[0028]

[0029] In the above technical solution, by detecting the loss L D , restore the loss L rec (R,S), and recognition loss γ cls By jointly optimizing (p,g), the license plate detection model can detect, recover and recognize the license plate in one forward inference pass. This overall loss function can be used to optimize the license plate detection model during training.

[0030] In some optional implementations, the detection loss is the sum of the confidence error and the classification error:

[0031] L D = l obj +l cls

[0032] Among them, l obj is the confidence error, l cls is the classification error.

[0033] In the above technical solution, detection loss refers to detection and positioning loss, including confidence error and classification error.

[0034] In some optional implementations, the confidence error:

[0035]

[0036] Among them, λ obj and λ noobj They represent the weights of detected objects and undetected objects, respectively. i is the predicted probability, The grid representing each feature map will be traversed, and each grid will produce three different prediction boxes.

[0037] In some optional implementations, the classification error:

[0038]

[0039] Among them, λ class is the weight of each category, The probability of belonging to this category is 1, and it is 0 when it is not this category; p i (c) is the predicted probability of this category.

[0040] An electronic device provided in an embodiment of the present application includes: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, any of the license plate detection methods described above is performed.

[0041] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned license plate detection methods is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of the steps of a license plate detection method provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a license plate detection model provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a space variation module provided in an embodiment of the present application;

[0046] Figure 4 A possible structure of an electronic device provided by an embodiment of the present application is shown.

[0047] Icons: 1-feature module, 2-spatial variation module, 21-local network, 22-grid generator, 23-sampler, 3-recovery recognition module, 41-processor, 42-memory, 43-communication interface, 44-communication bus. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0049] According to research, there is a method for license plate detection using convolutional neural networks. Because license plate detection and license plate recognition are strongly correlated, that is, accurate border prediction helps improve recognition accuracy, and accurate recognition results can be used to eliminate positive examples. The proposed model can be trained end-to-end without the need for additional heuristic rules (license plate color, character space), avoiding the intermediate character segmentation process. This detection method is end-to-end when making predictions, which has a great advantage in speed. It only needs to increase the input of character information during training. However, due to some extreme scenarios, the license plate images captured have image distortion, rotation, uneven lighting, blur and other low image quality. At this time, the existing detection methods will become extremely difficult to detect license plates on captured license plate images, and the recognition accuracy is low.

[0050] In order to overcome the challenges in the above-mentioned real-life scenarios, the present embodiment provides a license plate detection method, an electronic device and a computer-readable storage medium. The license plate detection model used in the license plate detection method is a controllable spatial transformer network (RP-CSTN), which can calculate the position of the license plate and predict the license plate number in a forward prediction. It is an end-to-end detection network framework, in which the sub-module STN can adjust the images in the above-mentioned extreme cases into high-quality, easy-to-recognize images.

[0051] Please refer to Figure 1 , Figure 1 A flowchart of the steps of a license plate detection method provided in an embodiment of the present application. The license plate detection method is applied to a license plate detection model, wherein the license plate detection model is referred to Figure 2 , Figure 2 This is a schematic diagram of a license plate detection model provided in an embodiment of the present application. The license plate detection model includes: a feature module 1, a spatial variation module 2 and a recovery and recognition module 3.

[0052] The license plate detection method specifically includes:

[0053] Step 100: feature module 1 extracts features from the license plate image to be identified, obtains a feature map and selects a license plate frame;

[0054] The input license plate image may be an image captured by a camera, such as a 512x512x3 image.

[0055] Step 200: The spatial variation module 2 corrects the license plate frame to obtain a corrected feature map.

[0056] Among them, the spatial transformation module is the STN module, which mainly predicts the parameters of the nonlinear transformation (affine transformation), corrects the selected box, and inputs it into the recovery and recognition module.

[0057] Step 300: The restoration and recognition module 3 dedistorts the license plate frame to obtain a standard license plate frame according to the corrected feature map, and recognizes the license plate characters according to the standard license plate frame.

[0058] The recovery and recognition module 3 recovers the standard license plate (i.e., feature reconstruction) according to the feature map, and then performs character recognition. The standard license plate here refers to a license plate generated according to the standard license plate style specification and the GT license plate number.

[0059] In the embodiment of the present application, the license plate detection method is implemented through the license plate detection model, and a spatial transformation module 2 (spatial transform network, STN) is added to the license plate detection model. The STN can learn the parameters of the affine transformation, thereby restoring the image with poor quality such as image distortion, rotation, uneven lighting and blur to a standard image, thereby improving the recognition accuracy. For the overexposure and blur caused by night, the license plate detection model of this embodiment can realize the restoration and calibration of the distorted image due to the addition of STN, which greatly improves the detection accuracy.

[0060] In some optional embodiments, the feature module 1 includes a backbone network of SSD; in step 100, features of the license plate image to be identified are extracted to obtain a feature map and a license plate frame is selected, including: the backbone network of SSD selects the license plate frame through a one-stage target detection algorithm.

[0061] In the embodiment of the present application, the feature module 1 is mainly composed of a convolutional layer, which mainly plays the role of extracting features and selecting a license plate frame. The feature module 1 of this embodiment adopts the SSD backbone network, which is a one-stage target detection algorithm. The core of the one-stage target detection algorithm is regression, which omits the region proposal link of the two-stage target detection algorithm, that is, there is no pre-classification and regression process, and the specific categories are directly divided, and the border is regressed. Therefore, the license plate frame can be quickly selected.

[0062] In some optional implementations, please refer to Figure 3 , Figure 3A schematic diagram of a spatial variation module 2 provided in an embodiment of the present application, wherein the spatial variation module 2 includes a local network 21, a grid generator 22 and a sampler 23; in step 200, the license plate frame is corrected to obtain a corrected feature map, including: the local network 21 calculates affine transformation parameters according to the license plate frame; the grid generator 22 constructs a sampling grid according to the affine transformation parameters; and the sampler 23 obtains the corrected feature map according to the sampling grid and the feature map.

[0063] When classifying, it is usually necessary to consider the locality, translation invariance, reduction invariance, rotation invariance, etc. of the input samples to improve the accuracy of classification. The essence of these invariances is the classic methods of image processing, namely, image cropping, translation, scaling, and rotation, which are actually spatial coordinate transformations of images. One of the spatial transformations is the affine transformation. In the affine transformation, the coefficient matrix θ is the affine transformation coefficient. By adjusting the coefficient matrix θ, the image can be enlarged, reduced, translated, rotated, etc.

[0064] Therefore, in this embodiment, the affine transformation coefficient θ is generated by the local network 21, and after the affine transformation coefficient θ is obtained, the input of the previous layer is affine transformed, and the affine transformation result is input to the next layer. The local network 21 is a network used to regress the transformation parameter θ. Its input is the license plate frame selected by the feature module 1 frame, and the output θ is the output of a 6-dimensional (2x3) vector. These parameters are the parameters of the affine transformation. The local network 21 is composed of a convolution layer and a regression layer. The grid generator 22 (Grid Generator) constructs a sampling grid based on the predicted transformation parameters. It is the output obtained after a set of points in the input image are sampled and transformed. The sampler 23 uses the sampling grid and the input feature map as input to generate output, and obtains the result after the feature map is transformed, that is, the corrected feature map.

[0065] In some optional embodiments, the recovery and recognition module 3 includes an instance normalization layer; step 300, before dedistorting the license plate frame to obtain a standard license plate frame according to the corrected feature map, and recognizing the license plate characters according to the standard license plate frame, also includes: normalizing the input data through the instance normalization layer to avoid the situation where the input data value is too large.

[0066] In some optional implementations, the license plate detection model is trained according to a loss function, where the loss function is:

[0067]

[0068] Among them, L D is the detection loss, β and γ are weight parameters, and N is the size of a batch in training;

[0069] Recovery loss L rec (R, S) is the mean square error between the restored image R and the standard license plate S:

[0070]

[0071] Let PN represent the number of license plate characters, g i (1≤i≤PN) represents the GT license plate character. i (1≤i≤PN) represents all the license plate characters that need to be predicted in this paper, and each predicted license plate character p i Contains nc i The loss function for character prediction is cross entropy:

[0072]

[0073] In the embodiment of the present application, by detecting the loss L D , restore the loss L rec (R,S), and recognition loss γ cls By jointly optimizing (p,g), the license plate detection model can detect, recover and recognize the license plate in one forward inference pass. This overall loss function can be used to optimize the license plate detection model during training.

[0074] In some optional implementations, the detection loss is the sum of the confidence error and the classification error:

[0075] L D = l obj +l cls

[0076] Among them, l obj is the confidence error, l cls is the classification error.

[0077] In the embodiment of the present application, the detection loss refers to the detection positioning loss, including confidence error and classification error.

[0078] In some optional implementations, the confidence error:

[0079]

[0080] Among them, λ obj and λ noobj They represent the weights of detected objects and undetected objects, respectively. i is the predicted probability when there is a prediction box for this grid Indicates 1, no Indicates 0; The grid representing each feature map will be traversed, and each grid will produce three different prediction boxes.

[0081] In some optional implementations, the classification error:

[0082]

[0083] Among them, λ class is the weight of each category, The probability of belonging to this category is 1, and it is 0 when it is not this category; p i (c) is the predicted probability of this category.

[0084] Figure 4 A possible structure of an electronic device provided by an embodiment of the present application is shown. Figure 4 The electronic device includes: a processor 41, a memory 42 and a communication interface 43. These components are interconnected and communicate with each other through a communication bus 44 and / or other forms of connection mechanisms (not shown).

[0085] The memory 42 includes one or more (only one is shown in the figure), which may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The processor 41 and other possible components may access the memory 42, and read and / or write data therein.

[0086] The processor 41 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 41 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a dedicated processor, including a neural network processor (NPU), a graphics processor (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. In addition, when there are multiple processors 41, some of them can be general-purpose processors and the other part can be dedicated processors.

[0087] The communication interface 43 includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices to exchange data. The communication interface 43 can include an interface for wired and / or wireless communication.

[0088] One or more computer program instructions may be stored in the memory 42 , and the processor 41 may read and execute these computer program instructions to implement the license plate detection method provided in the embodiment of the present application.

[0089] Understandably, Figure 4 The structure shown is for illustration only. The electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 Different structures are shown. Figure 4 The components shown in the figure can be implemented by hardware, software or a combination thereof. The electronic device may be a physical device, such as a PC, a laptop, a tablet computer, a mobile phone, a server, an embedded device, etc., or a virtual device, such as a virtual machine, a virtualized container, etc. Moreover, the electronic device is not limited to a single device, but may also be a combination of multiple devices or a cluster consisting of a large number of devices.

[0090] The present application also provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are read and executed by a computer processor, the license plate detection method provided by the present application is executed. For example, the computer-readable storage medium can be implemented as Figure 4 A memory 42 in an electronic device.

[0091] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0092] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0094] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0095] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A license plate detection method, characterized in that: The license plate detection method is applied to a license plate detection model, which includes: a feature module, a space variation module and a restoration recognition module; The method comprises: The feature module extracts features of the license plate image to be identified, obtains a feature map and selects a license plate frame; The spatial variation module corrects the license plate frame to obtain a corrected feature map; and The restoration and recognition module dedistorts the license plate frame to obtain a standard license plate frame according to the corrected feature map, and recognizes the license plate characters according to the standard license plate frame; The license plate detection model is trained according to a loss function, wherein the loss function is: Among them, L D is the detection loss, β and γ are weight parameters, and N is the size of a batch in training; L rec (R, S) is the mean square error between the restored image R and the standard license plate S: γ cls (p,g) is the cross entropy: Among them, p is the probability predicted by the model, g is the probability of the true label, m is the number of training batches, and g i Indicates GT license plate characters, p i Represents all the license plate characters that need to be predicted in this paper. Each predicted license plate character p i Contains nc i PN represents the number of license plate characters.

2. The license plate detection method according to claim 1, characterized in that: in, The feature module includes a backbone network of SSD; The method of extracting features of the license plate image to be recognized, obtaining a feature map and selecting a license plate frame includes: The SSD backbone network selects the license plate frame through a one-stage target detection algorithm.

3. The license plate detection method according to claim 1, characterized in that: The spatial variation module includes a local network, a grid generator and a sampler; the correction of the license plate frame to obtain a corrected feature map includes: The local network calculates affine transformation parameters according to the license plate frame; The grid generator constructs a sampling grid according to the affine transformation parameters; The sampler obtains a corrected feature map based on the sampling grid and the feature map.

4. The license plate detection method according to claim 1, characterized in that: The restoration and recognition module includes an instance normalization layer; before the license plate frame is dedistorted to obtain a standard license plate frame according to the corrected feature map, and the license plate characters are recognized according to the standard license plate frame, it also includes: The input data is normalized by the instance normalization layer.

5. The license plate detection method according to claim 1, characterized in that: The detection loss is the sum of the confidence error and the classification error: L D ＝l obj +l cls Among them, l obj is the confidence error, l cls is the classification error.

6. The license plate detection method according to claim 5, characterized in that: The confidence error: Among them, λ obj and λ noobj They represent the weights of detected objects and undetected objects, respectively. i is the predicted probability, The grids of each feature map will be traversed, and each grid will generate three different prediction boxes; obj means there is a license plate, and noobj means there is no license plate; Represents the true label, which is equal to 1 if the grid contains the predicted box, otherwise it is equal to 0.

7. The license plate detection method according to claim 5, characterized in that: The classification error: Among them, λ class is the weight of each category, The probability of belonging to this category is 1, and it is 0 when it is not this category; p i (c) is the predicted probability of this category.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the license plate detection method according to any one of claims 1 to 7 is performed.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the license plate detection method as described in any one of claims 1 to 7.

Images