A license plate character recognition method based on an improved LPRNet network

Abstract

The application provides a license plate character recognition method based on an improved LPRNet network, comprising: obtaining a license plate character image to be processed; constructing an improved LPRNet network model, training the improved LPRNet network model, then inputting the license plate character image to be processed, and outputting a recognition result of the license plate character. The improved LPRNet network model introduces a symmetric space transformation network, which significantly improves the reliability of subsequent feature extraction and classification tasks. In addition, the application performs hierarchical progressive processing on the input license plate image: from the bottom layer network to capture fine-grained local features such as character edges and textures, to the middle layer network to aggregate spatial correlation information between characters, and then to the top layer network to extract global features of the overall layout of the license plate, realizing efficient hierarchical extraction of multi-scale features, significantly enhancing the adaptive ability of the model to license plates of different resolutions, and the feature recognition ability in complex backgrounds and harsh shooting conditions.

Description

Technical Field

[0001] This invention relates to the field of license plate recognition system technology, and in particular to a license plate character recognition method based on an improved LPRNet network. Background Technology

[0002] With rapid economic development and accelerated urbanization, transportation demand has increased significantly, leading to increased pressure on traffic management. Against this backdrop, Intelligent Transportation Systems (ITS) and Advanced Driving Assistance Systems (ADAS) have emerged as important means to alleviate traffic congestion and improve traffic efficiency. Within ITS, license plate recognition technology, which uses computer vision and image processing techniques to extract vehicle license plate information from images or videos, has applications in many fields such as road traffic management and parking management.

[0003] Despite significant progress in license plate recognition technology, traditional license plate recognition systems are primarily designed for static scenarios. While they can detect and recognize common license plates such as blue and green plates under relatively ideal conditions, they exhibit obvious limitations in more complex dynamic scenarios. Firstly, complex weather conditions significantly impact performance: in adverse environments such as rain, snow, or fog, the system's performance degrades considerably, failing to accurately extract license plate area and character information. Secondly, varying lighting conditions pose challenges: in both low and high light, license plate recognition systems are prone to detection failures or misidentifications. Furthermore, dynamic distance variations affect performance: in dynamic scenarios where vehicles are far from or close to the camera, the clarity and size of the license plate change considerably, making it difficult for existing systems to adapt to such complex variations.

[0004] In recent years, deep learning has enabled new developments in license plate recognition technology. LPRNet (License Plate Recognition Network), a deep learning-based license plate recognition network, can automatically extract features from license plate images and perform character recognition through end-to-end learning, achieving improvements in both accuracy and efficiency compared to traditional methods. However, existing LPRNet networks still have some shortcomings. For example, its Spatial Transform Network (STN) module relies primarily on global image features for overall geometric transformation when performing license plate geometric correction, making it difficult to accurately capture local distortions at the character level. In practical applications, when license plates exhibit complex local bending, twisting, or other distortions, the STN module's correction process is prone to character misalignment and overstretching, resulting in corrected images that fail to meet the high-quality input requirements of subsequent feature extraction and classification tasks, thus affecting the overall performance of the recognition system.

[0005] Furthermore, in terms of feature extraction, the existing LPRNet network has difficulty accurately capturing and effectively fusing some fine-grained character features and spatial arrangement patterns between characters, which leads to the need to improve the recognition accuracy and generalization robustness in complex scenarios.

[0006] Therefore, how to optimize license plate recognition methods for dynamic and complex environments to improve recognition accuracy remains a key focus and challenge in this field of research. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art and provide a license plate character recognition method based on an improved LPRNet network. This method addresses various complex situations that may be encountered in license plate character recognition, such as different fonts, lighting variations, and character occlusion, by making targeted improvements and optimizations to the LPRNet network to enhance recognition accuracy and robustness.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a license plate character recognition method based on an improved LPRNet network, comprising the following steps:

[0010] Obtain the license plate character image to be processed;

[0011] An improved LPRNet network model, SD-LPRNet, is constructed. The improved LPRNet network model SD-LPRNet includes a cascaded S-STN geometric correction module, a CBR basic feature extraction module, and a DilateFormer feature extraction module. The S-STN geometric correction module is used to perform character-level spatial transformation and correction on the input license plate image. The CBR basic feature extraction module is used to extract the low-level visual information of the image. The DilateFormer feature extraction module is used to extract multi-scale features of the image. After pooling, Dropout, and secondary feature extraction, the recognition result of the license plate characters is output.

[0012] The improved LPRNet network model SD-LPRNet is trained, and then the license plate character image to be processed is input, and the recognition result of the license plate character is output.

[0013] Secondly, the present invention provides a license plate character recognition system for implementing the above method, comprising the following modules:

[0014] The image acquisition module is used to acquire the license plate character image to be processed;

[0015] The character recognition module inputs the license plate character image to be processed into the trained improved LPRNet network model SD-LPRNet to obtain the recognition result of the license plate character.

[0016] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor executing the machine-executable instructions to implement the above-described method.

[0017] Fourthly, the present invention provides a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the above-described method.

[0018] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0019] This invention provides a license plate character recognition method based on an improved LPRNet network. The improved LPRNet network improves the Spatial Transform Network (S-STN) by constructing a centerline and using Thin Plate Spline Transform (TPS) to achieve adaptive correction of tilt, bending and irregular distortion in license plate images. It effectively overcomes the problem of excessive stretching or misalignment of some characters caused by traditional Spatial Transform Networks relying only on global features. This achieves high-quality geometric correction of complex distorted license plates and provides a standardized and regular input data foundation for subsequent feature extraction.

[0020] Furthermore, the DilateFormer feature extraction module used in the improved LPRNet network has a pyramid-shaped multi-scale perceptual structure. By integrating the Multi-Scale Extended Attention (MSDA) mechanism and the Conditional Placement Embedding (CPE) strategy, it can dynamically adjust the feature extraction range under different receptive fields. This enables the simultaneous capture and deep fusion of fine-grained texture features (such as stroke details) and global semantic features (such as character spacing and spatial layout) of license plate characters. Consequently, it significantly enhances the model's feature expression ability when facing different resolutions, image blur, or background interference, greatly improving the accuracy of license plate character recognition. It is widely applicable to various complex scenarios such as road monitoring, parking lot management, and traffic violation capture, and has strong practicality and promotional value. Attached Figure Description

[0021] Figure 1 This is a diagram of the MSDA model architecture in this invention.

[0022] Figure 2 This is a flowchart of the DilateFormer module in this invention.

[0023] Figure 3This is a diagram of the SD-LPRNet network architecture in this invention.

[0024] Figure 4 This is a diagram of the S-STN geometric correction module architecture.

[0025] Figure 5 The graph shows the performance curves of the old and new LPRNet models under CNLP.

[0026] Figure 6 The graph shows the performance curves of the old and new LPRNet models under CCPD. Detailed Implementation

[0027] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0028] Step S1: Obtain the license plate character image to be processed;

[0029] Step S2: Construct the improved LPRNet network model SD-LPRNet and process the license plate character image to obtain the license plate character recognition results. For example... Figure 3 As shown, the improved LPRNet network model SD-LPRNet includes a cascaded S-STN geometric correction module, a CBR basic feature extraction module, and a DilateFormer feature extraction module.

[0030] Specifically:

[0031] 2-1 After acquiring the license plate character image to be processed, the S-STN geometric correction module is first used to correct it to obtain a regular license plate image;

[0032] Geometric correction of license plate images is fundamental to ensuring subsequent recognition accuracy. Its core objective is to address the irregularities in license plates caused by factors such as shooting angle, installation tilt, and lens distortion in real-world scenarios. This step employs the geometric correction module of the Supervised Spatial Transformer Network (S-STN), such as... Figure 4 As shown, the S-STN geometry correction module uses ResNet-50 as the backbone network, and also introduces Thin Plate Spline Transform (TPS) and explicit geometric supervision.

[0033] Specifically, the original license plate image is first input into the ResNet-50 backbone network, where deep residual features are extracted using FPN (Feature Pyramid Network). In this step, a supervised localization branch is constructed to obtain the center point coordinates of the license plate characters through pixel-level prediction. The center points are then linearly or curvilinearly fitted using the least squares method to construct a virtual center line representing the character arrangement trend. Simultaneously, the coordinates of the four vertices of the license plate are predicted. The effective character region of the license plate is determined by combining the virtual center line and the four vertices, and the distribution of control points used for TPS transformation is calculated accordingly. Subsequently, the control point offsets calculated in the previous step are input into the TPS thin-plate spline transform operator to establish a nonlinear mapping relationship. Pixel-level resampling is performed on the feature map or image to accurately correct complex local distortions of the license plate caused by the shooting angle (such as barrel distortion and wavy curvature), outputting a horizontally arranged, proportionally regular character image. Finally, this image is sent to the recognition module for text recognition, completing the license plate correction.

[0034] 2-2 After obtaining the regular license plate image, the underlying visual information (basic features) of the image is mined through the CBR basic feature extraction module to provide raw materials for high-order feature optimization.

[0035] The CBR basic feature extraction module comprises a series of convolutional layers (Conv), batch normalization (BN) layers, and an activation function (ReLU), achieving the dual goals of feature extraction and model optimization. The convolutional layers use 3×3 kernels with a stride of 1 and padding of 1, fully extracting basic features such as character edges, textures, and grayscale distribution while maintaining the feature map size. The 3×3 kernel design balances receptive field size and computational efficiency, and allows for the combination of complex features through multi-layer stacking. The batch normalization (BN) layer, as a key optimization component, standardizes the feature map output from the convolutional layers, adjusting the data distribution to a normal distribution with a mean close to 0 and a variance close to 1. This effectively reduces the "internal covariate shift" phenomenon, accelerates model training convergence, and reduces sensitivity to initial weights. The subsequent Rectified Linear Unit (ReLU) activation function sets negative input values ​​to zero, constructing a non-linear mapping relationship. This not only solves the gradient saturation problem of the traditional Sigmoid function but also enhances the model's ability to express non-linear features, making the features more discriminative. The overall design of the CBR module balances the completeness of feature extraction with the efficiency of model training. The output local feature map not only retains the key visual information of the license plate characters, but also improves the stability of the data through standardization, laying a solid foundation for multi-scale feature optimization.

[0036] 2-3 The low-level visual information extracted in step 2-2 is processed by max pooling and small basic blocks, and then input into the DilateFormer feature extraction module for multi-scale feature fusion and optimization to obtain a multi-scale feature map of the image; wherein the small basic block includes four cascaded convolutional layers. The DilateFormer module adopts a pyramid structure and processes the information sequentially through the following sub-steps:

[0037] like Figure 2 As shown, the DilateFormer feature extraction module includes a cascaded overlapping word segmenter, a first MSDA (Multi-scale Extended Entity) module, a first overlapping downsampler, a second MSDA module, a second overlapping downsampler, a first multi-self-attention (MHSA) module, a third overlapping downsampler, and a second MHSA module; wherein, the architecture of the first MSDA module and the second MSDA module is the same, as shown in the figure. Figure 1 As shown, low-level features are first extracted through a depthwise separable convolutional (DwConv) layer. After normalization, the features are input into a multi-scale sliding window expanded attention (SWDA) layer to capture multi-scale local information and inter-block interaction relationships. After concat operation, a linear layer and a normalization layer, the features are input into a multilayer perceptron (MLP) to integrate features and improve abstract representation capabilities.

[0038] (1) Pattern embedding and initial feature extraction: The overlapping word segmentation algorithm (3×3 convolution with overlapping receptive fields) is used to slide scan the feature map and map the local image region into high-dimensional tokens. While achieving dimensionality reduction and compression, the spatial continuity between adjacent characters is preserved, thus realizing feature map pattern embedding. The initial features are extracted through the model, and the feature parsing is achieved by adjusting the convolution kernel stride.

[0039] (2) Multi-scale local feature extraction: The first two steps extract multi-scale local features through the MSDA (Multi-scale Extended Entity) module. The MSDA module extracts low-level features through depthwise separable convolution (DwConv), captures multi-scale local information and inter-block interaction relationships through multi-scale sliding window expanded attention (SWDA), and integrates features through multi-layer perceptron (MLP) to improve abstract representation capabilities.

[0040] (3) Global information enhancement: The last two steps enhance global semantic association through the multiple self-attention (MHSA) module to achieve the fusion of local and global information.

[0041] (4) Adaptive location embedding: The conditional location embedding (CPE) method is used to adaptively process multi-spatial resolution data, thereby improving the understanding and robustness of spatial location relationships.

[0042] (5) Feature integration: Image patches are integrated by an overlapping undersampler with 3 kernel sizes and 2 step sizes to output an optimized multi-scale feature map.

[0043] 2-4 The multi-scale feature maps output by the DilateFormer feature extraction module are processed by max pooling and small basic blocks. Then, in order to further refine the features and reduce the model complexity, the optimized feature maps are processed by pooling operations and dropout mechanism to obtain high-level features.

[0044] Specifically, the pooling operation employs parallel operations of max pooling and average pooling: max pooling selects the maximum value within each receptive field, which can highlight the key edges and salient features of characters (such as the inflection points of Chinese characters and the sharp corners of letters); average pooling calculates the mean value within each receptive field, which can preserve the overall outline and grayscale distribution features of characters. The feature maps output by the two are fused by the concat operation to obtain a more comprehensive and balanced high-level feature representation.

[0045] To address the overfitting problem common in deep networks, a dropout mechanism is introduced with a dropout probability of 0.5. During training, some neurons are randomly deactivated, forcing the model to learn more robust feature representations and avoiding over-reliance on the outputs of specific neurons. This mechanism not only reduces the computational complexity of the model (by decreasing the number of neurons involved in the computation) but also enhances its resistance to noise, blur, and other interference factors by increasing the model's randomness, resulting in stronger generalization performance in complex real-world scenarios. The feature maps processed by deep learning effectively filter out redundant information, resulting in more concise core features and providing high-quality input for final feature extraction.

[0046] 2-5 Input the high-level features into the container feature integration container for secondary feature extraction to obtain the final feature vector; input the final feature vector into the classifier (Softmax is used in this embodiment) to output the recognition result of the license plate characters.

[0047] The container module consists of a set of 1×1 convolutional layers, dropout layers, and an output mapping layer. The 1×1 convolutional layers reduce the dimensionality of the high-dimensional feature vector obtained after multi-scale fusion and map it to the license plate character category space (including Chinese characters, letters, numbers, and whitespace placeholders), obtaining the final feature representation. Subsequently, this feature vector is input into the classifier, which outputs a probability distribution sequence of the license plate characters.

[0048] During the model training phase, CTC Loss (Connected-Interstitial Temporal Classification Loss Function) is used as the objective function for end-to-end optimization. Since license plate recognition is a sequence learning task, CTC Loss calculates the conditional probability between the predicted sequence and the true label by introducing a blank label and summing all possible character permutation paths. The formula is as follows:

[0049]

[0050] in, Given the input prediction sequence, For real labels, For path, It is a set of paths.

[0051] To further demonstrate the effectiveness of the method of the present invention, different methods, including the method of the present invention, were used to test existing public datasets.

[0052] like Figure 5 The figures show the differences in prediction accuracy and detection speed among four models: LPRNet, S-LPRNet, D-LPRNet, and SD-LPRNet (this invention). The accuracy curves show that all models experience a significant initial increase, then plateau as the epochs increase, eventually reaching a stable level. Among them, SD-LPRNet consistently maintains the highest prediction accuracy.

[0053] Table 1. Performance Comparison of New and Old LPRNet Models under CNLP

[0054] network Dataset S-STN DilateFormer Accuracy / % Detection speed / ms Params (MB) LPRNet CNLP × × 94.12 12.8 1.76 S-LPRNet CNLP √ × 95.36 16.2 1.76 D-LPRNet CNLP × √ 95.59 15.3 1.78 SD-LPRNet CNLP √ √ 96.45 18.5 1.79

[0055] Through the Figure 4Analysis of the test results in Table 1 verifies that the accuracy achieved using LPRNet as the base model is 94.12%. Based on this, the S-STN algorithm further improves the accuracy to 95.36% without reducing detection speed. Experimental results show that S-STN is an effective method for improving LPRNet performance. Combining D-LPRNet with DilateFormer, without using S-STN, achieves an accuracy of 95.59%. Even without S-STN, DilateFormer still improves model performance. Combining S-STN with DilateFormer achieves an accuracy of 96.45%. Although the detection time increases (from 12.8 ms to 18.5 ms), the slight sacrifice in detection speed results in a significant improvement in accuracy. Furthermore, the final SD-LPRNet model has only 1.79 MB of Params (model weights), making it suitable for mobile terminals, embedded systems, and edge devices. This will reduce deployment costs and improve model usability.

[0056] Depend on Figure 6 As can be seen from the “Accuracy” chart on the left, the accuracy curve of SD-LPRNet approaches 1, indicating that the method of this invention has high training accuracy.

[0057] Therefore, SD-LPRNet exhibits good balance on CCPD data, outperforming LPRNet in accuracy, and demonstrates strong overall performance.

[0058] Based on this, SD-LPRNet was simulated and compared.

[0059] Table 2 Performance Comparison Study of License Plate Recognition Models under CCPD

[0060] network Dataset Accuracy / % Detection speed / ms ANe CCPD 95.38 24.9 RDNet CCPD 96.07 19.7 LPRNet CCPD 94.89 12.2 SD-LPRNet CCPD 97.57 18.1

[0061] In terms of accuracy, SD-LPRNet performed best at 97.57%, indicating that it outperformed the other three networks in license plate recognition.

[0062] In summary, SD-LPRNet demonstrates excellent performance, maintaining high accuracy while keeping the detection speed quite fast. This makes it well-suited for the CCPD dataset, especially for applications requiring very high accuracy.

[0063] It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments. It should be understood that although this specification describes the embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A license plate character recognition method based on an improved LPRNet network, characterized in that, Includes the following steps: Obtain the license plate character image to be processed; An improved LPRNet network model, SD-LPRNet, is constructed. The improved LPRNet network model SD-LPRNet includes a cascaded S-STN geometric correction module, a CBR basic feature extraction module, and a DilateFormer feature extraction module. The S-STN geometric correction module is used to perform character-level spatial transformation and correction on the input license plate image. The CBR basic feature extraction module is used to extract the low-level visual information of the image. The DilateFormer feature extraction module is used to extract multi-scale features of the image. After pooling, Dropout, and secondary feature extraction, the recognition result of the license plate characters is output. The improved LPRNet network model SD-LPRNet is trained, and then the license plate character image to be processed is input, and the recognition result of the license plate character is output.

2. The license plate character recognition method based on the improved LPRNet network according to claim 1, characterized in that, The S-STN geometric correction module first semantically segments and locates the center point of the character and fits the center line. At the same time, it obtains the coordinates of the four corner points of the license plate based on the edge detection algorithm and constructs the correction area in combination with the center line of the text. Subsequently, it calculates the offset and deformation coefficient of the pixel points for the correction area and uses Thin Plate Spline Transform (TPS) to convert the tilted or distorted license plate image into a regular corrected image.

3. The license plate character recognition method based on the improved LPRNet network according to claim 1, characterized in that, The CBR basic feature extraction module includes a series of convolutional layers, batch normalization layers, and ReLU activation function layers.

4. The license plate character recognition method based on the improved LPRNet network according to claim 1, characterized in that, The DilateFormer feature extraction module includes a series of overlapping word segmenters, a first MSDA module, a first overlapping downsampler, a second MSDA module, a second overlapping downsampler, a first MHSA module, a third overlapping downsampler, and a second MHSA module, which ultimately outputs a multi-scale feature map of the image.

5. The license plate character recognition method based on the improved LPRNet network according to claim 4, characterized in that, The first MSDA module and the second MSDA module have the same architecture. First, low-level features are extracted through depthwise separable convolutional layers. After normalization, they are input into a multi-scale sliding window to expand the attention layer to capture multi-scale local information and inter-block interaction relationships. After concat operation, a linear layer and a normalization layer, they are input into a multilayer perceptron to integrate features and extract multi-scale local features.

6. The license plate character recognition method based on the improved LPRNet network according to claim 1, characterized in that, The specific results of license plate character recognition after pooling, Dropout, and secondary feature extraction include: Max pooling and average pooling operations are performed in parallel on the multi-scale feature maps output by the DilateFormer feature extraction module. The outputs of max pooling and average pooling are concatenated and merged. The Dropout mechanism is introduced to randomly zero out some neurons, and then a second feature extraction is performed through a convolutional network to obtain the final feature vector. The final feature vector is input into the classifier, which outputs the recognition result of the license plate characters.

7. A license plate character recognition system implementing the method as described in any one of claims 1-6, characterized in that, Includes the following modules: The image acquisition module is used to acquire the license plate character image to be processed; The character recognition module inputs the license plate character image to be processed into the trained improved LPRNet network model SD-LPRNet to obtain the recognition result of the license plate character.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-6.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method as described in any one of claims 1-6.

Images