License plate recognition method in low-illumination environment

Abstract

The invention discloses a license plate recognition method in a low-illumination environment, which relates to the technical field of license plate recognition and is characterized by comprising the following steps of: enhancing a license plate picture to be recognized through CLAHE-GAMMA and then sending the license plate picture to be recognized into an improved YOLOv5n segmentation network; carrying out license plate segmentation by using an improved YOLOv5n segmentation network; and extracting four boundary points of the mask obtained by segmentation, stretching the mask into a rectangle, and sending the rectangle into an improved LPRNet character recognition network for license plate recognition. According to the improved algorithm provided by the invention, the license plate feature extraction capability of the YOLOv5n segmentation network can be effectively improved, the feature fusion capability of the YOLOv5n segmentation network can be enhanced, the license plate character recognition accuracy of the LPRNet network can also be effectively improved, a good effect is achieved on pixel precision or frame number in comparison with other license plate detection algorithms, and the method is suitable for large-scale popularization and application. And the detection precision and speed requirements of license plate recognition under the condition of low illumination in a vehicle charging place can be met.

Description

Technical Field

[0001] The invention belongs to the technical field of license plate recognition, and in particular relates to a license plate recognition method in a low-illumination environment. Background Art

[0002] In electric vehicle charging places, in order to prevent non-electric vehicles from occupying charging parking spaces, cameras can be installed on charging piles and combined with target recognition technology to identify the type of car and license plate number, and then the charging parking spaces can be effectively utilized by warning and driving away non-electric vehicles and assisting with ground lock control. At the same time, vehicle statistical information can also be used for big data analysis of charging vehicle information and effective scheduling of charging piles. However, the accuracy of license plate recognition in low light, strong light, rain, snow and fog environments is challenging.

[0003] At present, the implementation method of license plate recognition tasks is mainly based on the semantic segmentation-character recognition process. The former can be used to identify the boundary features of different targets and help achieve the angle correction of the license plate, and the latter can be used to further identify the type and number of the license plate. Semantic segmentation is a very challenging subject in the field of computer vision. In addition to locating the target, it also needs to further detect each pixel involved in the target and assign a label to it.

[0004] There are roughly two methods for implementing semantic segmentation. Most of the early semantic segmentation methods were based on the grayscale, contrast, saturation, and even texture, spectrum and other morphological features of the image to identify the boundary features of different targets, thereby segmenting the image into different semantic categories. There are problems such as blurred edges and low efficiency. With the development of deep learning technology, Long et al. proposed FCN (Fully Convolutional Network) in 2014. This is the first model to introduce deep learning technology into the field of semantic segmentation. Its disadvantages are the loss of details caused by convolution downsampling and the lack of consideration of context. Ronneberger et al. proposed UNet in 2015. Its feature is the use of encoder-decoder to form a "U"-shaped structure. The subsequent DeepLab series of segmentation networks tried backbone networks such as VGG, ResNet, and Xception, and used dilation convolution on a large scale. Semantic segmentation is widely used in fields such as autonomous driving, geographic information systems, and medical image analysis.

[0005] There are two types of character recognition methods. In the early days, traditional character recognition methods mostly matched processed characters with templates and used artificially constructed features and character classification for recognition. With the development of deep learning technology, the recognition methods of text based on deep recognition can be roughly divided into two types: one is to segment and recognize the character string, that is, the segmentation method; the other is to directly perform the recognition network with sequence annotation on the entire character string, that is, the non-segmentation method. Character recognition is widely used in document processing in the fields of calligraphy, government affairs, and commerce. Summary of the invention

[0006] The technical problem to be solved by the present invention is to solve the deficiencies in the prior art and to design a new license plate recognition method in a low-light environment. This license plate recognition method has a perfect structure and its main idea is enhancement-segmentation-recognition, which can meet the detection accuracy and speed requirements of license plate recognition in low-light conditions in vehicle charging places.

[0007] The solution adopted by the present invention to solve the technical problem is:

[0008] A method for license plate recognition in a low-light environment, characterized in that it comprises the following steps:

[0009] Step 1: After the license plate image to be identified is enhanced by CLAHE-GAMMA, it is sent to the improved YOLOv5n segmentation network;

[0010] Step 2: Use the improved YOLOv5n segmentation network for license plate segmentation;

[0011] Step 3: Extract the four boundary points of the segmented mask and stretch the mask into a rectangle, send it to the improved LPRNet character recognition network, and use the improved LPRNet for license plate recognition.

[0012] As a preferred embodiment of the present invention, the CLAHE-GAMMA enhancement in step 1 comprises the following steps:

[0013] S01: Convert the image from RGB color space to HSV color space, perform CLAHE transformation on V channel only, and convert the image after CLAHE transformation from HSV space back to RGB space;

[0014] S02: Average the values ​​of the R channel, G channel and B channel of the image converted back to the RGB space and perform adaptive GAMMA transformation.

[0015] As a preferred embodiment of the present invention, the improved YOLOv5n segmentation network in step 1 replaces the CBS module in the YOLOv5 backbone network with DynamicConv, and combines the C3 module in the YOLOv5 backbone network with DynamicConv, and applies the "low FLOPs trap" idea in the process; then replaces the C3 module in the YOLOv5 neck network with the RepNCSPELAN module proposed by YOLOv9.

[0016] As a preferred implementation of the present invention, the improved LPRNet character recognition network in step 1 specifically adds an EMA attention module after the Dropout layer of the original LPRNet character recognition network.

[0017] As a preferred implementation of the present invention, the formula for CLAHE transformation in S01 is:

[0018] s=cr γ

[0019]

[0020] Where s is the output value, c is the gamma compensation coefficient, r is the input value, γ is the correction coefficient, and M is the normalized channel average.

[0021] As a preferred implementation of the present invention, when performing CLAHE transformation on the V channel in S01, the contrast threshold is set to 5 and the grid size is set to 8*8.

[0022] As a preferred implementation of the present invention, the input value r needs to be normalized in advance.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention feeds the license plate image to be recognized into the improved YOLOv5n segmentation network after CLAHE-GAMMA enhancement, extracts the four boundary points of the segmented mask and stretches the mask into a rectangle, and feeds it into the improved LPRNet character recognition network to complete the license plate recognition. The improved YOLOv5n segmentation network replaces the CBS module in the YOLOv5 backbone network with DynamicConv, and combines the C3 module in the backbone network with DynamicConv; replaces the C3 module in the YOLOv5 neck network with the RepNCSPELAN module proposed by YOLOv9; and adds the EMA attention mechanism after the two Dropout layers of the improved LPRNet character recognition network.

[0025] The improved algorithm proposed in this paper can effectively improve the YOLOv5n segmentation network's ability to extract license plate features and enhance its feature fusion capability. In addition, it can effectively improve the accuracy of the LPRNet network in recognizing license plate characters. In comparison with other license plate detection algorithms, good results have been achieved in terms of pixel accuracy or frame rate. It can meet the detection accuracy and speed requirements of license plate recognition under low illumination conditions in vehicle charging places. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A step diagram of a license plate recognition method in a low-light environment proposed by the present invention;

[0027] Figure 2 A diagram showing the steps of CLAHE-GAMMA enhancement of a license plate recognition method in a low-light environment proposed by the present invention;

[0028] Figure 3 The YOLOv5 segmentation network structure is used in the license plate recognition method under low illumination environment proposed by the present invention;

[0029] Figure 4 An improved YOLOv5n segmentation network structure for a license plate recognition method in a low-light environment proposed by the present invention;

[0030] Figure 5 The structure of GELAN cited in the method for license plate recognition in a low-light environment proposed by the present invention;

[0031] Figure 6 A method for license plate recognition in a low-light environment RepNCSPELAN and the structure of its components proposed by the present invention;

[0032] Figure 7 The LPRNet network structure is used in the method for license plate recognition in a low-light environment proposed by the present invention;

[0033] Figure 8 The CA module and the EMA module are used in the license plate recognition method in a low-light environment proposed by the present invention;

[0034] Fig. 9 The invention discloses an improved LPRNet network structure for a license plate recognition method in a low-light environment. DETAILED DESCRIPTION

[0035] The specific implementation of the present invention is described below in conjunction with the accompanying drawings and embodiments:

[0036] It should be noted that the structures, colors, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0037] At the same time, in the description of the present invention, it should be understood that the terms "one end", "the other end", "middle", "upper", "one side", "top", "inside", "front", "center", "both ends" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0038] In addition, the terms "first", "second", "third" and "fourth" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", "third" and "fourth" may explicitly or implicitly include at least one of such features.

[0039] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "screw-on" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0040] like Figure 1 The present invention provides a method for recognizing a license plate in a low-light environment, comprising the following steps:

[0041] Step 1: After the license plate image to be identified is enhanced by CLAHE-GAMMA, it is sent to the improved YOLOv5n segmentation network;

[0042] Step 2: Use the improved YOLOv5n segmentation network for license plate segmentation;

[0043] Step 3: Extract the four boundary points of the segmented mask and stretch the mask into a rectangle, send it to the improved LPRNet character recognition network, and use the improved LPRNet for license plate recognition.

[0044] End-to-end license plate recognition is achieved by combining CLAHE and GAMMA transform, YOLOv5 segmentation network and LPRNet character recognition network.

[0045] The specific improvements are as follows: the license plate image to be recognized is enhanced by CLAHE-GAMMA and then sent to the improved YOLOv5n segmentation network, the four boundary points of the segmented mask are extracted and the mask is stretched into a rectangle, and then sent to the improved LPRNet character recognition network to complete the license plate recognition.

[0046] The method improvements include: the improved YOLOv5n segmentation network replaces the CBS module in the YOLOv5 backbone network with DynamicConv, and combines the C3 module in the backbone network with DynamicConv; replaces the C3 module in the YOLOv5 neck network with the RepNCSPELAN module proposed by YOLOv9; and adds the EMA attention mechanism after the two Dropout layers of the improved LPRNet character recognition network. This method flow can meet the detection accuracy and speed requirements of license plate recognition in low-light conditions at vehicle charging sites.

[0047] Compared with the original model, the improved model has improved the average accuracy of the mask class of the segmentation model by about 2%, while losing a small amount of frames while maintaining real-time performance; the accuracy of the recognition model has increased by about 9%.

[0048] like Figure 2 As shown, the CLAHE-GAMMA enhancement in step 1 includes the following steps:

[0049] S01: Convert the image from RGB color space to HSV color space, perform CLAHE transformation on V channel only, and convert the image after CLAHE transformation from HSV space back to RGB space;

[0050] S02: Average the values ​​of the R channel, G channel and B channel of the image converted back to the RGB space and perform adaptive GAMMA transformation.

[0051] CLAHE (Contrast Limited Adapted Histogram Equalization) is a fast and effective image enhancement method that can equalize the channel histogram distribution of an image, thereby enhancing the contrast of the image and improving readability. This method is effective in reducing the difficulty of learning the data set for the network and improving the model's effect on license plate segmentation for images that are difficult to read, such as underexposed and overexposed images, such as the vehicle license plate images in low-light conditions in the present invention.

[0052] CLAHE is based on HE (Histogram Equalization). HE can make the channel values ​​as evenly distributed as possible by adjusting the channel histogram distribution. However, when there are both too bright and too dark areas in the image, the dark area will be adjusted to be too bright, and the noise will be amplified, weakening the image details. AHE (Adapted Histogram Equalization) is an upgrade of HE. Its optimization of HE is to divide the image into multiple areas and perform HE on each area separately. Although this method can solve the problem of HE being too balanced, calculating the histogram distribution for each area separately will bring more calculations. Moreover, since there is no linkage between areas, if the histogram distribution of adjacent areas is very different, the edges will be too obvious, making the final output image look like it has been cut into several pieces. CLAHE optimizes AHE. It uses interpolation to optimize the transition problem of the edges of each area, and uses the method of limiting contrast to suppress the problem of noise amplification. The specific method is to compare the value in the calculated histogram with the set threshold, and evenly distribute the part exceeding the threshold to the histogram so that it is evenly distributed on the entire histogram.

[0053] GAMMA transformation is a nonlinear transformation that can adjust underexposed and overexposed images to make the light and dark distribution more uniform and the dark details more obvious. Its formula is:

[0054] s=cr γ

[0055] Where s is the output value, c is the gamma compensation coefficient, r is the input value, and γ is the correction coefficient. The input value r in this formula needs to be normalized in advance.

[0056] In order to make the image enhancement method more generalizable, the present invention uses an adaptive GAMMA transform. The adaptive GAMMA transform is more adaptable than the GAMMA transform because it does not require manual setting of the correction coefficient γ, and the gamma correction coefficient can be calculated from the normalized channel average of the image. Its formula is:

[0057]

[0058] Where M is the normalized channel average.

[0059] The present invention uses the CLAHE-GAMMA enhancement method to process the license plate data set. The specific method is as follows: first, the image is converted from the RGB color space to the HSV color space. Since HSV represents Hue, Saturation and Value respectively, in order not to cause unnecessary damage to the original color of the image, only the V channel is subjected to CLAHE transformation, wherein the contrast threshold is set to 5, the grid size is set to 8*8, and finally the image is converted back from the HSV space to the RGB space; secondly, in order to reduce the color cast effect of GAMMA transformation on the three-channel image, the RGB three-channel values ​​of the image after CLAHE transformation are all averaged and an adaptive GAMMA transformation is performed.

[0060] YOLOv5 is an object detection model launched by Ultralytics. Its network structure can be roughly divided into the backbone network, the neck network and the head network. Its network structure reflects the idea of ​​modular construction and is composed of a large number of basic modules stacked together, including CBS, C3 and SPPF, as well as basic upsampling and splicing operations. Among them, the backbone network contains 5 CBS modules and 4 C3 modules, and there is a fast spatial pyramid pooling layer (Spatial Pyramid PoolingFast, SPPF) at the end. This module replaced SPP in the 6.0 version update of YOLOv5. The backbone network downsamples the input image multiple times through convolution to obtain feature maps with smaller and smaller sizes and more and more channels. Secondly, the neck network of YOLOv5 contains 4 CBS modules and 4 C3 modules, and there is feature splicing between this layer and from the backbone network. The neck network upsamples the feature map and fuses the information from the shallow network in a similar way to residual connection to generate feature maps of different scales. Finally, the segmentation head of YOLOv5 segments the feature map output by the neck. The YOLOv5 segmentation network structure is as follows Figure 3 shown.

[0061] The improved YOLOv5n segmentation network replaces the CBS module in the YOLOv5 backbone network with DynamicConv, and combines the C3 module in the YOLOv5 backbone network with DynamicConv, using the "low FLOPs trap" idea in the process; then replaces the C3 module in the YOLOv5 neck network with the RepNCSPELAN module proposed by YOLOv9. The improved YOLOv5n segmentation network structure is as follows Figure 4 shown.

[0062] It is worth mentioning that there are differences in parameters such as the number of convolution kernels, stride, padding, and number of output channels in the CBS module, and the Bottleneck form in the C3 module at different locations is also different. YOLOv5 added segmentation functionality in the 7.0 update in 2022. Its segmentation network structure is mostly the same as the detection network. The only difference is that the detection head is replaced by the segmentation head in the head network.

[0063] GELAN (Generalized Efficient Layer Aggregation Network) is one of the basic modules in the YOLOv9 network proposed in 2024. It is designed to balance inference speed and accuracy. It is a promotion of ELAN (Efficient Layer Aggregation Network) in the YOLOv7 network released in 2022. It combines CSPNet and ELAN, so that any module can be used in the place where standard convolution was originally used in ELAN. This makes ELAN with a new architecture more flexible and adaptable to different tasks and scenarios, with universality and generalization. The structure of GELAN is as follows Figure 5 shown.

[0064] In the present invention, the four C3 modules in the YOLOv5 neck network are replaced by RepNCSPELAN with a GELAN structure in the YOLOv9 network, and its structure is as follows: Figure 6 .

[0065] LPRNet is a lightweight license plate recognition network, essentially a character sequence recognition network designed specifically for license plate recognition. Due to the particularity of the recognition object (most license plates are eight digits or less), it does not use RNN (Recurrent Neural Network) but uses a 1*13 wide convolution instead; in addition, it does not require character cutting in advance, and can output the predicted string by inputting the original license plate image.

[0066] The network structure of LPRNet is relatively simple. It consists of basic two-dimensional convolution, normalization, pooling, splicing and activation operations, such as Figure 7 As shown in the figure, the input image will be uniformly scaled to 94*24 size. The output after the backbone network can be regarded as a sequence representing the probability of the corresponding characters. The features from the shallow network are added to enhance the perception of the entire character context, and finally a probability matrix is ​​obtained. This probability matrix will be used to calculate the loss through CTCloss.

[0067] EMA (Efficient Multi-Scale Attention) is a plug-and-play attention module that avoids the side effects of building cross-channel relationships through channel dimensionality reduction, retains the information on each channel, and aims to reduce computational overhead. Its core idea is to transform some channels into batch dimensions, group channels into multiple sub-features without reducing the channel dimension, and build local cross-channel interactions between subnetworks so that spatial semantic features are evenly distributed in each feature group.

[0068] Specifically, the EMA module divides the input image channels into multiple parts during the feature grouping stage and passes through three parallel sub-networks. Two of the parallel networks perform horizontal pooling and vertical pooling on the grouped image channels respectively, and then use the 1*1 shared convolution operation in CA (Coordinate Attention) proposed in 2021 to fuse the calculation results of the two networks; a 3*3 convolution kernel is used in the third network. The first two 1*1 branches capture features in two spatial directions, and the third 3*3 branch captures multi-scale channel feature relationships. In this way, EMA can not only extract spatial structural information, but also cross-channel information. Next, the two sub-networks (because the two 1*1 networks are aggregated together) undergo average pooling and Softmax activation functions, and then are aggregated through matrix dot products. This is the process of cross-space learning.

[0069] Finally, the feature maps within each feature group are aggregated through the Sigmoid function. The output image of the EMA module has the same size and number of channels as the input image, which makes the EMA module plug-and-play and helps to deploy it more flexibly. The structures of the CA module and the EMA module are as follows: Figure 8 In order to further improve the accuracy of LPRNet for license plate recognition under different colors and lighting conditions.

[0070] The present invention adds the EMA attention mechanism after the two Dropout layers of LPRNet. The improved LPRNet network structure is as follows: Fig. 9 shown.

[0071] In order to verify the performance of the data preprocessing, improved YOLOv5 segmentation network structure and improved LPRNet network structure proposed in the present invention, pictures of gasoline vehicles and new energy vehicles containing license plates were collected in a parking lot in Beijing as the data set of the license plate segmentation network, and the open source Chinese license plate recognition data set CBLPRD was used as the data set of the license plate recognition network.

[0072] For the former, a total of 448 pictures of blue and green license plates taken from different angles and distances were collected, of which the ratio of blue to green license plates was about 9:10, and most of them were Beijing license plates, with a small number of Tianjin and Hebei license plates. After data set expansion and removal of dirty data, a total of 2122 pictures were obtained, of which 1880 were used for training and 242 were used for verification. The data set is annotated into two categories: blue license plates and green license plates. Specific data set expansion methods include adding Gaussian noise, fusing the original image with a completely black or completely white image to change the brightness, small rotation, and horizontal / vertical / diagonal flipping.

[0073] CBLPRD (China Balanced Licence Plate Recognition Dataset) is an open source license plate dataset, which contains about 300,000 license plate images. These images are generated by GAN (Generative Adversarial Network), with excellent quality and relatively balanced types, including ordinary blue plates, new energy small / large vehicles, black plates, single-layer yellow plates, tractor green plates, etc., which to a certain extent avoids the degradation of network performance caused by data set skew. Due to the large number of data sets, we randomly selected them without changing the proportion of data set categories as much as possible, and obtained 139,303 for training and 17,105 for verification.

[0074] In this experiment, the CPU is 9750h, the GPU is 1650 4G, the deep learning framework is Pytorch 2.1, the CUDA version is 12.1, the Python version is 3.8, the YOLOv5 hyperparameter settings training epochs is 200, the batch size is 8, the initial learning rate is 0.005, the cyclic learning rate is 0.01, the learning rate momentum is 0.937, and the weight decay coefficient is 0.0005. The network input image size is 640*640; the LPRNet hyperparameter settings training epochs is 100, the network input image size is 94*24, the initial learning rate is 0.01, the weight decay coefficient is 2×10-5, the learning rate momentum is 0.9, ReduceLROnPlateau is used to adjust the learning rate, and the batch size is 200. Performance evaluation indicators include class average precision mAP (mean Average Precision), parameter quantity (Parameters), 1 billion floating point operations (GFLOPs), frame number (Frames Per Second, FPS) and accuracy (Accuracy). Among them, class average precision mAP includes mAP50 and mAP50-95, which represent the mAP when the IoU (Intersection of Union) threshold is 0.5 and the average value of mAP in the IoU threshold range of 0.5 to 0.95. In addition, box_mAP is suitable for target detection tasks, and mask_mAP is suitable for semantic segmentation tasks. The current semantic segmentation model evaluation indicators also include mIoU (mean Intersection over Union) and mPA (mean Pixel Accuracy); the accuracy is the proportion of samples with completely correct character prediction results to the total number of samples, only for character recognition networks.

[0075] Table 1 shows the experimental results of the improved YOLOv5n segmentation model of the present invention and the original YOLOv5n segmentation model on the Beijing parking lot dataset before and after enhancement. It is necessary to explain that since segmentation is the task goal and the mAP50 values ​​are relatively good, the evaluation results focus on mask_mAP5095. As can be seen from the table, although the YOLOv5n segmentation network model has only over 1M parameters, the floating-point operations are as high as 6.7GFLOPs. The improved YOLOv5n segmentation network model refers to the "low FLOPs trap" idea. Although the number of parameters has skyrocketed, the floating-point operations are greatly reduced to 4.5GFLOPs. The unimproved YOLOv5n segmentation network model has a 1% improvement in the segmentation effect of blue license plates on the Beijing parking lot dataset before and after enhancement, but a slight decrease on green license plates. The overall box_mAP and mask_mAP have increased by 0.7% and 0.4% respectively. It can be seen that enhancing the dataset helps to improve the performance of the segmentation model and makes it easier for the network to learn the characteristics of the dataset, but it is obvious that the CLAHE-GAMMA enhancement method still has the possibility of better parameter adjustment. The YOLOv5n segmentation network model before and after the improvement has a significant improvement on the enhanced background area parking lot dataset. The mask_mAP for blue license plates and green license plates has increased by about 3 and 1 percentage points respectively; although the overall box_mAP has decreased by about 1%, the mask_mAP has increased by 2%, and about 12 points of frames have been lost; it can be seen that the modification of the YOLOv5n segmentation network has a certain improvement in the network's ability to learn features. In general, after the two operations of enhancing the dataset and improving the network, the present invention still improves the segmentation effect of the license plate while maintaining the real-time prediction.

[0076] Table 2 shows the experimental results of the improved LPRNet recognition model of the present invention and the original LPRNet on the CBLPRD dataset. As can be seen from the table, adding the EMA attention mechanism to the LPRNet network effectively improves the accuracy of the recognition network to 9.1%, while the speed is reduced by about 20 frames, but can still be maintained at around 140 frames. This shows that although LPRNet is simple, it still has deeper optimization possibilities and improvement potential. The present invention significantly improves its character recognition ability while maintaining the lightweight and fast characteristics of LPRNet.

[0077] Table 1 YOLOv5n_seg experimental results

[0078]

[0079] Table 2 LPRNet experimental results

[0080] Model Acc FPS LPRNet 0.796 169.492 Improved LPRNet 0.887 142.857

[0081] Ablation experiments were conducted to verify the effects of the improved methods of the present invention on the final results when applied separately. The ablation experiment results of the improved YOLOv5n segmentation network are shown in Table 3, where A, B, and C represent three improvements: replacing part of the CBS of the backbone and neck networks with DynamicConv, combining C3 in the backbone network with DynamicConv, and replacing the neck C3 with GELAN. As can be seen from the table, the "low FLOPs trap" idea of ​​ParameterNet can effectively reduce the amount of floating-point operations while greatly increasing the number of parameters. Whether replacing CBS with DynamicConv or combining DynamicConv with C3 has a certain improvement effect on the network, which increases by 0.7% and 1% in mask_mAP50-95 respectively, and has no significant effect on the number of frames; the effect of applying the GELAN module of YOLOv9 alone is not outstanding, not only does it decrease slightly in box_mAP5095 and mask_mAP5095, but also loses about 10 points of frame number; DynamicConv and C3_ The simultaneous application of DynamicConv achieved good results, with an increase of 0.2% and 2.1% on box_mAP5095 and mask_mAP5095 respectively, and a loss of about 3 frames; when they were applied simultaneously with GELAN, the improvement effect on box_mAP5095 and mask_mAP5095 was not significant, and some even had a small drop in points; when the three improvements were applied simultaneously, while box_mAP5095 decreased by 0.1%, mask_mAP5095 increased by 2.4%, and the frame rate decreased by about 12 frames but still maintained the real-time performance of the prediction. The improved network structure achieved a good balance between accuracy and speed.

[0082] The ablation experiment results of the improved LPRNet recognition network are shown in Table 4. Since the LPRNet network structure is relatively simple, this ablation experiment is actually a comparison of its different improvement effects. D and E represent the addition of EMA after the first and second Dropout layers, respectively. As can be seen from the table, adding EMA to the first Dropout layer can effectively improve the character recognition accuracy of the model, while causing less loss in speed; adding EMA to the second Dropout layer is not as good as the first Dropout layer in improving accuracy, and has a greater impact on speed; adding EMA after two Dropout layers can achieve a greater improvement in accuracy, with a frame loss of about 20 but still maintaining fast inference.

[0083] Table 3 YOLOv5n ablation experiment results

[0084]

[0085] Table 4 LPRNet ablation experiment results

[0086] Model D E Acc FPS LPRNet 0.796 169.492 +D √ 0.869 166.667 +E √ 0.825 151.515 +D+E √ √ 0.887 142.857

[0087] Table 5 lists the experimental results of the comparison between the improved segmentation network model proposed in this invention and the current mainstream semantic segmentation algorithm. Among them, mIoU and mPA are commonly used indicators for evaluating the performance of semantic segmentation network models, representing the average intersection-over-union ratio of each category of predicted results and true labels and the average proportion of correctly classified pixels in each category, respectively. YOLOv5 cannot officially output them, but they can be inferred using the predicted confusion matrix.

[0088] As can be seen from the table, the mIoU value of the improved YOLOv5n_seg model is low and the mPA value is high, indicating that the predicted mask covers the real mask well and contains too many areas outside the real mask. The possible reason is that the target predicted by the present invention is smaller than the image size, and the mask predicted by YOLOv5 is small in scale and has many jagged edges on small targets, resulting in a low mIoU. Compared with YOLOv5n_seg, the improved model has little improvement in mIoU, but has a significant improvement in mPA, while the speed loss is about 10 frames. Compared with other current semantic segmentation algorithms such as Unet and PSPNet, it lags behind by about 10% in mIoU and leads by a lot in mPA.

[0089] Table 5 Experimental results compared with other semantic segmentation algorithms

[0090]

[0091] In addition, Table 6 lists some other attempts to improve the YOLOv5n segmentation network during the experiment of the present invention. Among them, C3_GhostModule is to use DynamicConv to improve the GhostNet proposed in 2021, and replace the Bottleneck in the C3 module, which is similar to the original ParameterNet approach; SCINet is to add the SCINet low-light image enhancement network at the beginning of the backbone network; ADown is to replace all CBS modules with the downsampling module Adown proposed by YOLOv9. It can be seen from the results that the application results of C3_GhostModule are not ideal, and its box_mAP50-95 and mask_mAP50-95 are both reduced by about 1%, and the number of frames is also reduced by about 4 frames; although the application results of SCINet slightly reduce box_mAP50-95, there is more than 1% improvement in mask_mAP50-95, and the number of frames is increased by about 2 points; the application results of Adown are also box_mAP50-95 reduced by about 1%, and mask_mAP50-95 increased by about 1%.

[0092] In general, the result of combining GhostModule with C3 is not as good as the improved version of DynamicConv, SCINet and ADown, but the effect is not as good as the improvement of the present invention, which achieves a good balance between accuracy and speed. In addition, modifying the backbone network of the YOLOv5n segmentation network is very likely to cause a significant drop in accuracy, and modifying the neck network does not cause as serious a drop in points as the backbone network. Whether modifying Concat, CBS or C3, good results will be achieved, but the improvement is often limited.

[0093] Table 6 Experimental results of other improved solutions

[0094]

[0095]

[0096] The present invention proposes a low-light license plate recognition method based on improved YOLOv5-LPRNet for the license plate recognition task in electric vehicle charging places. The process of the method is: CLAHE-GAMMA enhancement of the data set, license plate segmentation using the improved YOLOv5n segmentation network, and license plate recognition using the improved LPRNet.

[0097] For YOLOv5, firstly, the CBS module in the backbone network of YOLOv5 is replaced with DynamicConv and the C3 module in the backbone network is combined with DynamicConv. The idea of ​​"low FLOPs trap" is used to significantly improve the accuracy. Secondly, the C3 module in the neck network of YOLOv5 is replaced with the RepNCSPELAN module proposed by YOLOv9. When the above improvements are applied simultaneously, the network accuracy is improved by about 2%, while the real-time reasoning is maintained. For LPRNet, the EMA attention module is added after its two Dropout layers to improve the network's ability to extract features. The improved model has an accuracy improvement of about 8% and maintains fast reasoning.

[0098] The above two networks were experimented on the Beijing parking lot dataset and the CBLPRD dataset respectively. The improved models performed well on their respective datasets, significantly improving the precision and accuracy while losing a small amount of frames but ensuring the real-time prediction. The improved YOLOv5n segmentation network has an accuracy improvement of about 2% compared with the original network, while losing about 12 frames; the improved LPRNet has an accuracy improvement of about 9% compared with the original network, while losing about 20 frames. The results show that the improved algorithm proposed in the present invention can effectively improve the YOLOv5n segmentation network's ability to extract license plate features and enhance its feature fusion capabilities. In addition, it can effectively improve the accuracy of the LPRNet network in recognizing license plate characters. In comparison with other license plate detection algorithms, good results have been achieved in terms of pixel accuracy or frame number.

[0099] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0100] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for license plate recognition in a low-light environment, characterized in that: It includes the following steps: Step 1: After the license plate image to be identified is enhanced by CLAHE-GAMMA, it is sent to the improved YOLOv5n segmentation network; Step 2: Use the improved YOLOv5n segmentation network for license plate segmentation; Step 3: Extract the four boundary points of the segmented mask and stretch the mask into a rectangle, send it to the improved LPRNet character recognition network, and use the improved LPRNet for license plate recognition.

2. A method for license plate recognition in a low-light environment as claimed in claim 1, characterized in that: The CLAHE-GAMMA enhancement in step 1 includes the following steps: S01: Convert the image from RGB color space to HSV color space, perform CLAHE transformation on V channel only, and convert the image after CLAHE transformation from HSV space back to RGB space; S02: Average the values ​​of the R channel, G channel and B channel of the image converted back to the RGB space and perform adaptive GAMMA transformation.

3. The method for license plate recognition in a low-light environment as claimed in claim 1, characterized in that: The improved YOLOv5n segmentation network in step 1 replaces the CBS module in the YOLOv5 backbone network with DynamicConv, and combines the C3 module in the YOLOv5 backbone network with DynamicConv, using the "low FLOPs trap" idea in the process; then replaces the C3 module in the YOLOv5 neck network with the RepNCSPELAN module proposed by YOLOv9.

4. The method for license plate recognition in a low-light environment as claimed in claim 1, characterized in that: The improved LPRNet character recognition network in step 3 specifically adds an EMA attention module after the Dropout layer of the original LPRNet character recognition network.

5. The method for license plate recognition in a low-light environment as claimed in claim 2, characterized in that: The formula for the CLAHE transformation in S01 is: SCrγ Where s is the output value, c is the gamma compensation coefficient, r is the input value, γ is the correction coefficient, and M is the normalized channel average.

6. A method for license plate recognition in a low-light environment as claimed in claim 2, characterized in that: When performing CLAHE transformation on the V channel in S01, the contrast threshold is set to 5 and the grid size is set to 8*8.

7. The method for license plate recognition in a low-light environment as claimed in claim 5, characterized in that: The input value r needs to be normalized in advance.

Images