An automatic license plate recognition method for hazy weather environment

Abstract

This application provides an automatic license plate recognition method for hazy weather environments. First, a foggy vehicle image in a hazy weather environment is uploaded. Then, a trained hazy image preprocessing module is used to defog the foggy vehicle image. Then, a license plate positioning module is used to locate the license plate. Finally, the located license plate is input into a license plate recognition module for license plate character recognition and output of the recognition result. The automatic license plate recognition method for hazy weather environments provided by this application achieves good defogging effects using a small amount of paired image data. The method is highly adaptable and has a high recognition accuracy for automatic license plate recognition in hazy weather environments, solving the existing problem of low license plate recognition accuracy in hazy weather environments and having broad application prospects.

Description

Technical Field

[0001] The present application belongs to the field of license plate recognition technology, and more specifically, relates to an automatic license plate recognition method for hazy weather environments. Background Art

[0002] With the continuous development of smart transportation and the rapid growth of the number of motor vehicles, license plate recognition technology has become an essential component of smart transportation, playing a vital role in tasks such as parking management, highway toll collection, and traffic violation photography. Existing license plate recognition algorithms can be roughly divided into traditional methods and deep learning-based methods. Traditional license plate recognition algorithms typically use image processing-based algorithms to locate and identify license plates, resulting in low recognition accuracy and limited application in specific scenarios.

[0003] In recent years, with the development of machine learning technology, license plate recognition algorithms based on deep learning models have greatly improved in terms of stability and recognition accuracy. However, due to the complex application scenarios of license plate recognition, current license plate recognition technology still suffers from low recognition accuracy and poor robustness in adverse weather conditions such as smog. Therefore, the low accuracy of license plate recognition in smog weather conditions is a key issue that needs to be addressed urgently. Summary of the Invention

[0004] To achieve the above-mentioned purpose, the technical solution adopted in this application is: to provide an automatic license plate recognition method for haze weather environment, first upload the foggy vehicle image in the haze weather environment, then use the trained haze image preprocessing module to defog the foggy vehicle image, and then use the license plate positioning module to locate the license plate, and finally input the located license plate into the license plate recognition module for license plate character recognition, and output the recognition result.

[0005] Optionally, the following steps are included:

[0006] Step 1, start;

[0007] Step 2: Before the haze image preprocessing module defogs the collected foggy vehicle images, it first fogs the clear, fog-free license plate images to batch generate images similar to vehicle images in real hazy weather environments. These images are paired with the original clear, fog-free license plate images to form a training set, which serves as the input for training the Cycle GAN defogging model.

[0008] Upload foggy vehicle images in haze weather to the haze image preprocessing module, and use the trained CycleGAN defogging model to defog the uploaded foggy vehicle images;

[0009] Step 3: After the image dehazing network is used to remove the interference of haze weather, a clear and fog-free vehicle image is generated, which is then input into the license plate positioning module based on YOLOv5 for license plate positioning.

[0010] Step 4: The license plate positioning module inputs the located license plate into the license plate recognition module, which uses the LPRNet network to perform end-to-end license plate character recognition;

[0011] Step five, end.

[0012] Optionally, in step 2, the Cycle GAN dehazing model includes two mirror-symmetric generative adversarial networks, each of which includes two discriminators and two generators.

[0013] Optionally, a Cycle GAN defogging model is used to transform foggy license plate images into fog-free license plate images;

[0014] Suppose the foggy image is the X domain and the fog-free image is the Y domain;

[0015] Generator 1 is responsible for generating Y-domain images from X-domain images, and Generator 2 is responsible for generating X-domain images from Y-domain images.

[0016] The two discriminators need to determine whether the input image is in the X domain or the Y domain;

[0017] The generator attempts to learn the mapping of the target domain through the original domain images, and the discriminator judges. In the process of confrontation between the generator and the discriminator, the images generated by the generator become closer and closer to the real target domain images.

[0018] Optionally, in order to preserve the similarity of certain features of the target domain image generated by the generator with the original domain image, the generative adversarial network restores the generated fake target domain image back to the original domain and calculates a loss function, where the loss function is an adversarial loss function and a cycle consistency loss function.

[0019] The adversarial loss function is used to learn the mapping relationship between the original domain and the target domain to ensure that the image is mapped from the foggy image to the fog-free image. The adversarial loss function is shown in formula (1):

[0020]

[0021] Where x and y represent images from domain X and domain Y respectively, Gxy is the generator responsible for generating images from domain X to domain Y, and Dy is the discriminator;

[0022] The cycle consistency loss function can ensure that the corresponding features of the original image in the original domain are retained during the image domain conversion process. The cycle consistency loss function is shown in formula (2):

[0023]

[0024] Where x and y represent images from the X domain and Y domain respectively, Gxy is the generator responsible for generating y domain images from the x domain, and Gyx is the generator responsible for generating x domain images from the y domain;

[0025] The complete loss function is the sum of the two loss functions, as shown in formula (3):

[0026] ψ(G xy ,G yx ,D x ,D y )=ψ GAN (G xy ,D y ,X,Y)+ψ GAN (G yx ,D x ,X,Y)+λψ cyc (3)

[0027] Among them, Gxy is the generator responsible for generating y-domain images from x-domain, Gyx is the generator responsible for generating x-domain images from y-domain, Dx and Dy are discriminators, and λ is the weight coefficient.

[0028] Optionally, in step 3, the license plate positioning module uses YOLOv5 to locate the license plate. YOLOv5 includes a Backbone module, a Neck module, and a Head module.

[0029] Backbone module is used for feature extraction;

[0030] The Neck module is used to mix and combine features;

[0031] The Head module is used to predict the output;

[0032] The CSP-Darknet53 backbone module contains five CSP modules. When performing feature extraction, the CSP module not only uses the feature map output by the fifth CSP, but also uses the feature maps output by other CSP layers.

[0033] Optionally, the license plate positioning module includes non-maximum suppression, which first groups all prediction boxes according to different categories, sorts them within the group according to the size of the confidence, selects the prediction box with the largest confidence, and then selects other prediction boxes in turn to determine whether the overlapping area between the prediction box and the selected prediction box is greater than a pre-set threshold. If it is greater than the set threshold, it is determined that the prediction is for the same license plate target and the prediction box is deleted. If the overlapping part is less than the set threshold, it is determined that the prediction is not for the same license plate target and the prediction box is retained. The algorithm is executed in a loop until all prediction boxes are processed.

[0034] Optionally, in step 4, the license plate recognition module includes an STN spatial transformer network and an LPRNet network, the STN spatial transformer network corrects the input license plate image, and the LPRNet network is used for license plate character recognition.

[0035] Optionally, the STN spatial transformer network corrects the input license plate image by first inputting the license plate image cropped by the haze image preprocessing module into the STN spatial transformer network, using the pooling layer to extract image features, passing the extracted features into the fully connected layer, and calculating the transformation parameters; finally, using the transformation parameters to correct the input image, and the corrected image will be used as the input of the LPRNet network.

[0036] Optionally, the LPRNet network takes a 94*24 RGB image as input, and after convolution and feature extraction, outputs a sequence representing the probability of a character;

[0037] CTC loss is used for training to address the problem of inconsistency between the output representative character probability sequence and the license plate character sequence length, and to recognize variable-length character sequences;

[0038] The sequence representing the character probability is decoded, and then the most likely N sequences are found through beam search and post-filtering, and the character sequence that best matches the template is returned for output. The output result is the license plate number sequence.

[0039] The automatic license plate recognition method for hazy weather environments of the present application adds a hazy image preprocessing module on the basis of the traditional license plate recognition algorithm, uses the Cycle GAN defogging model generative adversarial network to remove the interference pixels of hazy weather, and converts the vehicle image under hazy weather into a clear and fog-free image. The module combines the license plate positioning module and the license plate recognition module into one to achieve end-to-end automatic license plate recognition. In terms of image defogging, a good defogging effect can be obtained using a small amount of paired image data. The license plate recognition method proposed in the present application can be better applied to hazy weather scenes. It has strong adaptability and high recognition accuracy for automatic license plate recognition in hazy weather environments, solves the existing problem of low license plate recognition accuracy in hazy weather environments, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is the overall flow chart of the automatic license plate recognition algorithm in hazy weather environment;

[0042] Figure 2 This is a block diagram of the training process of the haze image preprocessing module;

[0043] Figure 3 This is the YOLOv5 network structure diagram;

[0044] Figure 4 This is the non-maximum suppression effect diagram;

[0045] Figure 5 This is the LPRNet network structure diagram. DETAILED DESCRIPTION

[0046] In order to make the technical problems, technical solutions and beneficial effects to be solved by this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] The automatic license plate recognition method for a hazy weather environment provided by an embodiment of the present application is now described. The automatic license plate recognition method for a hazy weather environment first uploads a foggy vehicle image collected in a hazy weather environment, then uses a haze image preprocessing module to defog the collected vehicle image, then uses a license plate positioning module to locate the license plate, and finally inputs the located license plate into a license plate recognition module for license plate character recognition, and outputs the recognition result. Figure 1 , specifically including the following steps:

[0048] Step 1, start;

[0049] Step 2: Collect license plate images under smog and upload foggy vehicle images under smog to the trained smog image preprocessing module.

[0050] The trained haze image preprocessing module is used to defog the uploaded foggy vehicle images, and the Cycle GAN defogging model is used to transform the foggy license plate images into fog-free license plate images.

[0051] Before preprocessing foggy license plate images, the Cycle GAN defogging model is first trained. The training set includes fog-free license plate images collected under normal weather conditions and foggy license plate images after fogging the fog-free license plate images. These images are input into the Cycle GAN defogging model in pairs, allowing the Cycle GAN defogging model to learn the mutual conversion between foggy and fog-free license plate images. Once the Cycle GAN defogging model is trained, it can be put into use to defog the received foggy vehicle images.

[0052] The Cycle GAN dehazing model is trained before preprocessing foggy license plate images. The training data is generated by fogging clear, fog-free license plate images, batch-generating images similar to vehicle images in real foggy weather environments. These images are paired with the original clear, fog-free images to form a training set, which serves as input for the Cycle GAN dehazing model training.

[0053] The Cycle GAN dehazing model consists of two symmetrical generative adversarial networks. Each generative adversarial network consists of two discriminators and two generators. The two discriminators and two generators work together to complete the conversion of two types of image styles.

[0054] The Cycle GAN dehazing model is used to transform foggy license plate images into fog-free license plate images. Figure 2 As shown in the figure, the foggy image is in the X domain and the fog-free image is in the Y domain. Generator 1 is responsible for generating the Y domain image from the X domain image, and Generator 2 is responsible for generating the X domain image from the Y domain image.

[0055] The two discriminators need to determine whether the input image is in the X domain or the Y domain.

[0056] The core idea of ​​the CycleGAN dehazing model proposed in this application is to simultaneously optimize the adversarial loss and cycle consistency loss during the training process, and use a small number of paired samples as the training set. Paired samples can provide constraints during model training, making the distribution generated by the model easier to fit to the source domain samples.

[0057] During the training process, the generator attempts to learn the mapping of the target domain through the original domain images, and the discriminator judges. In the process of confrontation between the generator and the discriminator, the images generated by the generator become closer and closer to the real target domain images.

[0058] In order to keep the target domain image generated by the generator similar to the original domain image in certain features, the generative adversarial network restores the generated target domain image back to the original domain and performs loss calculation, so that the characteristics of the original image can be well preserved during the mapping process.

[0059] The Cycle GAN dehazing model has two main loss functions. The first loss function is the adversarial loss, as shown in formula (1). The purpose of using the adversarial loss is to learn the mapping relationship between the original domain and the target domain, so that the generator can generate images that are closer and closer to the target domain. During training, the two generators continuously learn the mapping process between foggy and fog-free images, while the two discriminators also improve their ability to distinguish between real and fake images during training.

[0060]

[0061] Here, x and y represent images from domain X and domain Y respectively, Gxy is the generator responsible for generating images from domain X to domain Y, and Dy is the discriminator.

[0062] The adversarial loss is mainly used to ensure that the image is mapped from the foggy image to the fog-free image, but it cannot guarantee the correspondence between the two images. Therefore, CycleGAN also adds a cycle consistency loss, which can well ensure that the corresponding features of the original image in the original domain are retained during the image domain conversion process, as shown in formula (2).

[0063]

[0064] Among them, x and y represent images from the X domain and Y domain respectively, Gxy is the generator responsible for generating y domain images from the x domain, and Gyx is the generator responsible for generating x domain images from the y domain.

[0065] The complete loss function is the sum of the two loss functions, as shown in formula (3):

[0066] ψ(G xy ,G yx ,D x ,D y )=ψ GAN (G xy ,D y ,X,Y)+ψ GAN (G yx ,D x ,X,Y)+λψ cyc (3)

[0067] Among them, Gxy is the generator responsible for generating y-domain images from x-domain, Gyx is the generator responsible for generating x-domain images from y-domain, Dx and Dy are discriminators, and λ is the weight coefficient.

[0068] The trained image dehazing network can effectively remove the interference factors of hazy weather and make the picture clearer.

[0069] Furthermore, the model is trained using paired training data and constrained during model optimization. This significantly reduces the number of training samples compared to traditional Cycle GAN dehazing models, resulting in a high recognition accuracy rate with only a small number of training samples. Furthermore, the model is generalizable and versatile, and can be applied to different types of CAPTCHA images without changing the model structure, significantly reducing manual intervention.

[0070] Step 3: After the image dehazing network is used to remove the interference of haze weather, a clear and fog-free vehicle image is generated. This image is input into the license plate positioning module, and YOLOv5 is used for license plate positioning.

[0071] The license plate positioning module uses YOLOv5 for license plate positioning. The network structure of YOLOv5 is as follows: Figure 3 As shown, it mainly includes three parts: Backbone, Neck and Head.

[0072] Backbone is used for feature extraction.

[0073] Neck is used to mix and combine features.

[0074] Head is used to predict output.

[0075] The CSP-Darknet53 backbone module contains 5 CSP modules.

[0076] The CSP module uses not only the feature maps output by the fifth CSP layer but also those from other CSP layers during feature extraction. This improves feature diversity and enhances the learning capabilities of the CNN. This module maintains accuracy while being lightweight, reducing computational bottlenecks and memory costs.

[0077] When using YOLOv5 to locate the license plate, multiple prediction boxes will be generated around the same license plate position. These prediction boxes overlap with each other. These prediction boxes predict the same license plate position, and it is necessary to select the most accurate prediction box from them. Therefore, non-maximum suppression is needed to complete this process. The effect of non-maximum suppression is shown in the figure below. Figure 4 As shown in the figure, non-maximum suppression (NMS) first groups all prediction boxes by category, sorts them within the group by confidence, selects the prediction box with the highest confidence, and then selects other prediction boxes in turn. It determines whether the overlapping area between this prediction box and the selected prediction box is greater than a preset threshold. If it is greater than the threshold, it is determined that the prediction is for the same license plate target and the prediction box is deleted. If the overlapping area is less than the threshold, it is determined that the prediction is for different license plate targets and the prediction box is retained. This algorithm is executed in a loop until all prediction boxes have been processed.

[0078] The license plate positioning module is used to locate the license plate and use it as the input of the license plate recognition module to perform license plate character recognition.

[0079] In step 4, the license plate positioning module inputs the located license plate into the license plate recognition module. The license plate recognition module includes the STN spatial transformer network and the LPRNet network. The STN spatial transformer network corrects the input license plate image, and the LPRNet network is used for license plate character recognition.

[0080] During license plate recognition, the license plate output from the YOLOv5 localization module may exhibit image tilt, hindering further recognition. Therefore, before inputting it into the LPRNet network for license plate recognition, the cropped license plate image is first corrected using the STN spatial transformer network. The STN spatial transformation method is as follows: the license plate image cropped by the haze image preprocessing module is input into the STN spatial transformer network. The pooling layer extracts image features, which are then passed to the fully connected layer to calculate the transformation parameters. The transformation parameters are then used to correct the input image, which serves as the input to the LPRNet network. The STN spatial transformer network structure is shown in Table 1.

[0081] Table 1 STN network structure

[0082]

[0083] The license plate recognition process is implemented by the LPRNet network. The LPRNet network takes a 94*24 RGB image as input. After convolution operation and feature extraction, it outputs a sequence representing the probability of characters, which is related to the pixel width of the input image. And using CTC loss for training can effectively deal with the problem of inconsistency between the output sequence representing the probability of characters and the length of the license plate character sequence, and can well recognize variable-length character sequences. In the inference stage, the sequence representing the probability of characters is decoded, and then the most likely N sequences are found through beam search and post-filtering and the character sequence that best matches the template is returned for output. The output result is the license plate number sequence. The LPRNet network, such as Figure 5 The calculation of CTC loss is a prior art and will not be described here in detail.

[0084] Step five, end.

[0085] The automatic license plate recognition method for hazy weather environments of the present application adds a license plate image preprocessing function on the basis of the traditional license plate recognition algorithm, removes the interference factors of hazy weather, and makes the vehicle image clearer. The method combines the haze image preprocessing module, the license plate positioning module, and the license plate recognition module into one to achieve end-to-end automatic license plate recognition. In terms of image defogging, a good defogging effect can be obtained using a small amount of paired image data. The license plate recognition method proposed in the present application can be better applied to hazy weather scenes. It has strong adaptability and high recognition accuracy for automatic license plate recognition in hazy weather environments, solves the existing problem of low license plate recognition accuracy in hazy weather environments, and has broad application prospects.

[0086] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An automatic license plate recognition method for hazy weather environment, characterized by: First, upload a foggy vehicle image in a hazy weather environment. Then, use the trained haze image preprocessing module to defog the foggy vehicle image. Then, use the license plate positioning module to locate the license plate. Finally, input the located license plate into the license plate recognition module for license plate character recognition and output the recognition result. The following steps are involved: Step 1, start; Step 2: Before the haze image preprocessing module defogs the collected foggy vehicle images, it first fogs the clear, fog-free license plate images to batch generate images similar to vehicle images in real hazy weather environments. These images are paired with the original clear, fog-free license plate images to form a training set, which serves as the input for training the Cycle GAN defogging model. Upload foggy vehicle images in haze weather to the haze image preprocessing module, and use the trained Cycle GAN defogging model to defog the uploaded foggy vehicle images; Step 3: After the image dehazing network is used to remove the interference of haze weather, a clear and fog-free vehicle image is generated, which is then input into the license plate positioning module based on YOLOv5 for license plate positioning. Step 4: The license plate positioning module inputs the located license plate into the license plate recognition module, which uses the LPRNet network to perform end-to-end license plate character recognition; Step 5, end; In step 2, the Cycle GAN dehazing model includes two mirror-symmetric generative adversarial networks, each of which includes two discriminators and two generators; The Cycle GAN dehazing model is used to transform foggy license plate images into fog-free license plate images; Suppose the foggy image is the X domain and the fog-free image is the Y domain; Generator 1 is responsible for generating Y-domain images from X-domain images, and Generator 2 is responsible for generating X-domain images from Y-domain images. The two discriminators need to determine whether the input image is in the X domain or the Y domain; The generator attempts to learn the mapping from the original domain images to the target domain images, and the discriminator makes the judgment. In the process of confrontation between the generator and the discriminator, the images generated by the generator become closer and closer to the real target domain images. In order to keep the target domain images generated by the generator similar to the original domain images in some features, the generative adversarial network restores the generated fake target domain images back to the original domain and calculates the loss function, which is an adversarial loss function and a cycle consistency loss function. The adversarial loss function is used to learn the mapping relationship between the original domain and the target domain to ensure that the image is mapped from the foggy image to the fog-free image. The adversarial loss function is shown in formula (1): Where x and y represent images from domain X and domain Y respectively, Gxy is the generator responsible for generating images from domain X to domain Y, and Dy is the discriminator; The cycle consistency loss function can ensure that the corresponding features of the original image in the original domain are retained during the image domain conversion process. The cycle consistency loss function is shown in formula (2): Where x and y represent images from the X domain and Y domain respectively, Gxy is the generator responsible for generating y domain images from the x domain, and Gyx is the generator responsible for generating x domain images from the y domain; The complete loss function is the sum of the two loss functions, as shown in formula (3): ψ(G xy ,G yx ,D x ,D y )=ψ GAN (G xy ,D y ,X,Y)+ψ GAN (G yx ,D x ,X,Y)+λψ cyc (3) Among them, Gxy is the generator responsible for generating y-domain images from x-domain, Gyx is the generator responsible for generating x-domain images from y-domain, Dx and Dy are discriminators, and λ is the weight coefficient.

2. The automatic license plate recognition method for a hazy weather environment according to claim 1, characterized in that: In step 3, the license plate positioning module uses YOLOv5 to locate the license plate. YOLOv5 includes Backbone module, Neck module, and Head module. Backbone module is used for feature extraction; The Neck module is used to mix and combine features; The Head module is used to predict the output; The CSP-Darknet53 backbone module contains five CSP modules. When performing feature extraction, the CSP module not only uses the feature map output by the fifth CSP, but also uses the feature maps output by other CSP layers.

3. The automatic license plate recognition method for a hazy weather environment according to claim 2, characterized in that: The license plate positioning module includes non-maximum suppression. Non-maximum suppression first groups all prediction frames according to different categories, sorts them within the group according to the size of the confidence, selects the prediction frame with the largest confidence, and then selects other prediction frames in turn to determine whether the overlapping area between the prediction frame and the selected prediction frame is greater than a preset threshold. If it is greater than the set threshold, it is determined that the prediction is the same license plate target and the prediction frame is deleted. If the overlapping part is less than the set threshold, it is determined that the prediction is not the same license plate target and the prediction frame is retained. This judgment method is executed in a loop until all prediction frames are processed.

4. The automatic license plate recognition method for a hazy weather environment according to claim 1, characterized in that: In step 4, the license plate recognition module includes the STN spatial transformer network and the LPRNet network. The STN spatial transformer network corrects the input license plate image, and the LPRNet network is used for license plate character recognition.

5. The automatic license plate recognition method for a hazy weather environment according to claim 4, characterized in that: The STN spatial transformer network corrects the input license plate image by first inputting the license plate image cropped by the haze image preprocessing module into the STN spatial transformer network, using the pooling layer to extract image features, passing the extracted features into the fully connected layer, and calculating the transformation parameters; finally, using the transformation parameters to correct the input image, and the corrected image will serve as the input of the LPRNet network.

6. The automatic license plate recognition method for a hazy weather environment according to claim 5, characterized in that: The LPRNet network takes a 94*24 RGB image as input, and after convolution operation and feature extraction, outputs a sequence representing the probability of a character; CTC loss is used for training to address the problem of inconsistency between the output representative character probability sequence and the license plate character sequence length, and to recognize variable-length character sequences; The sequence representing the character probability is decoded, and then the most likely N sequences are found through beam search and post-filtering, and the character sequence that best matches the template is returned for output. The output result is the license plate number sequence.

Images