A method for recognizing a motion-blurred license plate based on deep learning

Abstract

The application relates to a motion blur license plate recognition method based on deep learning, and relates to a motion blur license plate recognition method. The application aims to solve the technical problems of insufficient recognition accuracy, missed detection and poor real-time performance in processing motion blur images in a dynamic traffic scene. The application combines an improved YOLOv8-KG detection algorithm with a DeblurGAN-ES deblurring algorithm. The YOLOv8-KG improves the detection and positioning accuracy of the blurred license plate through AKconv variable convolution kernel, V-GSCSP structure, convolution block attention mechanism and GIoU loss function; the DeblurGAN-ES introduces an EMA attention mechanism on the basis of DeblurGANv2 and adopts a ShuffleNetV2 backbone network to enhance the license plate edge and detail reconstruction capability and improve the real-time performance.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation systems and computer vision, specifically to a method for recognizing motion-blurred license plates based on deep learning. Background Technology

[0002] While license plate recognition technology has matured and can meet the requirements of simple scenarios, it still faces numerous technical challenges when conducting video evidence collection in dynamic traffic environments. Especially under conditions of high traffic density and high vehicle speeds, the captured video images are often affected by various adverse factors, such as changes in atmospheric transmittance, low-light environmental interference, and limitations in optical sensor performance, leading to significant image quality degradation. Motion blur is one of the main problems causing decreased image clarity, severely restricting license plate recognition efficiency and directly causing a series of practical operational pain points, such as: reducing the efficiency of traffic violation evidence collection (requiring significant manpower for secondary verification), affecting the passage speed of highway toll lanes, and causing inaccurate traffic flow statistics.

[0003] To address the aforementioned issues, while existing technologies have proposed several object detection and image deblurring algorithms, they still have limitations when processing blurred license plate images in highly dynamic traffic scenarios. Firstly, general license plate detection algorithms suffer from low recognition accuracy and missed detections when dealing with multiple blurred license plates. Secondly, general image deblurring algorithms have poor real-time performance and cannot effectively capture global structural information and local texture details in license plate images, making it difficult to accurately restore the details of the license plate characters.

[0004] Therefore, there is a need for a license plate recognition technology that has high precision, multi-target detection and recognition capabilities, can adapt to complex dynamic environments, and has good real-time performance and robustness, in order to improve the practicality and intelligence level of dynamic video evidence collection. Summary of the Invention

[0005] This invention aims to address the technical problems of insufficient recognition accuracy, missed detections, and poor real-time performance in processing motion-blurred images in dynamic traffic scenarios, and provides a deep learning-based method for recognizing motion-blurred license plates.

[0006] The deep learning-based method for recognizing motion-blurred license plates of the present invention employs a two-stage processing flow, including:

[0007] I. Precise localization steps for motion-blurred license plates: A blurred license plate detection model, YOLOv8-KG, optimized to adapt to motion blur features, is used to detect license plates in the image, and the license plate region is cropped and extracted.

[0008] II. Image sharpening steps for the target region: Based on the license plate region extracted in step one, input it into a DeblurGAN-ES model specifically designed for restoring blurred details. Perform targeted deblurring on this region to generate a sharpened license plate image, providing reliable input for subsequent character recognition.

[0009] Preferably, to improve the performance of the fuzzy license plate detection model, this invention makes several structural improvements based on the YOLOv8 model, resulting in the fuzzy license plate detection model YOLOv8-KG. The specific improvement methods are as follows:

[0010] Backbone network: AKconv is used to replace the four convolutional layers in the backbone network to adaptively match the multi-scale and irregular deformation features caused by motion blur, so as to achieve more efficient and flexible sampling.

[0011] Neck network: Introducing the V-GSCSP structure, which effectively enhances the cross-layer feature fusion capability while significantly reducing the computational complexity of the model;

[0012] Attention mechanism: Convolutional block attention mechanism (CBAM) is introduced into key nodes of the network to improve the model's ability to perceive small, blurry license plates;

[0013] Loss function: The GIoU loss function is used for training to reduce harmful gradients in sample quality and improve the localization accuracy of motion-blurred license plate targets with unclear boundaries.

[0014] Preferably, to improve the deblurring effect and running efficiency of the license plate deblurring model, this invention optimizes the license plate deblurring model DeblurGAN to obtain the license plate deblurring model DeblurGAN-ES. The specific optimization method is as follows:

[0015] The DeblurGAN-ES model uses ShuffleNetV2 as a lightweight backbone network to balance computational complexity with image reconstruction quality. The EMA attention mechanism is integrated into both the generator and discriminator to enhance the model's reconstruction and recognition capabilities in terms of license plate character edges, contour clarity, and background texture details.

[0016] More preferably, the backbone network of the generator outputs at least three feature maps at different scales during the feature extraction process, and sets an EMA attention module on the output path of each scale feature map to enhance the information transmission and interaction between local and global features, thereby more accurately restoring the character strokes and edge details of the license plate.

[0017] More preferably, the discriminator integrates at least one EMA attention module after the deep convolutional block to improve the ability to distinguish the authenticity of overall and local feature images.

[0018] More preferably, the training process of the generator is constrained by a composite loss function, which works synergistically to ensure the quality of the generated image. This function includes: adversarial loss based on the WGAN-GP framework to improve training stability and ensure the quality of image generation; perceptual loss based on the VGG19 network to ensure high-level semantic consistency between content and structure; and edge loss based on the Sobel operator to enhance the recovery of the clarity of license plate characters and border contours.

[0019] The two models described above are used in a user management module, a license plate detection module, a license plate deblurring module, and a license plate recognition module. The specific steps are as follows:

[0020] User management module: used to verify user identity and manage user-uploaded images;

[0021] License plate detection module: calls the YOLOv8-KG model, receives images uploaded by users, detects the license plate region in the image, crops and extracts the license plate region, and outputs the cropped license plate image information;

[0022] License plate deblurring module: Receives license plate image data cropped from the license plate detection module, calls the DeblurGAN-ES model to deblur it, and outputs a clear license plate image;

[0023] License plate recognition module: Receives the cleared license plate image and calls the LPR-Net algorithm to perform character recognition, outputting the final text recognition result.

[0024] This invention addresses the problems of low accuracy and poor applicability in vehicle license plate localization and recognition in motion-blurred images by proposing an improved motion-blurred vehicle license plate localization algorithm, YOLOv8-KG, based on YOLOv8n. First, the YOLOv8 backbone network is reconstructed using AKConv to overcome the limitations of fixed-size standard convolutional kernels, and an adaptive variable convolutional kernel is designed to achieve more flexible and efficient convolutional sampling. Second, to improve the feature extraction efficiency of the neck structure, a V-GSCSP structure is introduced to segment and concatenate SConv and DSConv, effectively reducing computation and parameter count while retaining the feature extraction capability of convolutional operations. Third, to enhance the model's ability to detect small targets, a CBAM attention mechanism is introduced to adaptively strengthen channels related to license plate detection, thereby improving the algorithm's sensitivity and accuracy. Finally, the GIoU loss function is used to suppress the harmful gradient effects of low-quality samples, effectively accelerating the model's convergence speed. Experimental results show that the improved YOLOv8-KG model achieves an accuracy of 92.9%, a recall of 75.1%, and a mean precision of 86%, respectively. Compared with the YOLOv8n model, the number of parameters is reduced by 11.9%, the computational cost is reduced by 15%, and the accuracy, recall, and mean precision are improved by 4.3%, 5.7%, and 7.9%, respectively.

[0025] The DeblurGAN-ES deblurring algorithm proposed in this invention shows significant advantages over existing mainstream deblurring algorithms in terms of industry-recognized objective evaluation indicators such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as well as subjective visual effects. It can effectively remove artifacts caused by motion blur, reduce the blurring effect of character edges, and accurately restore license plate details.

[0026] Experimental data show that when the YOLOv8-KG detection model proposed in this invention is combined with the DeblurGAN-ES deblurring model, the recognition accuracy is improved by 4.5% and the average recognition time is shortened by 16.6% compared with the recognition method using the existing YOLOv8 and DeblurGANv2 models. The method of this invention significantly improves recognition accuracy and real-time performance. Attached Figure Description

[0027] Figure 1 This is a diagram illustrating the overall technical framework of the motion fuzzy license plate recognition method of the present invention.

[0028] Figure 2 This is a diagram of the YOLOv8-KG algorithm structure in the embodiment;

[0029] Figure 3 This is a structural diagram of AKConv in the embodiment;

[0030] Figure 4 This is a diagram of the V-GSCSP structure in the embodiment;

[0031] Figure 5 This is a schematic diagram of the channel attention module in the embodiment;

[0032] Figure 6 This is a schematic diagram of the spatial attention module in the embodiment;

[0033] Figure 7 This is a comparison chart of detection results in the examples;

[0034] Figure 8 This is a schematic diagram of the improved generator network structure in the embodiment;

[0035] Figure 9 This is a schematic diagram of the improved discriminator structure in the embodiment;

[0036] Figure 10 A schematic diagram illustrating the calculation of perceived loss in the embodiment;

[0037] Figure 11 This is a comparison image of the restored images on the GOPRO dataset in the example;

[0038] Figure 12 The images show the deblurring results of different algorithms on the GOPRO dataset in the examples.

[0039] Figure 13 This is a diagram showing the deblurring effect of a blurred license plate in the example. Detailed Implementation

[0040] Specific Implementation Method 1: This implementation method is a motion-blurred license plate recognition method based on deep learning, specifically employing a two-stage processing flow, including:

[0041] I. Precise localization steps for motion-blurred license plates: A blurred license plate detection model, YOLOv8-KG, optimized to adapt to motion blur features, is used to detect license plates in the image, and the license plate region is cropped and extracted.

[0042] II. Image sharpening steps for the target region: Based on the license plate region extracted in step one, input it into a DeblurGAN-ES model specifically designed for restoring blurred details. Perform targeted deblurring on this region to generate a sharpened license plate image, providing reliable input for subsequent character recognition.

[0043] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the YOLOv8-KG fuzzy license plate detection model described in step one uses AKconv instead of the four convolutional layers in the YOLOv8 backbone network to adaptively match the multi-scale and irregular deformation features caused by motion blur, achieving more efficient and flexible sampling. Everything else is the same as in Specific Implementation Method One.

[0044] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the fuzzy license plate detection model YOLOv8-KG described in step one introduces a V-GSCSP structure into the neck network of the YOLOv8 model. This effectively enhances the cross-layer feature fusion capability while significantly reducing the computational complexity of the model. Everything else is the same as in Specific Implementation Method One or Two.

[0045] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the YOLOv8-KG fuzzy license plate detection model described in step one introduces a Convolutional Block Attention (CBAM) mechanism at the key nodes of the YOLOv8 network to improve the model's ability to perceive small, fuzzy license plates. Everything else is the same as in Specific Implementation Methods One to Three.

[0046] Specific Implementation Method Five: This implementation method differs from Specific Implementation Method Four in that the YOLOv8-KG fuzzy license plate detection model described in step one is trained using the GIoU loss function to reduce harmful gradients in sample quality and improve the localization accuracy for motion-fuzzy license plate targets with unclear boundaries. Everything else is the same as in Specific Implementation Method Four.

[0047] Specific Implementation Method Six: This implementation method differs from Specific Implementation Method One in that the license plate de-motion blurring model DeblurGAN-ES described in step two is an optimization based on the DeblurGAN model. ShuffleNetV2 is selected as the lightweight backbone network, balancing computational complexity with image reconstruction quality in a coordinated optimization process. EMA attention mechanisms are integrated into both the generator and discriminator to enhance the model's reconstruction and recognition capabilities in terms of license plate character edges, contour clarity, and background texture details. Everything else is the same as in Specific Implementation Method One.

[0048] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Method Six in that the backbone network of the generator outputs at least three feature maps at different scales during the feature extraction process. An EMA attention module is set on the output path of each scale feature map to enhance information transmission and interaction between local and global features, thereby more precisely restoring the character strokes and edge details of the license plate. Everything else is the same as in Specific Implementation Method Six.

[0049] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Method Seven in that the discriminator integrates at least one EMA attention module after the deep convolutional block, improving the ability to discriminate the authenticity of overall and local feature images. Everything else is the same as in Specific Implementation Method Seven.

[0050] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Method Eight in that the generator training process employs a composite loss function for constraint. This function works synergistically to ensure the quality of the generated images. It includes: adversarial loss based on the WGAN-GP framework to improve training stability and ensure the quality of generated images; perceptual loss based on the VGG19 network to ensure high-level semantic consistency between content and structure; and edge loss based on the Sobel operator to enhance the recovery of the sharpness of license plate characters and border contours. Everything else is the same as in Specific Implementation Method Eight.

[0051] The present invention is verified using the following examples:

[0052] This embodiment provides a motion-blurred license plate recognition method based on deep learning. The core of this method consists of two parts:

[0053] One is a fuzzy license plate detection algorithm based on YOLOv8 (hereinafter referred to as "YOLOv8-KG");

[0054] The second is a license plate de-motion blurring algorithm based on DeblurGANv2 (hereinafter referred to as "DeblurGAN-ES").

[0055] The following is a detailed description of each component.

[0056] 1. YOLOv8-KG Algorithm

[0057] See appendix Figure 2 This invention proposes YOLO-KG, which optimizes YOLOv8n in several ways: It introduces a variable convolutional kernel (AKconv) to reconstruct the backbone network, avoiding the limitation of standard convolution's inability to change kernel size, thus achieving more efficient and flexible sampling; it introduces V-GSCSP structure segmentation and splicing SConv (Standard Convolution) and DSConv (Depthwise Separable Convolution) in the neck network, retaining the powerful feature extraction capabilities of standard convolution while reducing computation and parameter count using depthwise convolution; it introduces the CBAM attention mechanism to adaptively enhance channels related to license plate detection, improving the algorithm's sensitivity and accuracy; finally, it uses the GIoU loss function to optimize bounding box regression, reducing gradient interference from low-quality samples, accelerating model convergence, and improving detection accuracy.

[0058] See appendix Figure 3 The backbone network uses AKConv to replace standard convolutions. The specific workflow of AKConv is as follows:

[0059] Step 1: Image Input: Input an image with dimensions (C, H, W); C is the number of channels, H is the image height, and W is the image width.

[0060] Step 2, Initial Sampling Shape: This process first generates a regular sampling grid, defining its top-left sampling point as the sampling origin (0, 0); then, the remaining sampling points are arranged in the next row of the regular sampling grid, and finally, the two are joined to form a complete sampling grid. This serves as the initial relative position of each sampling point during the convolution operation. At any spatial location... The defined convolution operation, with the relevant formulas, is as follows:

[0061]

[0062] in, P represents the convolution parameters; P0 is the initial coordinate, P... n It is the offset coordinate. ( ) is the feature function of the offset image;

[0063] Step 3, Convolution Operation: Perform standard Conv2d operation on the input image to obtain the initial feature map;

[0064] Step 4: Offset Learning: Obtain the offsets learned by the corresponding convolution kernel during the convolution operation. The training yields offsets with dimensions (B, 2N, H, W), where B is the batch size, N is the kernel size, H is the image height, and W is the image width. The offsets are then compared with the original coordinates. Summing yields the corrected coordinates;

[0065] Step 5: Feature resampling. Based on the corrected sampling coordinates, interpolate and resample the feature map to obtain the features corresponding to the irregular sampling shape.

[0066] Step 6, Feature Extraction: Stack the features obtained in Step 5 in the row or column direction, and perform row convolution or column convolution operations using convolution kernels of appropriate size and stride to extract features adapted to irregular sampling structures.

[0067] Step 7, Output Processing: The feature map from Step 7 is reshaped, convolved again, and normalized. The final output result is then generated using the SiLU activation function.

[0068] See appendix Figure 4 The V-GSCSP structure is used to divide and splice SConv and DSConv; the V-GSCSP module flow is as follows:

[0069] Step 1: Input Processing. The input feature map has C1 channels and is first processed through a standard convolutional (Conv) layer for initial feature transformation.

[0070] Step 2: Branch Path Construction. First Branch: Feature map input to the GS bottleneck module, as shown in the attached diagram. Figure 4 The GS bottleneck module shown consists of a main path containing two GSConv layers and a side path containing a standard convolutional layer. The outputs of the two paths are fused through residual connections. The second branch: the feature map directly retains and passes the key information of the original features through another standard convolution (Conv).

[0071] Step 3: Feature Concatenation. The output feature maps from the first and second branches are concatenated along the channel dimension to generate an intermediate feature map.

[0072] Step 4: Output Generation. The intermediate feature maps are integrated and dimensionality-reduced through a final standard convolutional layer to obtain the final output feature map with C2 channels;

[0073] The core building block of the V-GSCSP structure is the GSConv module, and its specific process is as follows:

[0074] Step 1: Initial Convolution. Input the feature map with C1 channels into the Conv convolutional layer, and adjust the number of channels to C2 / 2;

[0075] Step 2, Parallel Convolution. The feature map output from Step 1 is divided into two parts: one part is kept directly; the other part is fed into a depthwise separable convolutional layer (DWConv), where convolution operations are performed independently on each channel.

[0076] Step 3: Feature Concatenation. The features processed by DWConv are concatenated with the directly retained features using the Concat operation.

[0077] Step 4: Channel Shuffle. The concatenated features undergo a shuffle operation to shuffle the channel order, allowing local feature information to be evenly exchanged across different channels. This enables information from standard convolutions to be mixed into the output of depthwise separable convolutions, ultimately outputting a feature map with C2 channels.

[0078] The algorithm of this invention introduces the CBAM (Convolutional Block Attention Module) attention mechanism, which is divided into a channel attention submodule and a spatial attention submodule.

[0079] See appendix Figure 5The channel attention module process takes a feature map F of size C×H×W as input, and performs global average pooling and global max pooling on the feature map to obtain two channel-level vectors. and The two vectors are input into a shared multilayer perceptron, compressed to r dimensions (dimensionality reduction factor), and then restored to c dimensions. The two outputs are added element-wise, and then channel attention weights are obtained by sigmoid activation. The original feature map is multiplied by the channel weights to output the feature map. The relevant formulas are as follows:

[0080]

[0081]

[0082] in, It is the Sigmoid function. , , This is element-wise multiplication, where R is the set of real numbers.

[0083] See appendix Figure 6 The feature maps processed by the channel attention module are subjected to global average pooling and global max pooling in the channel dimension, respectively, to generate two cross-channel two-dimensional feature maps. and The two are concatenated to obtain a 2×H×W feature map; a single-channel feature map is generated through a 7×7 convolutional layer; spatial attention weights are obtained through a sigmoid activation function; and the spatial weights are then combined with the feature map. Multiply the corresponding pixels to generate an attention feature map. .

[0084] The relevant formulas are as follows:

[0085]

[0086]

[0087] in, For the sigmoid function, This represents a convolution with a 7×7 kernel, where... This is element-wise multiplication. This is the final feature map output.

[0088] This invention uses GIoU as the regression loss function. Compared with the traditional IoU loss function, GIoU introduces a penalty term, which is applied to the predicted bounding box. With real frame Even when the edges are completely disjoint (i.e., IoU=0), effective gradients can still be generated. This solves the gradient vanishing problem of IoU loss in this case, thus providing a more stable and effective convergence direction for model training. The formula is as follows:

[0089]

[0090] The formula for GIoU as the bounding box regression loss function is as follows:

[0091]

[0092] A and B represent the areas of the predicted bounding box and the ground truth bounding box, respectively; IoU represents the intersection-over-union ratio of the predicted bounding box A and the ground truth bounding box B, which is calculated by dividing the intersection area of ​​A and B by the union area. GIoU represents the area of ​​the smallest rectangle containing A and B; the value of GIoU ranges from [-1, 1]. When the predicted box and the ground truth box completely overlap, the value of GIoU is 1, and when the two boxes have no intersection and are infinitely far apart, the value is -1.

[0093] To demonstrate the performance advantages of the YOLOv8-KG algorithm proposed in this invention, this embodiment designs and conducts a series of simulation comparison experiments to verify its effectiveness in practical applications.

[0094] The experimental data used in this embodiment comes from a combination of actual data collection and publicly available datasets. First, images were collected using a Sony RX10 camera on the section of Hexing Road in Xiangfang District, Harbin City, in front of the Northeast Forestry University. The data collection period was from April 18th to April 20th, 2024, during the peak traffic hours from 16:30 to 17:30. A total of 6397 motion-blurred license plate images were collected, with a resolution of 1920×1080, in JPG format, covering license plates of small cars, large cars, and new energy vehicles.

[0095] The acquired images were manually annotated using the Labelme annotation tool, and the annotation results were saved in YOLO format. Due to the relatively uniform shooting angles and license plate categories of the acquired data, this embodiment introduces the China City License Plate Public Dataset (CCPD) to improve data diversity and generalization ability. The dataset includes two versions, CCPD2019 and CCPD2020, covering license plate images under nine different acquisition conditions.

[0096] The collected data was merged with the CCPD dataset, resulting in a total of 15,797 blurred license plate images, including 9,799 images in the training set, 2,998 images in the validation set, and 3,000 images in the test set.

[0097] The experimental environment is shown in Table 1, and the experimental parameters are shown in Table 2. This embodiment is based on the PyTorch deep learning framework and utilizes an NVIDIA RTX 4090 GPU for training and testing.

[0098] Table 1 Model Training Environment

[0099]

[0100] Table 2 Experimental parameters

[0101]

[0102] In this embodiment and comparative experiments, mean precision (AP), mean average precision (mAP), recall (R), precision (P), frames per second (FPS), number of parameters, and floating-point operations (GFLOPs) were used as performance evaluation metrics. Here, mAP@0.5 represents the mean average precision calculated with an intersection-over-union (IoU) threshold of 0.5, meaning that a detection is considered correct only if the overlap between the predicted bounding box and the ground truth bounding box exceeds 50%.

[0103] The improved YOLOv8-KG model based on YOLOv8 was compared with mainstream object detection network models such as SSD, Fast-RCNN, YOLOv5s, and YOLOv7-tiny. The experimental results are shown in Table 3.

[0104] Table 3 Comparison of experimental results for different networks

[0105]

[0106] According to the results in Table 3, YOLOv8-KG has the highest precision, recall, and mean precision, while having the lowest number of parameters and computational cost, indicating that YOLOv8-KG outperforms other comparative models.

[0107] To verify the impact of the improved modules on detection performance, five ablation experiments were conducted based on the YOLOv8 algorithm, introducing the AKConv module, V-GSCSP module, CBAM attention mechanism, and GIoU loss function in sequence. The experimental results are shown in Table 4.

[0108] Table 4 Ablation Experiment Results

[0109]

[0110] As shown in Table 4, the experimental results reveal that by gradually introducing AKConv, V-GSCSP, CBAM, and GIoU modules into the YOLOv8 baseline model, the core performance indicator mAP@0.5 continuously improved, ultimately increasing from 78.1% to 86.0%, a cumulative increase of 7.9 percentage points. This demonstrates that the proposed improvement strategy effectively enhances the detection accuracy of motion-blurred license plate targets. Notably, this accuracy improvement did not lead to an increase in computational overhead; on the contrary, the number of parameters and computational cost decreased after introducing AKConv and V-GSCSP. The final YOLOv8-KG model achieves the highest accuracy while maintaining a relatively low level of parameter and computational cost compared to other combinations. Therefore, the improvement strategy proposed in this embodiment achieves simultaneous optimization in detection accuracy and computational efficiency, exhibiting significant comprehensive technical advantages.

[0111] The YOLOv8n and YOLOv8-KG algorithms were visually verified to demonstrate their effectiveness.

[0112] From the appendix Figure 7 As shown in Table 5, YOLOv8n exhibits both false positives and false negatives under different conditions: the first group experienced false negatives due to the target being located at the detection edge and the vehicle speed being relatively high; the second and third groups experienced false negatives due to the small size of the target and low contrast leading to loss of image details; and the fourth group experienced false positives due to interference from the color and shape of the sign. In contrast, YOLOv8-KG accurately detected targets under the same conditions, significantly reducing false negatives and false negatives, and improving the reliability and accuracy of detection.

[0113] Table 5 Analysis of Test Results

[0114]

[0115] 2. DeblurGAN-ES algorithm

[0116] See appendix Figure 8 This embodiment provides a deeply improved generator network structure, which incorporates the following two key innovative improvements aimed at enhancing performance and efficiency based on the DeblurGAN-v2 generator:

[0117] The first key improvement lies in replacing the feature extraction backbone of the network with the ShuffleNetV2 network. ShuffleNetV2 is known for its unique channel shuffling and grouped convolution mechanism. Using this network as the backbone aims to reduce the computational complexity and number of parameters of the model while ensuring model performance, thereby significantly improving the overall speed of image deblurring processing.

[0118] The second key improvement lies in embedding the EMA (Efficient Multi-Scale Attention) module into the encoder path of the feature pyramid in the ShuffleNetV2 backbone network from the bottom up. Specifically, during the feature extraction process at each level of the backbone network, the EMA attention module is embedded sequentially after the feature maps of the 32, 64, 192, and 1088 channels.

[0119] The core advantage of the EMA module lies in its ability to construct an efficient cross-channel spatial interaction mechanism through parameter-sharing convolutions, enabling dynamic feature calibration of the feature map. This mechanism adaptively enhances key features crucial for deblurring tasks (such as object edges and textures) while suppressing unnecessary or interfering features, thereby effectively enhancing the generator's detail reconstruction capabilities.

[0120] More importantly, this design, which embeds the EMA module after multi-scale feature maps, effectively addresses the inherent deficiency of the original DeblurGAN-v2 generator in lacking effective interaction between local and global information. By applying attention weighting at different scales, it enhances the information transfer and interaction between local and global features, helping to better recover details and structure in the image and generate more realistic deblurred images.

[0121] Furthermore, the feature pyramid improved by the EMA module can perform differentiated receptive field adjustments during the bottom-up feature extraction process. This enables the network to extract and process blurred features at different scales more effectively through equivalent dynamic convolution kernels, significantly improving its ability to recover non-uniformly blurred images.

[0122] In addition to the core improvements mentioned above, the overall architecture of the generator still follows the encoder-decoder paradigm shown in the attached figure, and retains structures such as skip connections, multi-scale feature fusion modules, and global residual connections to ensure high-quality image reconstruction.

[0123] See appendix Figure 9 The DeblurGAN-ES algorithm employs a dual-scale discriminator structure, consisting of a global discriminator and local discriminators. The global discriminator takes the complete image as input, passing it through three convolutional blocks (increasing the channel count from 256 to 512) and a combination module of two convolutional blocks and an EMA attention mechanism, outputting a global discrimination result used to calculate the global adversarial loss. The local discriminator takes a randomly cropped sub-image from the complete image as input, using the same structure and process as the global discriminator, outputting a local discrimination result, and calculating the local adversarial loss accordingly. The global discriminator constrains the overall consistency of the generated images, while the local discriminators enhance the realism of details. Finally, the discriminator outputs a numerical value representing the true probability, providing an adversarial training signal.

[0124] The generator loss function of this invention consists of adversarial loss, content loss, and edge loss.

[0125] The formulas related to the generator's total loss function are shown below:

[0126]

[0127] in, This represents the total loss function of the network; , and These represent adversarial loss, content loss, and edge loss, respectively. , and These are the weighting coefficients for the three factors. This invention sets the weighting coefficients as follows: , , .

[0128] The adversarial loss employs WGAN-GP, which improves training stability and generation quality through a gradient penalty mechanism. The generator adversarial loss expression is shown below:

[0129]

[0130] in, This indicates a blurred image.

[0131] See appendix Figure 10 This invention uses perceptual loss as content loss, and selects the output feature map of the conv3_3 layer in the VGG19 network for calculation. The perceptual loss calculation steps are as follows:

[0132] Step 1: Input blurred image The image is fed into the generator to obtain the deblurred image. ;

[0133] Step 2: Generate the image With real images They are then fed into the pre-trained VGG19 network;

[0134] Step 3: Extract the output feature maps of the two sets of images in the intermediate layer (conv3_3) of VGG19;

[0135] Step 4: Calculate the mean square error between the two sets of feature maps, which is the final perceptual loss.

[0136] The edge loss function uses the Sobel operator, and its mathematical expression is as follows:

[0137]

[0138] in, Indicates the width of the feature map. Indicates the height of the feature map; This represents the Sobel operator, used to extract edge features from an image.

[0139] This invention selects WGAN-GP with gradient penalty as the loss function, as shown in the following formula:

[0140]

[0141] in, It is a hyperparameter used to control the strength of the gradient penalty term; in this invention, it is set to 10. These are real samples. It is the actual data distribution; It is random noise input to the generator. This represents the distribution of the input random noise; Represents real data samples and generate data samples The interpolated samples obtained by random interpolation between them, i.e. ,in for Uniformly distributed random numbers, It is the data distribution after interpolation. The generator produces the generated samples. Indicates the discriminator's decision The probability of a true sample. Indicates the discriminator's decision The probability of a true sample. This indicates that the discriminator evaluates the interpolated samples. The output, It is the discriminator's interpolation of samples The gradient.

[0142] This embodiment aims to comprehensively verify the effectiveness of the DeblurGAN-ES algorithm of this invention. The experiment is based on the publicly available GOPRO dataset and a self-built blurred license plate dataset. The GOPRO dataset contains 3124 sets of clear images and their corresponding blurred images, each with a resolution of 1280×720. In the experiment, these images are divided into two parts: one part, consisting of 2013 pairs, is used as the training set, and the remaining 1111 pairs are used to evaluate the model.

[0143] This embodiment utilizes the PyTorch framework for network construction and conducts various experiments in a cloud server environment with an NVIDIA GeForce RTX 4090 graphics card. The model training environment is shown in Table 1. The training parameters are set as follows: initial learning rate 1×10⁻⁶. -4The optimization method used was Radam, with a batch size of 10 and 3000 iterations. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were selected as the quantitative evaluation metrics.

[0144] To verify the advancement of the algorithm of this invention, the DeblurGAN-ES algorithm model was compared with the Gong, DeepDeblur, DeblurGAN, and DeblurGAN-v2 algorithms on the GOPRO dataset. The quantitative experimental results are shown in Table 6.

[0145] Table 6. Quantitative comparison results of different algorithms on the GOPRO dataset

[0146]

[0147] As shown in Table 6, DeblurGAN-ES performs best on the GOPRO dataset, with a PSNR of 29.92dB and an SSIM of 0.941, which are 2.1% and 2.9% higher than DeblurGAN-v2, respectively. This indicates that the method described above has significant improvements in both overall consistency and detail recovery.

[0148] The qualitative comparison results are attached. Figure 11 As shown in the figure, this figure illustrates the restoration effects of five different algorithms on the GOPRO test set images. Each row in the figure represents the output of one algorithm (from top to bottom: Gong, DeepDeblur, DeblurGAN, DeblurGAN-v2, and DeblurGAN-ES of this invention). For detailed comparison, in addition to showing the full-image restoration, three representative local areas are selected for magnification and detailed analysis.

[0149] Observation of Group A shows that the fifth image has the best deblurring effect, with fewer artifacts and a more realistic appearance.

[0150] In the detailed comparison of Group B (neck and face region), it can be seen that the facial contours restored by various comparison algorithms are blurry and the facial features are difficult to distinguish. However, the DeblurGAN-ES algorithm of this invention restores the facial contours clearly and the facial features are distinguishable, which is significantly better than other algorithms, proving the advantages of this invention in restoring the details of complex textures (such as skin and facial features).

[0151] In the comparison of Group C (license plate area), which is most relevant to the application scenario of this invention, the advantages of this invention are most prominent. Various comparison algorithms recover license plate characters with blurry edges and obvious artifacts, making accurate recognition difficult. In contrast, the DeblurGAN-ES algorithm of this invention successfully recovers clear, sharp character strokes and edges, revealing fine details that other algorithms cannot restore. This result fully demonstrates that this invention can provide high-quality, detailed image input for subsequent license plate recognition tasks.

[0152] To better demonstrate the superiority of the DeblurGAN-ES algorithm, a comparative experiment on the attention mechanism and backbone network based on the DeblurGAN-v2 algorithm will be conducted. The experimental datasets are the GOPRO dataset and a self-built license plate dataset. The improved algorithms in the experiments are, in order: CA-MobileNetV3 (Model 1), CA-ShuffleNetV2 (Model 2), SE-MobileNetV3 (Model 3), SE-ShuffleNetV2 (Model 4), EMA-MobileNetV3 (Model 5), and EMA-ShuffleNetV2 (Model 6). The EMA-ShuffleNetV2 (Model 6) algorithm is the DeblurGAN-ES algorithm used in this example. The quantitative comparison results of the experiments are shown in Table 7.

[0153] Table 7. Quantitative comparison experiments of different modules on the GOPRO dataset.

[0154]

[0155] As shown in Table 7, the CA module mainly improves PSNR, the SE module is more helpful in improving SSIM, and the EMA module performs well in both metrics. Among them, EMA-ShuffleNetV2 has the highest PSNR (29.95dB) and the highest SSIM (0.939), indicating that this structure is superior to other models in both image quality and structural similarity.

[0156] See appendix Figure 12The images show the deblurring results of different algorithms on the GoPro dataset. Model 5 and Model 6 show better results compared to other algorithms, but detailed images reveal that Model 5 still lags behind Model 6 in terms of deblurring performance. Model 6 achieves the best deblurring effect, with fewer artifacts and a more realistic appearance. Observing the cropped local images reveals text artifacts, making it difficult to distinguish text in other images. Furthermore, this algorithm also performs well in deblurring areas with high contrast, such as the shadows in the detailed images.

[0157] To further demonstrate the superior performance of the proposed DeblurGAN-ES algorithm in removing motion blur from license plates, a comparative experiment was conducted on a self-built license plate dataset. The deblurring results are shown in the attached figure. Figure 13 As shown.

[0158] See appendix Figure 13 It is evident that Model 5 and Model 6, which integrate the EMA attention mechanism, perform best in terms of image restoration quality. In particular, Model 6 (EMA-ShuffleNetV2), ultimately adopted in this invention, as shown in row (g), can stably output high-quality, clear images under various blurry conditions. From a detailed analysis, Model 6 effectively removes motion artifacts, sharpens character edges, and significantly improves the overall image clarity, enabling accurate reconstruction of previously blurred character information, thus ensuring reliable reading by subsequent recognition algorithms.

[0159] A motion-blurred license plate recognition system based on deep learning, the system comprising:

[0160] User management module: used to verify user identity and manage local images uploaded by users;

[0161] License plate detection module: calls the YOLOv8-KG model, receives images uploaded by users, detects the license plate area in the image, crops and extracts the license plate area, and outputs the cropped license plate image information;

[0162] License plate deblurring module: Receives cropped license plate image data from the detection module, calls the DeblurGAN-ES model to deblur it, and outputs a clear license plate image.

[0163] License plate recognition module: Receives the cleared license plate image, calls the LPR-Net algorithm to perform character recognition, and outputs the final text recognition result.

[0164] In this embodiment, 1000 motion-blurred license plate images were selected as test data, including 500 images from normal environments and 500 images from complex environments. The networks before and after the improvement were tested respectively. The test results are shown in Table 8.

[0165] Table 8 Comparison of System Algorithms

[0166]

[0167] The comparison results show that the system algorithm designed in this invention, YOLOv8-KG+DeblurGAN-ES+LPR-Net, accurately identified 976 images out of 1000 test images, achieving an accuracy rate of 97.6% and an average recognition time of 1.5 seconds. Compared to the previous algorithm, the system algorithm of this invention improves the recognition accuracy by 4.5% and the recognition time by 16.6%, demonstrating the superiority of the improved algorithm.

Claims

1. A method for recognizing motion-blurred license plates based on deep learning, characterized in that... The method employs a two-stage processing flow, including: I. Precise localization steps for motion-blurred license plates: A blurred license plate detection model, YOLOv8-KG, optimized to adapt to motion blur features, is used to detect license plates in the image, and the license plate region is cropped and extracted. II. Image sharpening steps for the target region: Based on the license plate region extracted in step one, input it into a DeblurGAN-ES model specifically designed for restoring blurred details. Perform targeted deblurring on this region to generate a sharpened license plate image, providing reliable input for subsequent character recognition.

2. The method for recognizing motion-blurred license plates based on deep learning according to claim 1, characterized in that... The YOLOv8-KG fuzzy license plate detection model described in step one uses AKconv to replace four convolutional layers in the YOLOv8 backbone network to adaptively match the multi-scale and irregular deformation features caused by motion blur, thus achieving more efficient and flexible sampling.

3. The method for recognizing motion-blurred license plates based on deep learning according to claim 2, characterized in that... The YOLOv8-KG fuzzy license plate detection model described in step one introduces a V-GSCSP structure into the neck network of the YOLOv8 model, which effectively enhances the cross-layer feature fusion capability while significantly reducing the computational complexity of the model.

4. The method for recognizing motion-blurred license plates based on deep learning according to claim 3, characterized in that... The YOLOv8-KG fuzzy license plate detection model described in step one introduces the Convolutional Block Attention (CBAM) mechanism into the key nodes of the YOLOv8 network to improve the model's ability to perceive small, fuzzy license plates.

5. The method for recognizing motion-blurred license plates based on deep learning according to claim 4, characterized in that... The YOLOv8-KG fuzzy license plate detection model described in step one is trained using the GIoU loss function to reduce harmful gradients in sample quality and improve the localization accuracy of motion-fuzzy license plate targets with unclear boundaries.

6. The method for recognizing motion-blurred license plates based on deep learning according to claim 1, characterized in that... The license plate motion blur removal model DeblurGAN-ES described in step two is an optimization based on the DeblurGAN model. ShuffleNetV2 is selected as the lightweight backbone network to balance the optimization problem between computational complexity and image reconstruction quality. The EMA attention mechanism is integrated into both the generator and the discriminator to enhance the model's reconstruction and recognition capabilities in terms of license plate character edges, contour clarity, and background texture details.

7. The method for recognizing motion-blurred license plates based on deep learning according to claim 6, characterized in that... During the feature extraction process, the backbone network of the generator outputs at least three feature maps at different scales, and sets an EMA attention module on the output path of each scale feature map to enhance the information transmission and interaction between local and global features, thereby more accurately restoring the character strokes and edge details of the license plate.

8. The method for recognizing motion-blurred license plates based on deep learning according to claim 7, characterized in that... The discriminator integrates at least one EMA attention module after the deep convolutional block to improve the ability to distinguish the authenticity of overall and local feature images.

9. A method for recognizing motion-blurred license plates based on deep learning according to claim 8, characterized in that... The generator training process employs a composite loss function for constraint. This function works synergistically to ensure the quality of the generated images. It includes: adversarial loss based on the WGAN-GP framework to improve training stability and ensure the quality of generated images; perceptual loss based on the VGG19 network to ensure high-level semantic consistency between content and structure; and edge loss based on the Sobel operator to enhance the recovery of the clarity of license plate characters and border contours.

Images