License plate recognition method and system based on deep learning

Abstract

The invention provides a license plate recognition method and system based on deep learning. The method comprises the following steps: carrying out license plate target detection on a preprocessed to-be-recognized vehicle image through a trained YOLOv8 algorithm model; according to the detected license plate area, determining a license plate type and positioning a key point position on the license plate; segmenting the license plate image, and extracting individual license plate character regions; inputting the extracted license plate character area into the trained improved LPRNet network model, and carrying out character recognition; wherein the improved LPRNet network model comprises a cross attention module and a multi-scale feature fusion module, and the cross attention module is used for dynamically fusing semantic information of different levels in shallow and deep feature maps; and the multi-scale feature fusion module is used for carrying out scale fusion on the feature maps of different scales output by the cross attention module, so that the complexity of a license plate recognition model is reduced, and the reliability of license plate recognition in a complex scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of license plate recognition, and in particular to a license plate recognition method and system based on deep learning. Background Art

[0002] License plate recognition technology plays an important role in intelligent transportation systems and security monitoring. In existing technologies, license plate recognition systems usually use deep learning methods to train a license plate recognition model through a large number of sample images.

[0003] However, the existing solutions also have shortcomings, including:

[0004] High system complexity: Existing technologies often require training multiple independent character recognition models (e.g., models for different license plate types or different lighting conditions), which increases system complexity and computational burden. For example, some methods require training different character recognition models for daytime and nighttime scenes, or training independent character recognition models for license plate types in different countries / regions.

[0005] Low accuracy or reliability in complex scenes: Existing technologies still need to improve their recognition accuracy when processing license plate images in complex scenes. For example, shallow feature maps and deep feature maps contain semantic information at different levels, and feature maps of different scales, such as low-resolution feature maps and high-resolution feature maps, also contain different semantic or detailed information. However, existing technologies are unable to effectively integrate this semantic information at different levels and scales, resulting in low accuracy or reliability of license plate recognition in multi-scale scenes and complex lighting scenarios.

[0006] To address this problem, the present invention provides a license plate recognition method and system based on deep learning to solve the above problem. Summary of the Invention

[0007] In order to solve the problems existing in the prior art, the present invention innovatively proposes a license plate recognition method and system based on deep learning, which effectively solves the problems of high complexity of license plate recognition models or low reliability in complex scenes caused by prior art, effectively reduces the complexity of license plate recognition models and improves the reliability of license plate recognition in complex scenes.

[0008] A first aspect of the present invention provides a license plate recognition method based on deep learning, comprising:

[0009] Acquire vehicle images, perform preprocessing operations on the acquired vehicle images, and establish a training data set based on the preprocessed vehicle images;

[0010] Based on the training data set, the YOLOv8 algorithm model and the improved LPRNet network model were trained respectively;

[0011] Obtain and preprocess the current vehicle image to be identified, perform license plate target detection on the preprocessed vehicle image to be identified using the trained YOLOv8 algorithm model, and output the license plate area;

[0012] Based on the detected license plate area, determine the license plate type and locate the key points on the license plate; based on the license plate type and key point positions, segment the license plate image and extract the individual license plate character areas;

[0013] The extracted license plate character area is input into the trained improved LPRNet network model for character recognition; wherein the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module. The cross-attention module is used to dynamically fuse semantic information of different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module;

[0014] Output the license plate number composed of characters recognized by the improved LPRNet network model.

[0015] A second aspect of the present invention provides a license plate recognition system based on deep learning, comprising:

[0016] An acquisition module acquires vehicle images, performs preprocessing operations on the acquired vehicle images, and establishes a training data set based on the preprocessed vehicle images;

[0017] The training module trains the YOLOv8 algorithm model and the improved LPRNet network model based on the training data set;

[0018] The detection module obtains and preprocesses the current vehicle image to be identified, performs license plate target detection on the preprocessed vehicle image to be identified using the trained YOLOv8 algorithm model, and outputs the license plate area;

[0019] The segmentation and extraction module determines the license plate type and locates the key points on the license plate based on the detected license plate area; based on the license plate type and key point locations, it segments the license plate image and extracts the individual license plate character areas;

[0020] The character recognition module inputs the extracted license plate character area into the trained improved LPRNet network model to perform character recognition; wherein the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module. The cross-attention module is used to dynamically fuse semantic information at different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module;

[0021] The output module outputs the license plate number composed of characters recognized by the improved LPRNet network model.

[0022] The technical solution adopted by the present invention includes the following technical effects:

[0023] 1. In the technical solution of the present invention, the pre-processed vehicle image to be identified is subjected to license plate target detection by the trained YOLOv8 algorithm model, and the license plate area is output; based on the detected license plate area, the license plate type is determined and the key point positions on the license plate are located; based on the license plate type and key point positions, the license plate image is segmented to extract separate license plate character areas; the extracted license plate character areas are input into the trained improved LPRNet network model for character recognition; wherein, the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module, the cross-attention module is used to dynamically fuse semantic information of different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module, effectively solving the problem of high complexity of the license plate recognition model or low reliability in complex scenes caused by the existing technology, effectively reducing the complexity of the license plate recognition model and improving the reliability of license plate recognition in complex scenes.

[0024] 2. In the technical solution of the present invention, the training data set uses a high-definition camera and auxiliary light source to collect vehicle images under different lighting conditions, different weather conditions, different time periods, and different locations; the collected vehicle images are adaptively enhanced according to the lighting and weather conditions of the collected vehicle images; the vehicle images after image enhancement and adjustment are labeled, which not only enables the model to learn feature representations under different conditions and improve the robustness of recognition; it also improves the preprocessing effect of the collected vehicle-connected images, thereby improving the reliability of license plate recognition.

[0025] 3. In the technical solution of the present invention, the license plate type is determined and the key points on the license plate are located based on the detected license plate area; the license plate image is segmented according to the license plate type and key point positions, and the individual license plate character areas are extracted, ensuring the effective recognition of images of different license plate types.

[0026] 4. In the technical solution of the present invention, the extracted license plate character area is input into the trained improved LPRNet network model, and character recognition specifically includes: assigning different weights to each channel through the channel attention mechanism to enhance the expression ability of important channels and suppress the influence of unimportant channels. The improved LPRNet network model can automatically learn which channels are more important for the current task, thereby improving the ability of feature representation. For the license plate recognition task, because different channels may correspond to different features of the license plate (such as character edges, colors, etc.), by enhancing the weights of important channels, the recognition accuracy and reliability can be further improved.

[0027] 5. In the technical solution of the present invention, when the illumination intensity of the collected vehicle image is weak, a weak-light license plate image enhancement network based on a differential convolutional neural network is constructed, and the texture information of the license plate image is decoupled into four directions: horizontal, vertical, and diagonal. Texture enhancement is performed on weak-light images of different scale spaces, further improving the accuracy and reliability of license plate image recognition.

[0028] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 Schematic diagram of the process of the embodiment 1 of the present invention (I);

[0031] Figure 2 Schematic diagram (2) of the process of Example 1 of the present invention;

[0032] Figure 3 Schematic diagram of the structure of the improved LPRNet network model in the method of Example 1 of the present invention;

[0033] Figure 4 Schematic diagram of the process of the method of embodiment 1 in the present invention (3);

[0034] Figure 5 This is a schematic diagram of the structure of the system of Example 2 in the solution of the present invention. DETAILED DESCRIPTION

[0035] In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplicity and clarity and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits descriptions of well-known components and processing technologies and processes to avoid unnecessary limitations on the present invention.

[0036] Example 1

[0037] like Figure 1-Figure 2 As shown, the present invention provides a license plate recognition method based on deep learning, comprising:

[0038] S1, obtaining a vehicle image, performing a preprocessing operation on the obtained vehicle image, and establishing a training data set based on the preprocessed vehicle image;

[0039] S2, train the YOLOv8 algorithm model and the improved LPRNet network model based on the training data set;

[0040] S3, obtains and preprocesses the current vehicle image to be identified, performs license plate target detection on the preprocessed vehicle image to be identified using the trained YOLOv8 algorithm model, and outputs the license plate area;

[0041] S4, based on the detected license plate area, determining the license plate type and locating the key points on the license plate; segmenting the license plate image based on the license plate type and key point positions, and extracting the individual license plate character areas;

[0042] S5, inputting the extracted license plate character region into the trained improved LPRNet network model for character recognition; wherein the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module, wherein the cross-attention module is used to dynamically fuse semantic information of different levels in shallow and deep feature maps; and the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module;

[0043] S6, outputs the license plate number composed of characters recognized by the improved LPRNet network model.

[0044] The step S1 includes obtaining a vehicle image, performing a preprocessing operation on the obtained vehicle image, and establishing a training data set based on the preprocessed vehicle image. Specifically, the steps include:

[0045] S11, using high-definition cameras and auxiliary light sources to collect vehicle images under different lighting conditions, weather conditions, time periods, and locations;

[0046] In order to improve the recognition performance of the model under different lighting conditions, it is necessary to use a license plate image dataset containing various lighting conditions (such as sunlight, street lights, car lights, etc.) and weather conditions (sunny, cloudy, rainy, etc.) for training. This allows the model to learn feature representations under different lighting conditions and improve the robustness of recognition. The specific implementation process is as follows:

[0047] Acquisition equipment: Use a high-definition camera and auxiliary light source (LED fill light) to capture license plate images. The camera is installed in a fixed position (such as a road monitoring pole or parking lot entrance) to ensure a stable shooting angle.

[0048] Environmental coverage: Capture footage under different lighting conditions (such as strong light, weak light, and backlight) and weather conditions (sunny, cloudy, rainy, and snowy); and capture footage at different time periods (daytime and nighttime) and locations (urban roads, highways, and parking lots).

[0049] S12, performing image enhancement on the collected vehicle image according to the lighting and weather conditions of the collected vehicle image;

[0050] Image preprocessing is a key step in license plate recognition systems. One implementation approach is to add a CAE preprocessing module before the algorithm input. By training on noisy and clean image pairs, this module effectively removes image noise caused by low light conditions (such as Gaussian noise and motion blur) while enhancing character edge details. Compared to traditional histogram equalization, CAE can adapt to varying lighting conditions and preserve license plate structure.

[0051] Another implementation method may be adaptive enhancement, specifically including: S121, calculating the average brightness value of the collected vehicle image, and classifying the illumination intensity of the collected vehicle image into strong light, weak light, and normal light according to the average brightness value of the collected vehicle image; when the illumination intensity of the collected vehicle image is strong light, reducing the overall brightness of the image, increasing the contrast of the image, and using a shadow enhancement algorithm to restore details obscured by the strong light; when the illumination intensity of the collected vehicle image is weak light, increasing the overall brightness of the image, using a denoising algorithm to reduce noise, and using an edge detection algorithm to enhance the edges of the image;

[0052] Strong light, weak light, and normal light pre-judgment criteria: Before making adaptive adjustments, it is necessary to analyze and judge the environmental characteristics of the image. Specific pre-judgment criteria for light intensity:

[0053] (1) Strong light condition: By calculating the average brightness value of the image, if the average brightness value exceeds 200 (based on 8-bit image, range 0-255), it is judged as a strong light condition;

[0054] (2) Low light condition: If the average brightness value of the image is less than 50, it is judged as a low light condition;

[0055] (3) Normal lighting conditions: If the average brightness value is between 50 and 200, it is considered to be normal lighting conditions.

[0056] Adaptive adjustments for strong light conditions are as follows:

[0057] Brightness adjustment: Reduce the overall brightness of the image. The brightness value can be mapped to a lower range through linear transformation;

[0058] Contrast adjustment: Increase the contrast of the image, making dark areas darker, bright areas brighter, and enhancing details;

[0059] Shadow Enhancement: Use a shadow enhancement algorithm (adaptive histogram equalization) to restore details obscured by strong light.

[0060] Low-light condition adaptive adjustments are as follows:

[0061] Brightness Boost: Increases the overall brightness of the image to make it clearer.

[0062] Noise suppression: Use denoising algorithms (such as non-local means denoising) to reduce noise in low-light conditions.

[0063] Edge enhancement: Enhance the edges of the image through edge detection algorithms (such as the Canny operator) to improve the readability of characters.

[0064] S122, calculating the brightness standard deviation of the collected vehicle image, and classifying the illumination uniformity of the collected vehicle image as uniform illumination or uneven illumination based on the brightness standard deviation; if the illumination uniformity of the collected vehicle image is uneven, using a local contrast enhancement algorithm to balance the illumination of the image, and using an illumination correction algorithm to uniformize the illumination distribution of the image;

[0065] Illumination uniformity judgment standard: Calculate the brightness standard deviation of the image. If the standard deviation is less than 30, the illumination is considered uniform; if the standard deviation is greater than 50, the illumination is considered uneven.

[0066] Adjust uneven lighting as follows:

[0067] Local contrast adjustment: Use local contrast enhancement algorithms (such as local histogram equalization) to balance the lighting of the image;

[0068] Lighting correction: Use lighting correction algorithms (such as guided filtering-based lighting correction) to uniformly distribute the lighting in the image.

[0069] S123, calculating the contrast and color saturation of the collected vehicle image, and classifying the weather condition of the collected vehicle image as sunny, cloudy, or rainy or snowy based on the contrast and color saturation of the collected vehicle image; when the weather condition of the collected vehicle image is rainy or snowy, using a rain removal algorithm to remove traces of raindrops or snow flakes to improve image clarity;

[0070] Weather conditions judgment criteria:

[0071] Sunny day: The image has high contrast and normal color saturation (the specific threshold can be adjusted according to actual conditions and is not limited in the present invention);

[0072] Cloudy sky: The image has low contrast and low color saturation;

[0073] Rainy or snowy days: The image contains obvious traces of raindrops or snowflakes, and has low contrast and saturation.

[0074] Adaptive adjustment to weather conditions:

[0075] Rainy or snowy days: Use a deraining algorithm (such as a deep learning-based deraining network) to remove raindrops or snowflakes to improve image clarity.

[0076] Foggy days: Use dehazing algorithms (such as dehazing algorithms based on dark channel priors) to enhance the contrast and clarity of images.

[0077] S124, calculating the edge strength of the captured vehicle image, and classifying the image quality of the captured vehicle image as blurred or not blurred based on the edge strength; when the image quality of the captured vehicle image is blurred, using an image deblurring algorithm to increase the clarity of the image;

[0078] Clarity judgment standard: By calculating the edge strength of the image (such as using the Sobel operator to detect edges), if the average value of the edge strength is lower than a certain threshold (such as 30), the image is judged to be blurred.

[0079] Blurred image processing: Use image deblurring algorithms (such as deep learning-based deblurring networks) to restore image clarity.

[0080] S125, calculating the noise intensity of the collected vehicle image, and classifying the image quality of the collected vehicle image into high image noise and low image noise according to the noise intensity of the collected vehicle image; when the image quality of the collected vehicle image is high image noise, using an image denoising algorithm to reduce the noise in the image.

[0081] Noise level: By calculating the noise intensity of the image (such as the standard deviation of Gaussian noise), if the noise intensity exceeds a certain threshold (such as 10), the image is judged to have large noise.

[0082] Noise processing: Use denoising algorithms (such as denoising methods based on wavelet transform) to reduce noise in the image.

[0083] S13, annotating the vehicle image after image enhancement and adjustment, wherein the annotation content includes license plate position, license plate type, license plate characters, and license plate color.

[0084] The vehicle image after image enhancement and adjustment needs to be manually labeled, and the annotation content includes information such as license plate position, license plate type (single layer or multi-layer), license plate characters, and license plate color.

[0085] Among them, in steps S2-S3, the YOLOv8 algorithm model (YOLOv8-backbone model) and the improved LPRNet network model are trained respectively based on the training data set; the training data set can be divided into a training set and a test set according to a set ratio (for example, 1:1), and the YOLOv8 algorithm model and the improved LPRNet network model are trained and optimized to obtain the trained YOLOv8 algorithm model and the improved LPRNet network model.

[0086] During the training process of the YOLOv8 algorithm model and the improved LPRNet network model, data enhancement can be performed on the training data, that is, using knowledge distillation technology to generate diversified, high-quality enhanced data through the teacher model, combined with feature distillation and soft label guidance, to improve the generalization ability of the student model under complex data distribution. Specific methods include image reconstruction based on teacher model predictions, feature vector generation, adversarial sample generation, etc. (such as strong backlight, night fill light, different angles, different weather backgrounds, etc.), thereby surpassing traditional manual data enhancement methods (such as random brightness adjustment, contrast adjustment, adding noise, etc.) to achieve more efficient data enhancement. Taking the improved LPRNet network model training as an example, the methods for generating enhanced data based on teacher model predictions, image reconstruction, feature vector generation, adversarial sample generation, etc. are as follows:

[0087] (1) Image reconstruction based on multimodal teacher model prediction

[0088] 1) Teacher model feature extraction:

[0089] A multimodal teacher model (Double-YOLO-Kaist) is used to extract features from the original image and generate high-level feature representations. This model can extract feature maps separately through a two-stream parallel backbone extraction network and perform mixed-modal feature fusion through a feature sharing learning network.

[0090] 2) Feature vector generation:

[0091] The feature representation extracted by the teacher model is converted into a feature vector. The feature vector can be generated by a fully connected layer or an attention mechanism to represent the key features of the image.

[0092] 3) Image reconstruction:

[0093] Generate new images based on the feature vector using a Generative Adversarial Network (GAN) or a Variational Autoencoder (VAE). The generated images can simulate different lighting conditions (such as strong backlight, night-time fill light), weather conditions (such as rainy days, snowy days), and angle changes (such as tilt, rotation).

[0094] (2) Feature vector generation

[0095] 1) Feature Distillation:

[0096] Through feature distillation technology, the feature vector of the teacher model is passed to the student model (LPRNet). Feature distillation can be achieved through the knowledge distillation framework, and the student model learns by imitating the feature output of the teacher model.

[0097] 2) Feature Fusion:

[0098] The feature vector of the teacher model is fused with the feature vector of the student model (LPRNet) to enhance the feature representation capability of the student model. Feature fusion can be achieved through element-wise addition, concatenation, or attention mechanism.

[0099] (3) Adversarial Sample Generation

[0100] 1) Adversarial training:

[0101] Adversarial training techniques are used to generate adversarial examples by adding small perturbations. Adversarial examples can improve the robustness of the model to noise and abnormal situations.

[0102] 2) Adversarial sample generation based on teacher model:

[0103] Using the teacher model’s predictions as guidance, we generate adversarial examples. We then determine the perturbation direction of the adversarial examples by calculating the teacher model’s gradient.

[0104] 3) Application of adversarial examples:

[0105] The generated adversarial examples are added to the training dataset to improve the generalization ability and robustness of the student model (LPRNet).

[0106] (4) Specific implementation steps

[0107] 1) Data preprocessing:

[0108] Perform preprocessing on the original image, including normalization, denoising and enhancement.

[0109] 2) Teacher model training:

[0110] Train the teacher model to accurately extract features under different modalities.

[0111] 3) Feature extraction and vector generation:

[0112] Use the teacher model to extract features from the original image and generate a feature vector.

[0113] 4) Image reconstruction:

[0114] Use GAN or VAE to generate new images based on the feature vector, simulating different lighting, weather, and angle conditions.

[0115] 5) Feature Distillation and Fusion:

[0116] The feature vector of the teacher model is passed to the student model and feature fusion is performed.

[0117] 6) Adversarial Sample Generation:

[0118] Adversarial examples are generated through adversarial training and added to the training dataset.

[0119] 7) Student model training:

[0120] The student model is trained using reconstructed images and adversarial examples to improve its generalization ability and robustness.

[0121] Then, the current vehicle image to be identified is acquired and preprocessed. The trained YOLOv8 algorithm model performs license plate target detection on the preprocessed vehicle image to be identified, and outputs the license plate region. The preprocessing process is the same as the image preprocessing process in step S1 and is not described here. The trained YOLOv8 algorithm model performs license plate target detection on the preprocessed vehicle image to be identified, and outputs the license plate region (including the area of the license plate image).

[0122] Among them, in step S4, based on the detected license plate area, the license plate type is determined and the key points on the license plate are located; based on the license plate type and key point positions, the license plate image is segmented to extract the individual license plate character areas, specifically including:

[0123] S41, judging whether the license plate type is single-layer or multi-layer based on the detected license plate area, and locating the four corner points on the license plate;

[0124] Specifically, the key point detection network HRNet (High-Resolution Network, a deep neural network architecture for visual tasks) is used to detect key points in the license plate area, and geometric constraints (such as the rectangular shape constraint of the license plate) are applied to further optimize the position of the key points (four corner points) to ensure the accuracy of positioning.

[0125] S42, when the license plate type is a single-layer license plate, the license plate area is segmented into multiple character areas, and the segmentation is dynamic using a sliding window of fixed width. The sliding window is initially positioned at the key point in the upper left corner of the license plate and moves along the license plate area with a fixed step size each time. It is determined whether the sliding window contains any characters. If so, the license plate character area in the sliding window is extracted.

[0126] Single-layer license plate recognition: usually has a long rectangular shape with characters arranged in a row.

[0127] Single-layer license plate segmentation: The license plate area is divided into multiple character areas based on the aspect ratio and key point position of the license plate. Dynamic segmentation is performed using a fixed-width sliding window. The initial position of the sliding window is determined based on the key point position. It usually starts from the key point in the upper left corner of the license plate and moves along the license plate area, moving a fixed step size each time. At each position, the image features within the window (such as edge strength and color consistency) are analyzed to determine whether the window contains characters. If a deviation between the character width and the standard width is detected, the window width is dynamically adjusted to adapt to the actual character width. The width of the sliding window is usually set to 1.2 times the standard width of the character, and the step size is set to 0.8 times the standard width of the character to ensure that characters are not missed.

[0128] S43, when the license plate type is a multi-layer license plate, the license plate area is divided into multiple lines of character areas, and each line of character area is segmented. A fixed-width sliding window is used for dynamic segmentation. The initial position of the sliding window is the key point in the upper left corner of the license plate. It moves along the license plate area with a fixed step length each time. It is determined whether the window contains characters. If it contains characters, the license plate character area of the sliding window is extracted.

[0129] Multi-layer license plate recognition: has a shorter rectangular or square shape, with characters arranged in two or more rows.

[0130] Multi-layer license plate segmentation: The license plate area is divided into multiple rows based on the plate's height and keypoint locations. Character segmentation is then performed on each row, following the same principles as for single-layer license plates. Each row is segmented based on its height and keypoint locations, and the sliding window parameters are adjusted based on the specific dimensions of each row. For example, for a multi-layer license plate with a plate height of H, the height of each row is typically around H / 2. The sliding window width and step size are set based on the expected number of characters and character width per row.

[0131] In step S5, the extracted license plate character region is input into the trained improved LPRNet network model, and character recognition is performed, specifically including:

[0132] S51, using a CNN convolutional neural network to extract features from the input license plate image to be identified, generating a shallow feature map and a deep feature map; the shallow feature map includes character edge features, character texture features, and local shape features of the license plate image to be identified; the deep feature map includes overall shape features of the license plate, character semantic information, and license plate category in the license plate image to be identified;

[0133] One of the improvements to the LPRNet model is that, during the feature extraction phase, it combines traditional CNN convolutional neural networks with Cross-Attention. This dynamically integrates semantic information from different levels by using a query matrix derived from shallow feature maps and a key-value matrix derived from deep feature maps. In license plate recognition, traditional CNN convolutional neural networks excel at extracting local features (such as edges and textures), while Cross-Attention can capture global dependencies, particularly associations between features at different levels. By combining traditional CNN with Cross-Attention, semantic information from different levels can be dynamically integrated, improving recognition capabilities in complex lighting conditions and blurred images.

[0134] CNN convolutional neural network feature extraction: Use traditional CNN convolutional neural network (such as ResNet, VGG) to extract features from the input image and generate shallow feature maps and deep feature maps.

[0135] Shallow feature map: contains low-level features (such as edges, textures, etc.), referring to the 1st to 3rd layers. The shallow feature map mainly captures the local features of the image, such as the edges and textures of the characters and the local shape of the license plate.

[0136] Deep feature map: contains high-level semantic features (such as license plate shape, characters, etc.), referring to layers 4 to 6. The deep feature map mainly captures the global features of the image, such as the overall shape of the license plate, the semantic information of the characters, and the category of the license plate.

[0137] S52, dynamically fuses the shallow feature map and the semantic information of different levels in the deep feature map through the cross attention module, and outputs the hierarchical fused license plate feature map;

[0138] Among them, a cross attention module is inserted into the middle layer of the CNN convolutional neural network to dynamically fuse the shallow and deep feature maps. Step S52 specifically includes:

[0139] S521, generating a query matrix Q based on the shallow feature map, and generating a key matrix K and a value matrix V based on the deep feature map;

[0140] Query matrix Q (Query): comes from the shallow feature map, indicating the location and content of low-level features. Generation method: A convolutional layer transforms the shallow feature map to generate the query matrix Q. Formula:

[0141] Q=W Q ·F shallow ,

[0142] Among them, W Q is the query transformation matrix, F shallow It is a shallow feature map.

[0143] Key matrix K (Key) and value matrix V (Value): from the deep feature map, representing the content of high-level semantic features. Generation method: The deep feature map is transformed through two convolutional layers to generate the key matrix K and value matrix V respectively. Formula:

[0144] K=W K ·F deep ,

[0145] V=W V ·F deep

[0146] Among them, W K and W V are the transformation matrices of keys and values, respectively, F deep It is a deep feature map.

[0147] S522, calculate the attention score between the query matrix Q and the key matrix K, where the attention score is used to represent the correlation between each position in the shallow feature map and each position in the deep feature map; wherein the calculation method of the attention score is specifically as follows:

[0148]

[0149] Among them, Attention Scores is the attention score, d k is the dimension of the key matrix K, and T is the transpose of the key matrix K;

[0150] S523, performing weighted summation on the value matrix V according to the attention score to generate a hierarchically fused license plate feature map.

[0151] Weighted summation formula:

[0152] F fused =Attention Scores·V,

[0153] Ffused The cross-attention mechanism dynamically fuses shallow feature maps with deep feature maps to generate a hierarchically fused license plate feature map. This feature map combines the detailed information of the shallow feature map with the semantic information of the deep feature map, enhancing the comprehensiveness and accuracy of feature representation.

[0154] Dynamic fusion of semantic information at different levels occurs in the intermediate layers of the feature extraction phase. It dynamically fuses shallow and deep feature maps using a cross-attention mechanism. Shallow feature maps contain low-level features (such as edges and textures), while deep feature maps contain high-level semantic features (such as license plate shape and characters). Dynamic fusion aims to enhance the comprehensiveness and accuracy of feature representation, especially under changing lighting conditions or in complex environments.

[0155] S53, scale fusion is performed on the features of different scales in the hierarchically fused license plate feature image through a multi-scale feature fusion module.

[0156] Wherein, step S53 specifically includes:

[0157] S531, extracting features of the hierarchically fused license plate feature map through a CNN convolutional neural network to obtain a low-resolution feature map and a first high-resolution feature map; wherein the low-resolution feature map is a feature map obtained by extracting the hierarchically fused license plate feature map through a deep layer of the CNN convolutional neural network, and the first high-resolution feature map is a feature map obtained by extracting the hierarchically fused license plate feature map through a shallow layer of the CNN convolutional neural network;

[0158] After dynamic fusion, the single feature map F after dynamic fusion fused This information is fed into the Feature Pyramid Network (FPN) as input. The FPN, a multi-scale feature fusion module, fuses features at multiple scales to enhance detection of license plates of varying sizes, while the channel attention mechanism further enhances the weights of important feature channels. This processing step aims to optimize the feature representation, making it more suitable for subsequent license plate recognition tasks.

[0159] Bottom-up path:

[0160] Extract F through convolutional neural network (CNN) fused The features of the license plate feature map after hierarchical fusion are used to obtain a series of feature maps with different resolutions. The resolution of these feature maps decreases layer by layer, and the semantic information is enhanced layer by layer. Specifically:

[0161] Low-resolution feature maps (generated by convolution): contain rich semantic information but have low spatial resolution and are located in the deep layers of the network after convolution (layers 4 to 6).

[0162] High-resolution feature maps (generated by convolution): contain rich detail information but weak semantic information, and are located in the shallow layers of the network after convolution (layers 1 to 3).

[0163] Although the bottom-up approach generates feature maps of varying resolutions, the semantic information of these feature maps increases layer by layer, while the spatial resolution decreases. For the high-resolution feature task of license plate recognition, low-resolution feature maps, while rich in semantic information, have low spatial resolution and cannot be directly used for high-precision localization and detection.

[0164] S532, gradually generating a second high-resolution feature map by upsampling the low-resolution feature map;

[0165] The upsampling operation is a top-down path:

[0166] The top-down path propagates low-resolution semantic information to high-resolution feature maps, so that high-resolution feature maps not only retain detail information but also enhance semantic information. In this way, the generated multi-scale feature maps can have both high resolution and strong semantic information, thereby improving the performance of the model in multi-scale scenarios. The specific implementation method is:

[0167] The low-resolution feature map (generated by convolution) is gradually converted into a second high-resolution feature map (generated by upsampling) through upsampling operations. Upsampling can use simple nearest neighbor interpolation or bilinear interpolation.

[0168] S533, the second high-resolution feature map F upsample With the first high-resolution feature map F low-level Add tensors to generate scale fusion feature map F out .

[0169] The feature maps on the top-down path and the bottom-up path are horizontally connected. The specific operation is to upsample the second high-resolution feature map F upsample The first high-resolution feature map F generated by convolution low-level Add tensors to generate scale fusion feature map F out This fusion method can improve the comprehensiveness and accuracy of feature representation.

[0170] F out =F upsample +F low-level

[0171] Preferably, step S5 may further include:

[0172] S54, scale fusion feature map F out Perform global average pooling to obtain statistical information for each channel;

[0173] The scale fusion feature map F generated by FPN out Apply the attention mechanism to further enhance the feature expression capability:

[0174] The core idea of the channel attention mechanism is to enhance the expressiveness of important channels and suppress the influence of unimportant channels by assigning different weights to each channel. The implementation steps are as follows:

[0175] Squeeze (compression): input feature map F out Perform global average pooling to obtain statistical information for each channel. This will give a description on the channel dimension. Obtain statistical information for each channel:

[0176]

[0177] Where H and W are the height and width of the feature map. c,i,j represents the pixel value of the cth channel at position (i, j), z c Represents the output value of the c-th channel after global average pooling.

[0178] S55, generates the weight of each channel through the fully connected layer based on the statistical information of each channel:

[0179] Excitation: Generate the weight of each channel through a fully connected layer based on the statistical information of each channel:

[0180] σ=Sigmoid(FC(ReLU(FC(z)))),

[0181] Here, FC represents a fully connected layer, Sigmoid and ReLU are commonly used activation functions, z represents the output value after global average pooling, and the compression ratio r is an integer used to control the dimension of the intermediate representation of the fully connected layer. Specifically, assuming the number of channels of the input feature map is C, the dimension of the intermediate representation of the fully connected layer is C / r, and the compression ratio r is set to 16 to reduce the number of parameters.

[0182] S56, the weight of each channel generated is fused with the scale feature map F out Multiply each channel in to get the final feature map F output . That is F output =σ·F out .

[0183] Through the channel attention mechanism, the model can automatically learn which channels are more important to the current task, thereby improving the ability of feature representation. This mechanism is particularly suitable for license plate recognition tasks, because different channels may correspond to different features of the license plate (such as character edges, color, etc.). By increasing the weight of important channels, the recognition accuracy can be significantly improved.

[0184] The improvements of the technical solution of the present invention on the LPRNet model are mainly reflected in the cross attention module, the multi-scale feature fusion module (FPN), and the channel attention mechanism:

[0185] In the LPRNet model, combining FPN and channel attention mechanism can significantly improve the performance of the model:

[0186] Multi-scale feature fusion: By fusing feature maps of different scales through FPN, the detection capability of license plates of different sizes is enhanced. This enables the model to better adapt to changes in license plates at different distances and angles.

[0187] Channel Attention Enhancement: An attention mechanism is applied to the channel dimension of the feature map to further enhance the weights of important feature channels. This enables the model to focus more precisely on the key features of the license plate, improving recognition accuracy.

[0188] Through these improvements, the LPRNet model has been enhanced in terms of feature extraction, multi-scale detection, and attention mechanism, and can better adapt to license plate recognition tasks in complex environments.

[0189] Specifically, if Figure 3 As shown in the figure, the complete structure diagram of the improved LPRNet model is as follows:

[0190] The specific functions and roles of each module are as follows:

[0191] Convolutional layers: Multiple convolutional layers are used to extract features of an image. These convolutional layers learn local features of the image and gradually extract more and more abstract information.

[0192] Small Inception block: A variant of the Inception structure is used in the network, which can capture features at different scales and enhance the model's adaptability to scale changes.

[0193] Spatial Transformer Network (STN): Used to correct the distortion of license plate images so that the network can still accurately identify the license plate even when the license plate position, angle, lighting and other conditions change.

[0194] Cross-Attention Module: A cross-attention module is inserted into the middle layer of the CNN to dynamically fuse shallow and deep feature maps. The query comes from the shallow feature map, and the key-value comes from the deep feature map, dynamically fusing semantic information at different levels.

[0195] Multi-scale feature fusion: Feature maps of different scales are fused through the Feature Pyramid Network (FPN) to enhance the detection capability of license plates of different sizes.

[0196] Channel Attention: Apply the attention mechanism on the channel dimension of the feature map to further enhance the weights of important feature channels.

[0197] Bidirectional Recurrent Neural Network (Bi-RNN): It is responsible for processing sequence data and capturing the dependencies between sequences through forward and backward propagation.

[0198] Connected Temporal Multilayer Perceptron (CTC): This is a loss function that handles the situation where the length of input data and output labels are inconsistent. It enables the network to learn the correspondence between images and their label sequences.

[0199] By improving LPRNet, we can enhance the ability to focus on light-sensitive areas (such as character edges) to improve the ability to extract license plate features under different lighting conditions, change the character positioning accuracy under uneven lighting, and thus improve the model's adaptability to lighting changes. The reasons are as follows:

[0200] (1) Enhanced feature extraction

[0201] Limitations of traditional CNN: Traditional CNN mainly relies on convolution operations when extracting features. Although it can extract rich local features (such as edges, textures, etc.), it is easily affected by uneven lighting when dealing with lighting changes, resulting in inaccurate feature extraction.

[0202] Advantages of Cross-Attention: By introducing global dependencies, the cross-attention mechanism can dynamically adjust the weights of feature extraction, allowing the model to pay more attention to lighting-sensitive areas (such as character edges). By using queries from shallow feature maps and key-values from deep feature maps, cross-attention can capture the correlation between shallow and deep features, thereby enhancing adaptability to lighting changes.

[0203] (2) Dynamic feature fusion

[0204] 1) Multi-level feature fusion: Through the cross-attention mechanism, shallow feature maps (containing low-level features) and deep feature maps (containing high-level semantic features) can be dynamically fused. This fusion method can integrate the detailed information of shallow features and the semantic information of deep features, improving the comprehensiveness and accuracy of feature representation.

[0205] 2) Handling uneven illumination: In the presence of uneven illumination, shallow feature maps may contain more noise and uneven information, while deep feature maps can provide more stable semantic information. Through the cross-attention mechanism, the model can automatically adjust the fusion weights according to the lighting conditions, reducing the impact of uneven illumination on feature extraction.

[0206] (3) Enhanced model adaptability

[0207] 1) Adaptive Adjustment: The cross-attention mechanism dynamically adjusts the weights of feature fusion using attention scores, enabling the model to adaptively respond to different lighting conditions. For example, in bright light conditions, the model will pay more attention to features in dark areas; in dim light conditions, the model will pay more attention to features in bright areas.

[0208] 2) Improved robustness: By combining traditional CNNs with cross-attention, the model not only extracts rich local features but also captures global dependencies, thereby improving robustness to lighting changes. This design enables the model to maintain high recognition accuracy even under complex lighting conditions.

[0209] (4) Improvement of character positioning accuracy

[0210] 1) Accurate feature representation: By dynamically fusing semantic information from different levels, the model can more accurately locate the position of license plate characters. Especially in conditions of uneven lighting, the cross-attention mechanism helps the model better focus on character edges, improving character positioning accuracy.

[0211] 2) Reduce false detections and missed detections: By enhancing the ability to focus on light-sensitive areas, the model can reduce false detections and missed detections caused by lighting changes, thereby improving recognition accuracy and stability. It can recognize all 78 license plate characters on the market, improving the comprehensiveness of recognition.

[0212] Preferably, if Figure 4 As shown, the technical solution of the present invention further provides a license plate recognition method based on deep learning, which, before step S4, further includes:

[0213] S40, when the illumination intensity of the collected vehicle image is weak light, construct a weak light license plate image enhancement network based on a differential convolutional neural network, decouple the texture information of the license plate image into four directions: horizontal, vertical, and diagonal, and perform texture enhancement on weak light images of different scale spaces.

[0214] Specifically, optimization under special lighting conditions: For license plate recognition under weak or complex lighting conditions, a low-light license plate image enhancement network based on a differential convolutional neural network can be constructed. The texture information of the license plate is decoupled into four directions: horizontal, vertical, and diagonal. Texture enhancement is performed on low-light images of different scales to improve the perceptual quality of low-light images. The specific implementation is as follows:

[0215] (1) Construction of a low-light license plate image enhancement network based on differential convolutional neural network

[0216] Network architecture design:

[0217] Input layer: receives low-light license plate images.

[0218] Differential convolution layer: Captures subtle changes and details in the image through differential convolution operations.

[0219] Feature extraction layer: Use multiple layers of convolution and pooling layers to extract high-level features of the image.

[0220] Texture enhancement layer: Enhances texture information in different directions through a specific filter group.

[0221] Output layer: Outputs the enhanced license plate image.

[0222] Differential convolution operation:

[0223] Differential convolution kernel: A specific convolution kernel is designed to capture small changes in the image.

[0224] Multi-branch structure: A multi-branch structure is used to process texture information in different directions separately.

[0225] (2) Network training:

[0226] Dataset preparation: Use a dataset of license plate images under weak and complex lighting conditions.

[0227] Loss function design: Combining perceptual loss and reconstruction loss to ensure that the enhanced image retains details and reduces noise.

[0228] Optimization algorithm: Use the Adam optimization algorithm for network training.

[0229] (3) Decoupling of license plate texture information

[0230] Direction-sensitive filter banks:

[0231] Horizontal filter: captures texture information in the horizontal direction.

[0232] Vertical filter: captures texture information in the vertical direction.

[0233] Diagonal direction filter: captures texture information in the diagonal direction (including the upper left to lower right and upper right to lower left directions).

[0234] Decoupling process:

[0235] Filter application: Apply the above filter groups separately to process the image.

[0236] Feature fusion: Fuse texture features from different directions to generate a comprehensive texture feature map.

[0237] (4) Texture enhancement in different scale spaces

[0238] Multi-scale decomposition:

[0239] Laplacian pyramid decomposition: decomposes the image into subbands of different scales.

[0240] Wavelet transform: Perform wavelet decomposition on the image to extract features of different scales.

[0241] Texture Enhancement:

[0242] Scale-specific enhancement: A specific enhancement algorithm (such as contrast enhancement and noise suppression) is applied to each sub-band at each scale.

[0243] Detail enhancement: Highlight the detail information of the image by enhancing the high-frequency sub-band.

[0244] Refactoring:

[0245] Multi-scale reconstruction: reconstruct the enhanced sub-bands into a complete image.

[0246] This improvement is mainly used in image acquisition and preprocessing. The details are as follows:

[0247] License plate recognition in low-light or complex lighting conditions: During the image preprocessing stage, a low-light license plate image enhancement network based on a differential convolutional neural network is used to process the captured images to improve the perceptual quality of low-light images and ensure the accuracy of subsequent license plate detection and recognition.

[0248] Construction of a low-light license plate image enhancement network based on differential convolutional neural network: During the preprocessing process, by constructing and applying this network, the image is denoised, enhanced, and other operations are performed to enhance the image details and texture information, thereby improving the overall image quality.

[0249] Decoupling of license plate texture information: In the preprocessing stage, the image is processed through a direction-sensitive filter group to decouple the texture information of the license plate into three directions: horizontal, vertical, and diagonal, thereby enhancing the expressiveness of texture features.

[0250] Texture enhancement at different scales: Also in the preprocessing stage, the image is decomposed into multiple scales using methods such as Laplace pyramid decomposition or wavelet transform. Then, texture enhancement is performed on sub-bands of different scales. Finally, the image is reconstructed to improve image details and overall quality.

[0251] Through the comprehensive application of the above strategies, the license plate recognition effect of the LPRNet model under different lighting conditions can be significantly improved.

[0252] Combining advanced object detection models like YOLOv8 with LPRNet for license plate recognition improves the overall robustness and accuracy of the system. YOLOv8 can quickly and accurately locate license plates from images, while LPRNet is specifically optimized for license plate character recognition. The combination of the two enables efficient and accurate license plate recognition.

[0253] In the technical solution of the present invention, the pre-processed vehicle image to be identified is subjected to license plate target detection through the trained YOLOv8 algorithm model, and the license plate area is output; based on the detected license plate area, the license plate type is determined and the key point positions on the license plate are located; based on the license plate type and the key point positions, the license plate image is segmented to extract the individual license plate character areas; the extracted license plate character areas are input into the trained improved LPRNet network model for character recognition; wherein, the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module, the cross-attention module is used to dynamically fuse semantic information of different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module, effectively solving the problem of high complexity of the license plate recognition model or low reliability in complex scenes caused by the existing technology, effectively reducing the complexity of the license plate recognition model and improving the reliability of license plate recognition in complex scenes.

[0254] In the technical solution of the present invention, the training data set uses a high-definition camera and auxiliary light source to collect vehicle images under different lighting conditions, different weather conditions, different time periods, and different locations; the collected vehicle images are adaptively enhanced according to the lighting and weather conditions of the collected vehicle images; the vehicle images after image enhancement adjustment are labeled, which not only enables the model to learn feature representations under different conditions and improve the robustness of recognition; it also improves the preprocessing effect of the collected vehicle-connected images, thereby improving the reliability of license plate recognition.

[0255] In the technical solution of the present invention, the license plate type is determined and the key points on the license plate are located based on the detected license plate area; the license plate image is segmented according to the license plate type and key point positions, and the individual license plate character areas are extracted, ensuring the effective recognition of images of different license plate types.

[0256] In the technical solution of the present invention, the extracted license plate character area is input into the trained improved LPRNet network model, and character recognition specifically includes: assigning different weights to each channel through the channel attention mechanism to enhance the expression ability of important channels and suppress the influence of unimportant channels. The improved LPRNet network model can automatically learn which channels are more important for the current task, thereby improving the ability of feature representation. For the license plate recognition task, because different channels may correspond to different features of the license plate (such as character edges, colors, etc.), by enhancing the weights of important channels, the recognition accuracy and reliability can be further improved.

[0257] In the technical solution of the present invention, when the illumination intensity of the collected vehicle image is weak, a weak-light license plate image enhancement network based on a differential convolutional neural network is constructed, the texture information of the license plate image is decoupled into four directions: horizontal, vertical, and diagonal, and texture enhancement is performed on weak-light images of different scale spaces, further improving the accuracy and reliability of license plate image recognition.

[0258] Example 2

[0259] like Figure 5 As shown, the technical solution of the present invention also provides a license plate recognition system based on deep learning, including:

[0260] An acquisition module 101 acquires a vehicle image, performs a preprocessing operation on the acquired vehicle image, and establishes a training data set based on the preprocessed vehicle image;

[0261] A training module 102 is configured to train a YOLOv8 algorithm model and an improved LPRNet network model based on a training data set;

[0262] The detection module 103 acquires and preprocesses the current vehicle image to be identified, performs license plate target detection on the preprocessed vehicle image to be identified using the trained YOLOv8 algorithm model, and outputs the license plate area;

[0263] The segmentation and extraction module 104 determines the license plate type and locates the key points on the license plate based on the detected license plate area; segments the license plate image based on the license plate type and key point locations to extract the individual license plate character areas;

[0264] The character recognition module 105 inputs the extracted license plate character region into the trained improved LPRNet network model to perform character recognition; wherein the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module, wherein the cross-attention module is used to dynamically fuse semantic information of different levels in the shallow and deep feature maps; and the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module;

[0265] The output module 106 outputs the license plate number composed of characters recognized by the improved LPRNet network model.

[0266] It should be noted that in the embodiment of the present invention, the implementation process of each module in the acquisition module 101, training module 102, detection module 103, segmentation and extraction module 104, character recognition module 105, and output module 106 corresponds to the method steps in Example 1, and will not be repeated here in this embodiment.

[0267] In the technical solution of the present invention, the pre-processed vehicle image to be identified is subjected to license plate target detection through the trained YOLOv8 algorithm model, and the license plate area is output; based on the detected license plate area, the license plate type is determined and the key point positions on the license plate are located; based on the license plate type and the key point positions, the license plate image is segmented to extract the individual license plate character areas; the extracted license plate character areas are input into the trained improved LPRNet network model for character recognition; wherein, the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module, the cross-attention module is used to dynamically fuse semantic information of different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module, effectively solving the problem of high complexity of the license plate recognition model or low reliability in complex scenes caused by the existing technology, effectively reducing the complexity of the license plate recognition model and improving the reliability of license plate recognition in complex scenes.

[0268] In the technical solution of the present invention, the training data set uses a high-definition camera and auxiliary light source to collect vehicle images under different lighting conditions, different weather conditions, different time periods, and different locations; the collected vehicle images are adaptively enhanced according to the lighting and weather conditions of the collected vehicle images; the vehicle images after image enhancement adjustment are labeled, which not only enables the model to learn feature representations under different conditions and improve the robustness of recognition; it also improves the preprocessing effect of the collected vehicle-connected images, thereby improving the reliability of license plate recognition.

[0269] In the technical solution of the present invention, the license plate type is determined and the key points on the license plate are located based on the detected license plate area; the license plate image is segmented according to the license plate type and key point positions, and the individual license plate character areas are extracted, ensuring the effective recognition of images of different license plate types.

[0270] In the technical solution of the present invention, the extracted license plate character area is input into the trained improved LPRNet network model, and character recognition specifically includes: assigning different weights to each channel through the channel attention mechanism to enhance the expression ability of important channels and suppress the influence of unimportant channels. The improved LPRNet network model can automatically learn which channels are more important for the current task, thereby improving the ability of feature representation. For the license plate recognition task, because different channels may correspond to different features of the license plate (such as character edges, colors, etc.), by enhancing the weights of important channels, the recognition accuracy and reliability can be further improved.

[0271] In the technical solution of the present invention, when the illumination intensity of the collected vehicle image is weak, a weak-light license plate image enhancement network based on a differential convolutional neural network is constructed, the texture information of the license plate image is decoupled into four directions: horizontal, vertical, and diagonal, and texture enhancement is performed on weak-light images of different scale spaces, further improving the accuracy and reliability of license plate image recognition.

[0272] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A license plate recognition method based on deep learning, characterized in that: include: Acquire vehicle images, perform preprocessing operations on the acquired vehicle images, and establish a training data set based on the preprocessed vehicle images; Based on the training data set, the YOLOv8 algorithm model and the improved LPRNet network model were trained respectively; Obtain and preprocess the current vehicle image to be identified, perform license plate target detection on the preprocessed vehicle image to be identified using the trained YOLOv8 algorithm model, and output the license plate area; Based on the detected license plate area, determine the license plate type and locate the key points on the license plate; based on the license plate type and key point positions, segment the license plate image and extract the individual license plate character areas; The extracted license plate character area is input into the trained improved LPRNet network model for character recognition; wherein the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module. The cross-attention module is used to dynamically fuse semantic information of different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module; Output the license plate number composed of characters recognized by the improved LPRNet network model.

2. A license plate recognition method based on deep learning according to claim 1, characterized in that: Acquire vehicle images, perform preprocessing operations on the acquired vehicle images, and establish a training data set based on the preprocessed vehicle images, specifically including: Use high-definition cameras and auxiliary light sources to collect vehicle images under different lighting conditions, weather conditions, time periods, and locations; Perform image enhancement on the collected vehicle images according to the lighting and weather conditions of the collected vehicle images; The vehicle image after image enhancement and adjustment is annotated, and the annotated content includes license plate position, license plate type, license plate characters, and license plate color.

3. The license plate recognition method based on deep learning according to claim 2, characterized in that: Image enhancement of the collected vehicle images according to the lighting and weather conditions of the collected vehicle images specifically includes: Calculate the average brightness value of the collected vehicle image and classify the light intensity of the collected vehicle image into strong light, weak light, and normal light based on the average brightness value. When the light intensity of the collected vehicle image is strong light, reduce the overall brightness of the image, increase the contrast of the image, and use the shadow enhancement algorithm to restore the details obscured by the strong light. When the light intensity of the collected vehicle image is weak light, increase the overall brightness of the image, use the denoising algorithm to reduce noise, and use the edge detection algorithm to enhance the edges of the image. Calculate the brightness standard deviation of the collected vehicle image and classify the illumination uniformity of the collected vehicle image as uniform or uneven based on the brightness standard deviation. If the illumination uniformity of the collected vehicle image is uneven, use a local contrast enhancement algorithm to balance the illumination of the image, and use an illumination correction algorithm to even out the illumination distribution of the image. Calculate the contrast and color saturation of the captured vehicle image, and classify the weather conditions of the captured vehicle image into sunny, cloudy, and rainy and snowy based on the contrast and color saturation of the captured vehicle image. When the weather condition of the captured vehicle image is rainy or snowy, use a rain removal algorithm to remove traces of raindrops or snowflakes to improve image clarity. Calculating the edge strength of the captured vehicle image, and classifying the image quality of the captured vehicle image as blurred or not blurred according to the edge strength; when the image quality of the captured vehicle image is blurred, using an image deblurring algorithm to increase the clarity of the image; The noise intensity of the collected vehicle image is calculated, and the image quality of the collected vehicle image is divided into high image noise and low image noise according to the noise intensity of the collected vehicle image; when the image quality of the collected vehicle image is high image noise, an image denoising algorithm is used to reduce the noise in the image.

4. The method for license plate recognition based on deep learning according to claim 1, characterized in that: Based on the detected license plate area, the license plate type is determined and the key points on the license plate are located. Based on the license plate type and key point locations, the license plate image is segmented and the individual license plate character areas are extracted. Specifically, the following are included: According to the detected license plate area, determine whether the license plate type is single-layer or multi-layer, and locate the four corner points on the license plate; When the license plate type is a single-layer license plate, the license plate area is divided into multiple character areas and dynamically segmented using a fixed-width sliding window. The initial position of the sliding window is the key point in the upper left corner of the license plate. It moves along the license plate area with a fixed step length each time to determine whether the sliding window contains characters. If it does, the license plate character area in the sliding window is extracted. When the license plate type is a multi-layer license plate, the license plate area is divided into multiple lines of character areas, and each line of character area is segmented using characters. A fixed-width sliding window is used for dynamic segmentation. The initial position of the sliding window is the key point in the upper left corner of the license plate. It moves along the license plate area with a fixed step length each time to determine whether the window contains characters. If it contains characters, the license plate character area of the sliding window is extracted.

5. The method for license plate recognition based on deep learning according to claim 1, characterized in that: The extracted license plate character area is input into the trained improved LPRNet network model to perform character recognition, specifically including: A CNN convolutional neural network is used to extract features from the input license plate image to be identified, generating a shallow feature map and a deep feature map; the shallow feature map includes the edge features of the characters, the texture features of the characters, and the local shape features of the license plate image to be identified; the deep feature map includes the overall shape features of the license plate, the semantic information of the characters, and the license plate category in the license plate image to be identified; The cross-attention module dynamically fuses the semantic information of different levels in the shallow feature map and the deep feature map, and outputs the hierarchically fused license plate feature map; The multi-scale feature fusion module is used to perform scale fusion on the features of different scales in the hierarchically fused license plate feature map.

6. A license plate recognition method based on deep learning according to claim 5, characterized in that: The cross-attention module dynamically fuses the semantic information of different levels in the shallow feature map and the deep feature map. The output level fused license plate feature map specifically includes: Generate query matrix Q based on shallow feature map, and generate key matrix K and value matrix V based on deep feature map; Calculate the attention score between the query matrix Q and the key matrix K. The attention score is used to represent the correlation between each position in the shallow feature map and each position in the deep feature map. The calculation method of the attention score is as follows: Among them, Attention Scores is the attention score, d k is the dimension of the key matrix K, and T is the transpose of the key matrix K; The value matrix V is weighted and summed according to the attention score to generate a hierarchically fused license plate feature map.

7. The method for license plate recognition based on deep learning according to claim 5, characterized in that: The multi-scale feature fusion module is used to fuse the different scale features in the hierarchical fusion license plate feature map. Specifically, the following steps are performed: The features of the hierarchically fused license plate feature map are extracted through a CNN convolutional neural network to obtain a low-resolution feature map and a first high-resolution feature map; wherein the low-resolution feature map is a feature map obtained by extracting the hierarchically fused license plate feature map through a deep layer of the CNN convolutional neural network, and the first high-resolution feature map is a feature map obtained by extracting the hierarchically fused license plate feature map through a shallow layer of the CNN convolutional neural network; The low-resolution feature map is gradually generated into a second high-resolution feature map through upsampling operation; The second high-resolution feature map F upsample With the first high-resolution feature map F low-level Add tensors to generate scale fusion feature map F out .

8. The method for license plate recognition based on deep learning according to claim 7, characterized in that: The extracted license plate character area is input into the trained improved LPRNet network model, and character recognition specifically includes: Scale fusion feature map F out Perform global average pooling to obtain statistical information for each channel; The weight of each channel is generated through the fully connected layer based on the statistical information of each channel: The weight of each channel generated is fused with the scale feature map F out Multiply each channel in to get the final feature map F output .

9. A license plate recognition method based on deep learning according to any one of claims 3 to 8, characterized in that: Before determining the license plate type and locating the key points on the license plate based on the detected license plate area, the following steps are also included: When the illumination intensity of the collected vehicle image is weak, a weak-light license plate image enhancement network based on a differential convolutional neural network is constructed. The texture information of the license plate image is decoupled into four directions: horizontal, vertical, and diagonal, and texture enhancement is performed on weak-light images of different scales.

10. A license plate recognition system based on deep learning, characterized in that: include: An acquisition module acquires vehicle images, performs preprocessing operations on the acquired vehicle images, and establishes a training data set based on the preprocessed vehicle images; The training module trains the YOLOv8 algorithm model and the improved LPRNet network model based on the training data set; The detection module obtains and preprocesses the current vehicle image to be identified, performs license plate target detection on the preprocessed vehicle image to be identified using the trained YOLOv8 algorithm model, and outputs the license plate area; The segmentation and extraction module determines the license plate type and locates the key points on the license plate based on the detected license plate area; based on the license plate type and key point locations, it segments the license plate image and extracts the individual license plate character areas; The character recognition module inputs the extracted license plate character area into the trained improved LPRNet network model to perform character recognition; wherein the improved LPRNet network model includes a cross-attention module and a multi-scale feature fusion module. The cross-attention module is used to dynamically fuse semantic information at different levels in shallow and deep feature maps; the multi-scale feature fusion module is used to scale-fuse feature maps of different scales output by the cross-attention module; The output module outputs the license plate number composed of characters recognized by the improved LPRNet network model.

Images