An improved method for license plate detection and recognition under high-position video

Abstract

The present invention relates to the field of image processing technology, and in particular to an improved method for license plate detection and recognition under high-position video, comprising: first, using a high-definition smart camera to obtain image information of a parking section, using the LPD‑YOLOv8 algorithm to process the image information, and preliminarily detecting and selecting the license plate area; secondly, performing image preprocessing on the extracted license plate area; finally, using the improved LPR‑NET algorithm to recognize the corrected license plate image, and uploading the recognized license plate image information to the background. The method proposed in the present invention has a lightweight architecture and a novel method, and can be deployed with only one high-definition smart camera. Compared with traditional methods, this method has achieved significant improvements in detection speed and recognition accuracy, providing an efficient and accurate license plate recognition solution for roadside parking management on urban roads.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an improved method for license plate detection and recognition under high-position video. Background Art

[0002] License plate recognition technology is an image processing technology applied to dynamic video or static images of vehicles, aiming to realize automatic recognition of license plate numbers and license plate colors. This technology mainly includes four key steps: image acquisition, preprocessing, feature extraction, and classification recognition. It is an indispensable component of urban vehicle management systems. However, in roadside parking scenarios, due to uneven lighting conditions and the presence of obstructions, license plate images often appear blurred and deformed, which brings certain difficulties to accurate license plate recognition.

[0003] Traditional license plate recognition methods usually use digital image processing technology to locate, segment and identify license plates. Although they perform well in general environments, their detection performance is greatly affected when the lighting is uneven or the license plate is blurred. In recent years, the rise of deep learning technology has provided a new way to solve these problems.

[0004] When using deep learning methods, it is necessary to weigh the two key aspects of accuracy and real-time performance. The application of deep learning technology in the field of license plate recognition provides a powerful tool for improving accuracy. However, the resulting increase in computational complexity and model training time may have an adverse impact on real-time performance. In order to achieve a balance between accuracy and real-time performance, it is necessary to design lightweight deep learning models or adopt optimization strategies. The design of lightweight models can be achieved by reducing the number of network layers, the number of parameters, or adopting a specialized model structure. At the same time, in order to meet the requirements of real-time performance, technologies such as hardware acceleration or model pruning can also be introduced to reduce the computational burden of the inference stage.

[0005] In summary, improving the accuracy and efficiency of license plate recognition is an urgent problem to be solved. Traditional methods have limitations in complex scenes, uneven lighting or blurred license plates, so more advanced and adaptable technologies are needed to meet these challenges. Deep learning technology has become an effective means to solve these problems with its powerful feature learning ability, but in application, it is necessary to comprehensively consider accuracy and real-time performance, and find the best balance through model design and optimization strategies to meet the needs of different scenarios. Summary of the invention

[0006] The purpose of this invention is to solve the shortcomings existing in the prior art and propose an improved method for license plate detection and recognition under high-position video. Through deep learning technology, fast and accurate license plate detection and recognition can be achieved, and the interference of external factors can be effectively eliminated, thereby realizing end-to-end license plate recognition and other problems, thereby providing more powerful support for urban vehicle management.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] An improved method for license plate detection and recognition under high-position video includes the following steps:

[0009] S1. The high-position camera on the roadside collects monitoring road section information and captures vehicle behavior. If a vehicle enters or exits a parking space, the license plate recognition algorithm is activated;

[0010] S2. Train the LPD-YOLOv8 license plate detection network and the improved LPR-NET license plate recognition network. For vehicles that park in or out of parking spaces, use the LPD-YOLOv8 algorithm to process the vehicle area, preliminarily detect and select the license plate area.

[0011] S3, performing image preprocessing on the extracted license plate area;

[0012] S4. Use the improved LPR-NET algorithm to recognize the corrected license plate image, and upload the recognized license plate image information to the background.

[0013] Preferably, in step S2, the LPD-YOLOv8 algorithm includes replacing the C2f module in the original YOLOv8 algorithm feature extraction network with a C2f-PConv module, and introducing an attention mechanism module before the backbone network SPPF module.

[0014] Preferably, the specific method of replacing the C2f module with the C2f-PConv module is: replacing the convolution Conv in the C2f module with a partial convolution PConv to reduce redundant calculations and memory accesses, thereby achieving more effective performance in spatial feature extraction.

[0015] Preferably, the specific method of introducing the attention mechanism module before the backbone network SPPF module is: introducing the CA attention mechanism before the backbone network SPPF module, the CA attention mechanism decomposes the channel attention into two one-dimensional features by coordinating attention, and aggregates features along two spatial directions respectively; the generated feature map is encoded into a pair of direction-aware and position-sensitive attention maps, which can be complementarily applied to the input feature map to enhance the representation of the object of attention.

[0016] Preferably, in step S3, image preprocessing includes the following steps:

[0017] S31, using weighted average method to perform image grayscale processing;

[0018] S32, performing image binarization processing using the maximum inter-class variance method;

[0019] S33, performing image tilt correction by perspective transformation.

[0020] Preferably, in step S4, the improved LPR-NET algorithm includes: redesigning the license plate correction network of the original LPR-NET algorithm, and introducing a CBAM attention mechanism into the input license plate recognition network.

[0021] Preferably, the specific method of redesigning the license plate correction network of the original LPR-NET algorithm is: improving the feature extraction network in the LPR-NET network, replacing the license plate correction network STN with WPOD-Net; WPOD-Net further adds image edge knowledge to more accurately determine the license plate contour information; WPOD-Net, as a new license plate correction network, makes the license plate contour information more accurate and reliable by integrating image edge knowledge.

[0022] Preferably, the specific method of adding image edge knowledge to WPOD-Net is: using the Prewitt filter as a convolutional neural network layer, combining the edge information with the old information of the original network, and adding it to the original WPOD-Net as an embedding vector; through fusion with the original network information, the new embedding vector captures the characteristics of the image more comprehensively, especially the fusion of edge information makes the network more sensitive to the contour of the target.

[0023] Preferably, the specific method of introducing the CBAM attention mechanism into the input license plate recognition network is: for the intermediate feature map generated by the license plate recognition network, the CBAM attention mechanism infers the attention map along two independent dimensions (channel and space) through the CAM module and the SAM module; the CAM module assigns different weights to different channels to capture the correlation information between channels; the SAM module pays attention to the spatial relationship between different regions in the feature map; finally, the two attention maps obtained by the CAM module and the SAM module are multiplied with the input feature map to perform adaptive feature optimization.

[0024] Preferably, the improved LPD-YOLOv8 license plate detection network and the improved LPR-NET license plate recognition network are jointly trained, and the specific method includes:

[0025] The license plate detection and recognition networks are jointly trained. In each training iteration, a joint loss function is used, which consists of the detection loss of LPD-YOLOv8 and the recognition loss of LPR-NET. The losses of the two are weighed to ensure the accuracy of the license plate position and the high quality of the character information. In each training iteration, the license plate position information detected by LPD-YOLOv8 is passed to the improved LPR-NET for the recognition of license plate characters.

[0026] Preferably, the improved license plate detection and recognition method under high-position video proposed in the present invention takes the roadside parking scene as the actual application scene, and comprises:

[0027] The roadside parking scenario is specifically a parking scenario in which parking spaces are marked on both sides of the road for vehicles to park, and high-position cameras are deployed above roadside lamp poles and other brackets to capture vehicle behavior and vehicle information on the road.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention improves the YOLOv8 algorithm in the environment of license plate detection and proposes the LPD-YOLOv8 algorithm. Compared with the YOLOv8 algorithm, it has faster detection speed, higher detection accuracy and stronger robustness in the task of license plate recognition.

[0030] 2. The present invention adopts the C2f-PConv module to replace the original C2f module, which reduces redundant calculations and memory accesses and can more effectively extract spatial features, thereby improving the accuracy and speed of the license plate detection algorithm.

[0031] 3. Based on the original YOLOv8 algorithm, the present invention introduces a CA attention mechanism module into the backbone network, so that the model pays more attention to license plate detection, thereby improving the accuracy of the license plate detection algorithm.

[0032] 4. The present invention improves the license plate correction network in the LPR-NET algorithm, replaces the STN network with the WPOD-Net network, adds image edge knowledge to the WPOD-Net, and introduces the CBAM attention mechanism, thereby reducing the algorithm calculation complexity and improving the accuracy of license plate image correction.

[0033] 5. The present invention only requires one high-definition camera to be deployed, which saves hardware costs to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is the overall flow chart of the present invention;

[0035] Figure 2 It is a structural principle diagram of the C2f-PConv network module in the present invention;

[0036] Figure 3 It is the flow chart of the improved LPR-NET algorithm in the present invention;

[0037] Figure 4 The improved WPOD-Net network structure principle diagram in the present invention;

[0038] Figure 5 Schematic diagram of a roadside parking scene in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the advantages and features of the present invention, thereby making a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, an improved method for license plate detection and recognition under high-position video includes the following steps:

[0041] S1. The high-position camera on the roadside collects monitoring road section information and captures vehicle behavior. If a vehicle enters or exits a parking space, the license plate recognition algorithm is activated;

[0042] S2. For vehicles that park in or out of parking spaces, use the LPD-YOLOv8 algorithm to process the vehicle area, preliminarily detect and select the license plate area;

[0043] S3, performing image preprocessing on the extracted license plate area;

[0044] S4. Use the improved LPR-NET algorithm to recognize the corrected license plate image, and upload the recognized license plate image information to the background.

[0045] Specifically, in step S2, the LPD-YOLOv8 algorithm includes replacing the C2f module in the original YOLOv8 algorithm feature extraction network with a C2f-PConv module, and introducing an attention mechanism module before the backbone network SPPF module.

[0046] Specifically, Figure 2 As shown, it is a schematic diagram of the C2f-PConv network module structure of the present invention. The C2f-PConv network replaces the convolution (Conv) in the C2f module with a partial convolution (PConv). Among them, PConv is called a partial convolutional neural network, which can more effectively extract spatial features by reducing redundant calculations and memory accesses at the same time.

[0047] FLOPs of PConv: h×w×k 2 ×C p 2 ;

[0048] PConv memory access: h×w×2C p +k 2 ×C p2 ≈h×w×2C p ;

[0049] In the above formula, h and w are the width and height of the feature map; k is the size of the convolution kernel; C p is the number of channels for conventional convolution, which is usually Therefore, the FLOPs of PConv is only 1 / 16 of that of regular convolution, and the number of memory accesses of PConv is only 1 / 4 of that of regular convolution. p The channel does not participate in the calculation, so there is no need to access the memory.

[0050] The bottleneck module in the YOLOv8 feature extraction network C2f is composed of two Conv convolutions in sequence. Replacing Conv with PConv can reduce the network complexity of the C2f module and also help to extract license plate information features more effectively.

[0051] Among them, the specific method of introducing the attention mechanism module before the backbone network SPPF module is: introducing the CA attention mechanism before the backbone network SPPF module, the CA attention mechanism decomposes the channel attention into two one-dimensional features by coordinating attention, and aggregates features along two spatial directions respectively; the generated feature map is encoded into a pair of direction-aware and position-sensitive attention maps, which can be complementarily applied to the input feature map to enhance the representation of the object of attention.

[0052] Among them, the CA attention mechanism is an attention mechanism used to enhance the deep learning model's understanding of the spatial structure of the input data. First, receive the feature map [C, H, W] after the backbone network feature extraction; then perform two global average pooling on the input feature map, one in the width direction and one in the height direction, to obtain a feature map [C, H, 1] with average pooling in width and a feature map [C, 1, W] with average pooling in height, and then merge the above two feature maps to obtain a new feature layer with a shape of [C, 1, H + W]. After 1×1 convolution dimensionality reduction, normalization and activation function, a richer representation f is obtained; then the feature map f is convolved with a kernel of 1×1 according to the original height and width to obtain feature maps F with the same number of channels as the original. h and F w ; After the Sigmoid activation function, the attention weights g of the feature map in height and width are obtained respectively h and the attention weight g in the width direction w ; Finally, by multiplying the original feature map with weighted calculations, we will get the final feature map with attention weights in the width and height directions.

[0053] Among them, the global average pooling in the width direction is:

[0054] Among them, the global average pooling in the height direction is:

[0055] Among them, a richer representation f is obtained by 1×1 convolution dimensionality reduction, normalization and activation function: f = δ(F1([z h ,z w ]));

[0056] Among them, the height attention weight g h =σ(F h (f h ));

[0057] Width attention weight g w =σ(F w (f w ));

[0058] Among them, the final feature map with attention weights in the width and height directions is:

[0059]

[0060] Where C is the number of channels, representing the different feature channels in the feature map; H, h are the heights, representing the vertical dimensions of the feature map; W, w are the widths, representing the horizontal dimensions of the feature map.

[0061] Specifically, in step S3, the framed license plate image is preprocessed, including the following steps:

[0062] S31, using weighted average method to perform image grayscale processing;

[0063] S32, performing image binarization processing using the maximum inter-class variance method;

[0064] S33, performing image tilt correction by perspective transformation.

[0065] like Figure 3 Shown is the flow chart of the LPR-NET algorithm of the present invention, which uses the improved LPR-NET algorithm to recognize the corrected license plate image, redesigns the license plate correction network of the original LPR-NET algorithm, and introduces the CBAM attention mechanism into the input license plate recognition network.

[0066] Specifically, the specific method of redesigning the license plate correction network of the original LPR-NET algorithm is: improving the feature extraction network in the LPR-NET network, replacing the license plate correction network STN with WPOD-Net; WPOD-Net further adds image edge knowledge to more accurately determine the license plate contour information; WPOD-Net, as a new license plate correction network, makes the license plate contour information more accurate and reliable by integrating image edge knowledge.

[0067] Among them, the STN network is mainly composed of three parts: parameter prediction: LocalisationNet; coordinate mapping: GridGenerator; pixel collection: Sampler. Localisation Net can realize image translation, scaling, cropping and rotation; Grid Generator realizes the correspondence of pixel coordinates; Sampler realizes the differentiability of coordinate solution. STN can only accept fixed-size input. When inputting with a larger size, the number of parameters of the fully connected layer will increase sharply. It cannot process multiple transformations at the same time, but performs a single spatial transformation on the entire input.

[0068] Among them, WPOD-NET is called the Warped Planar Object Detection Network, which includes 21 convolutional layers, 14 of which are in the residual block. All convolutions are of size 3×3, and the entire network uses the ReLU activation function except for the detection module. There are four maximum pooling layers with a step size of 2, and the resolution is reduced by 16 times. Finally, the detection module has two parallel convolutional layers: one uses a softmax activation function for classification; the other does not have an activation function, and the network outputs 13×13×8 parameters (208 as the input resolution) for regressing affine transformation parameters. WPOD-NET returns the bounding box area surrounding the license plate and displays the license plate as a frontal view. It can accept inputs of any size and is lighter than the STN network, reducing computational complexity.

[0069] Among them, the ReLU activation function formula is: The softmax activation function formula is:

[0070]

[0071] like Figure 4 As shown in the figure, it is the schematic diagram of the improved WPOD-Net network structure. In WPOD-NET, the Prewitt filter is used as the convolutional neural network layer to combine the edge information with the old information of the original network and add it to the original WPOD-NET as an embedding vector.

[0072] Among them, the Prewitt filter is a linear filter that applies a 3×3 matrix to each pixel in the image to calculate the edge strength of the pixel. The 3×3 matrix is ​​the Prewitt operator, which contains two 3×3 matrices, which are used to calculate the edge strength in the horizontal direction and the vertical direction respectively.

[0073] Among them, the Prewitt operator in the horizontal direction is calculated: Compute the Prewitt operator in the vertical direction:

[0074] For the intermediate feature maps generated by the license plate recognition network, the CBAM module infers attention maps along two independent dimensions (channel and spatial) through CAM and SAM.

[0075] CAM: Input feature map H×W×C is subjected to maximum pooling and average pooling to obtain a 1×1×C feature map; input two-layer MLP neural network: the number of neurons in the first layer (r is the scaling rate), the activation function is ReLU, the number of neurons in the second layer is C, and the activation function is the sigmoid function; the element-wise addition operation, followed by the sigmoid activation operation, generates the final channel attention feature.

[0076] SAM: The spatial attention module uses the feature map output by the channel attention module as the input feature map of this module. The feature map H×W×C is subjected to channel-based maximum pooling and average pooling to obtain two H×W×1 feature maps, which are concatenated based on the channel. After a 7×7 convolution operation, the dimension is reduced to H×W×1, and then the spatial attention feature is generated through sigmoid. Finally, the vector is multiplied by the input feature map of the module to obtain the final generated feature.

[0077] Among them, the sigmoid activation function formula is:

[0078] Among them, maximum pooling is to select the maximum value in the image area as the value of the area after pooling. During back propagation, the gradient is back propagated through the maximum value of the forward propagation process, and the gradient at other positions is 0; average pooling is to use the average value in the selected image area as the value of the area after pooling.

[0079] Among them, assuming that the input size is i, the output size is o, the kernel size is k, and the step size is s, the following formula is satisfied:

[0080] CAM assigns different weights to each layer of channels to represent different levels of importance, while SAM assigns different weight values ​​to different areas of the feature map from a planar perspective. By adding CAM and SAM attention mechanisms to the improved WPOD-NET, features can be extracted from two different dimensions, providing more accurate and clear license plate feature information for the next step of character recognition.

[0081] Embodiment: A roadside parking scenario is specifically a parking scenario in which parking spaces are marked on both sides of the road for vehicles to park, and high-position cameras are deployed above brackets such as roadside lamp poles to capture vehicle behavior and vehicle information on the road.

[0082] The present invention improves the existing license plate detection and recognition algorithm based on a deep learning algorithm, and can realize end-to-end network training, that is, jointly training the license plate detection network LPD-YOLOv8 and the improved license plate character recognition network LPR-NET, and then deploying them to a high-definition camera, thereby realizing end-to-end real-time license plate detection and recognition.

[0083] Specifically, we first collect the dataset and use the Labelme annotation tool to annotate the license plate, and then jointly train the license plate detection and recognition networks. In each training iteration, a joint loss function is used, which consists of the detection loss of LPD-YOLOv8 and the recognition loss of LPR-NET. The losses of the two are weighed to ensure the accuracy of the license plate position and the high quality of the character information. In each training iteration, the license plate position information detected by LPD-YOLOv8 is passed to the improved LPR-NET for the recognition of license plate characters.

[0084] like Figure 5 The figure shows a schematic diagram of a roadside parking scene of the present invention. The improved license plate detection and recognition method under high-position video proposed by the present invention takes the roadside parking scene as the actual application scene. The roadside parking scene is specifically a parking scene where parking spaces are marked on both sides of the road for vehicles to park, and high-position cameras are deployed above brackets such as roadside lamp poles to capture vehicle behavior and vehicle information on the road. The camera installation height is 4 to 6 meters, and the ratio of the pole distance L to the road width R

[0085] In summary, the method proposed in the present invention has a lightweight architecture, a novel method, and can be deployed with only one high-definition smart camera. Compared with traditional methods, this method has achieved significant improvements in detection speed and recognition accuracy, providing an efficient and accurate license plate recognition solution for urban roadside parking management.

[0086] The description and practice disclosed in the present invention are easy to think and understand for ordinary technicians in the technical field, and several improvements and modifications can be made without departing from the principles of the present invention. Therefore, modifications or improvements made without departing from the spirit of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An improved method for license plate detection and recognition under high-position video, characterized in that: The steps include: S1. The high-position camera on the roadside collects monitoring road section information and captures vehicle behavior. If a vehicle enters or exits a parking space, the license plate recognition algorithm is activated; S2. Train the LPD-YOLOv8 license plate detection network and the improved LPR-NET license plate recognition network. For vehicles that park in or out of parking spaces, use the LPD-YOLOv8 algorithm to process the vehicle area, preliminarily detect and select the license plate area. S3, performing image preprocessing on the extracted license plate area; S4, using the improved LPR-NET algorithm to recognize the corrected license plate image, and uploading the recognized license plate image information to the background; In step S2, the LPD-YOLOv8 algorithm includes replacing the C2f module in the original YOLOv8 algorithm feature extraction network with the C2f-PConv module, and introducing an attention mechanism module before the backbone network SPPF module; The specific method of replacing the C2f module with the C2f-PConv module is: replacing the convolution Conv in the C2f module with the partial convolution PConv; The specific method of introducing the attention mechanism module before the backbone network SPPF module is: introducing the CA attention mechanism before the backbone network SPPF module, the CA attention mechanism decomposes the channel attention into two one-dimensional features by coordinating attention, and aggregates the features along two spatial directions respectively; the generated feature map is encoded into a pair of direction-aware and position-sensitive attention maps, and the attention map can be complementarily applied to the input feature map to strengthen the representation of the object of attention; In step S4, the improved LPR-NET algorithm includes: redesigning the license plate correction network of the original LPR-NET algorithm, and introducing the CBAM attention mechanism in the input license plate recognition network; The specific method of redesigning the license plate correction network of the original LPR-NET algorithm is: improving the feature extraction network in the LPR-NET network, replacing the license plate correction network STN with WPOD-Net; WPOD-Net further adds image edge knowledge to more accurately determine the license plate contour information; WPOD-Net, as a new license plate correction network, makes the license plate contour information more accurate and reliable by integrating image edge knowledge; The specific method of adding image edge knowledge to WPOD-Net is: using the Prewitt filter as a convolutional neural network layer, combining edge information with old information from the original network, and adding it to the original WPOD-Net as an embedding vector; by fusing with the original network information, the new embedding vector captures the features of the image more comprehensively, especially the fusion of edge information makes the network more sensitive to the contours of the target; The specific method of introducing the CBAM attention mechanism into the input license plate recognition network is: for the intermediate feature map generated by the license plate recognition network, the CBAM attention mechanism infers the attention map along two independent dimensions through the CAM module and the SAM module; the CAM module assigns different weights to different channels to capture the correlation information between channels; the SAM module focuses on the spatial relationship between different areas in the feature map; finally, the two attention maps obtained by the CAM module and the SAM module are multiplied with the input feature map for adaptive feature optimization.

2. According to the improved method for license plate detection and recognition under high-position video according to claim 1, it is characterized in that: In step S3, image preprocessing includes the following steps: S31, using weighted average method to perform image grayscale processing; S32, performing image binarization processing using the maximum inter-class variance method; S33, performing image tilt correction by perspective transformation.

3. The improved method for license plate detection and recognition under high-position video according to claim 1 is characterized in that: The improved LPD-YOLOv8 license plate detection network is jointly trained with the improved LPR-NET license plate recognition network. The specific methods include: The license plate detection and recognition networks are jointly trained. In each training iteration, a joint loss function is used, which consists of the detection loss of LPD-YOLOv8 and the recognition loss of LPR-NET. The losses of the two are weighed to ensure the accuracy of the license plate position and the high quality of the character information. In each training iteration, the license plate position information detected by LPD-YOLOv8 is passed to the improved LPR-NET for the recognition of license plate characters.

Images