License plate recognition method and system for automatic highway toll collection

Abstract

The invention discloses a license plate recognition method and system for automatic highway toll collection, and the method comprises the steps: collecting a vehicle image through a plurality of image sensors, sequentially carrying out the noise reduction, contrast enhancement and white balance adjustment of the vehicle image, and generating an initial vehicle image; synthesizing different light source images in the initial vehicle image by using an image fusion technology; performing coarse positioning on a license plate area in the synthesized vehicle image based on an improved YOLOv7 model in combination with a Canny edge detection algorithm; and segmenting license plate characters in the license plate region image by using a semantic segmentation model of U-Net, and performing end-to-end character recognition through a lightweight CRNN convolutional recurrent neural network model to obtain a license plate recognition result. And the recognition speed is greatly improved while the recognition accuracy is ensured.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a method and system for automatically recognizing license plates for toll collection on expressways. Background Art

[0002] Traditional license plate recognition technology primarily relies on a single camera to capture images, which presents numerous limitations in complex highway environments. Highway lighting conditions are complex and varied, ranging from bright sunlight to the dim light of tunnels and nighttime headlights. These conditions can cause license plate images to be overexposed, underexposed, or reflective, making it difficult to clearly visualize the license plate characters and resulting in low recognition accuracy. Furthermore, vehicles travel at high speeds on highways, and the insufficient frame rate of conventional image capture equipment can easily cause image blur and smearing, leading to a loss of license plate details and inefficient recognition. Summary of the Invention

[0003] The purpose of the present invention is to solve the above problems and to design a method and system for automatic license plate recognition for highway toll collection.

[0004] To achieve the above-mentioned purpose, the technical solution of the present invention is as follows: further, in the above-mentioned automated highway toll license plate recognition method, the highway toll license plate recognition method comprises the following steps:

[0005] capturing vehicle images using multiple image sensors, performing noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and synthesizing different light source images in the initial vehicle image using image fusion technology to obtain a synthesized vehicle image;

[0006] The license plate area in the synthetic vehicle image is roughly located based on the improved YOLOv7 model combined with the Canny edge detection algorithm. The Transformer module is introduced to extract long-range feature information. The OCR text detection algorithm is used to assist in determining whether it is a real license plate, thereby obtaining a license plate area image.

[0007] The U-Net semantic segmentation model is used to segment the license plate characters in the license plate area image, and the morphological repair algorithm and contour tracking technology are used to correct the character adhesion and breakage. The lightweight CRNN convolutional recurrent neural network model is used to perform end-to-end character recognition to obtain the license plate recognition result.

[0008] Furthermore, in the above-mentioned method for automatic highway toll license plate recognition, the vehicle image is captured by multiple image sensors, noise reduction, contrast enhancement, and white balance adjustment are sequentially performed on the vehicle image to generate an initial vehicle image, and different light source images in the initial vehicle image are synthesized using image fusion technology to obtain a synthesized vehicle image, including:

[0009] A vehicle image is captured by multiple image sensors, pixel weights are calculated based on the two dimensions of spatial distance and grayscale difference using a bilateral filtering algorithm, and the filtering strength is adjusted according to the image noise level to perform noise reduction processing on the vehicle image to obtain a noise-reduced vehicle image;

[0010] Using CLAHE adaptive histogram equalization to divide the denoised vehicle image into 8x8 small grids, and independently enhancing the contrast of each grid to obtain an enhanced vehicle image;

[0011] Counting pixels with grayscale values ​​between 128 and 192 in the enhanced vehicle image, calculating the average RGB value, establishing a dynamic correction library for different light sources, and performing correction using the correction library to obtain a corrected vehicle image;

[0012] Using the Laplace pyramid fusion technology, five rectified vehicle images of the same vehicle at different times and under different illumination are combined into one image to obtain a composite vehicle image.

[0013] Furthermore, in the above-mentioned automated highway toll license plate recognition method, the improved YOLOv7 model is combined with the Canny edge detection algorithm to roughly locate the license plate area in the synthetic vehicle image, and the Transformer module is introduced to extract long-range feature information, including:

[0014] A Transformer module is embedded in the deep feature extraction stage of the backbone network of the original YOLOv7 model, and the multi-head self-attention mechanism in the Transformer module focuses on the correlation between different regions;

[0015] The feature map output by YOLOv7 is preprocessed using the Canny edge detection algorithm, the image is smoothed using Gaussian filtering, and a double threshold algorithm is used to distinguish between definite edges and possible edges;

[0016] Perform edge density analysis on each candidate detection frame generated by YOLOv7 and calculate the ratio of edge pixels within the frame. If the ratio is lower than the threshold, the area is determined to be the vehicle body and background and is excluded.

[0017] Check the integrity of the edge contour. If the edge within the detection frame is broken, exclude it and obtain a roughly positioned license plate image.

[0018] Furthermore, in the above-mentioned method for automatic highway toll license plate recognition, the method of using an OCR text detection algorithm to assist in determining whether the license plate is a real license plate and obtaining a license plate area image includes:

[0019] After using the EAST algorithm to quickly locate the text area in the image, the license plate format rules are verified and the number of characters is checked;

[0020] If the number of characters in the detected text area is significantly deviated, it will be directly excluded. The character arrangement characteristics will be analyzed. If the Chinese character is at the first position followed by letters and the overall aspect ratio meets the license plate standard, the detection area image will be output;

[0021] The character format of the detection area image is verified. If the detected text area meets the three conditions at the same time, it is confirmed as the real license plate area and the license plate area image is obtained.

[0022] Furthermore, in the above-mentioned method for automatic highway toll license plate recognition, the segmentation of the license plate characters in the license plate area image using a U-Net semantic segmentation model includes:

[0023] Separate each character in the license plate area image from the background using the U-Net semantic segmentation model;

[0024] The upper layer of the semantic segmentation model is used to compress the image through multiple convolutions and pooling to extract deep semantic features;

[0025] The lower layer of the semantic segmentation model is used for upsampling and deconvolution to gradually restore the feature map to its original size. At the same time, the detailed information of the upper layer is combined with the global information of the lower layer through jump connections to obtain the license plate characters.

[0026] Furthermore, in the above-mentioned method for automatic highway toll license plate recognition, the correction of character adhesion and breakage using a morphological repair algorithm and contour tracking technology includes:

[0027] The license plate characters are binarized, and then the morphological opening operation is used to remove noise and small adhesion areas, and the closing operation is used to connect slightly broken characters; the contour detection is performed on the repaired image to extract the contour of each character.

[0028] Furthermore, in the above-mentioned method for automatic highway toll license plate recognition, the end-to-end character recognition is performed using a lightweight CRNN convolutional recurrent neural network model to obtain a license plate recognition result, including:

[0029] Each convolutional layer is followed by a ReLU activation function and a maximum pooling layer to extract the visual features of the characters; a two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence; and a fully connected layer and a CTC loss function are used to convert the feature sequence into a character sequence.

[0030] Furthermore, in an automated highway toll license plate recognition system, the highway toll license plate recognition system includes the following modules:

[0031] An image acquisition and synthesis module is configured to acquire vehicle images using multiple image sensors, perform noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and synthesize different light source images in the initial vehicle image using image fusion technology to obtain a synthesized vehicle image;

[0032] A license plate region extraction module is used to roughly locate the license plate region in the synthetic vehicle image based on an improved YOLOv7 model combined with a Canny edge detection algorithm, introduce a Transformer module to extract long-range feature information, and use an OCR text detection algorithm to assist in determining whether it is a real license plate, thereby obtaining a license plate region image;

[0033] The license plate character recognition module is used to segment the license plate characters in the license plate area image using the U-Net semantic segmentation model, correct the adhesion and breakage of characters using the morphological repair algorithm and contour tracking technology, and perform end-to-end character recognition through the lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

[0034] Furthermore, in an automated highway toll collection license plate recognition system, the license plate character recognition module includes the following units:

[0035] The binarization unit is used to binarize the license plate characters, then use morphological opening operations to remove noise and small adhesion areas, and use closing operations to connect slightly broken characters; contour detection is performed on the repaired image to extract the contour of each character.

[0036] Furthermore, in an automated highway toll collection license plate recognition system, the license plate character recognition module includes the following units:

[0037] The character conversion unit is used to extract the visual features of characters by following each convolutional layer with a ReLU activation function and a maximum pooling layer. A two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence, and a fully connected layer and a CTC loss function are used to convert the feature sequence into a character sequence.

[0038] Its beneficial effects are as follows: 1. It can significantly improve image quality, ensuring clear license plate images with complete details even under extreme lighting conditions. 2. It can effectively eliminate interference from other vehicle components (headlights, logos) and complex backgrounds, and accurately lock the license plate area even if the license plate is tilted at a certain angle or partially obscured. 3. While ensuring recognition accuracy, it greatly improves recognition speed, achieving end-to-end fast and accurate character recognition. This technical solution can cope with various complex situations such as license plate aging and stain obscuration, and the accuracy of license plate recognition is improved. 4. Through algorithm optimization and model lightweighting, it meets the real-time requirements of highway toll stations for license plate recognition. Vehicles can complete license plate recognition and toll collection without having to stop for a long time, significantly improving the traffic efficiency of toll stations, reducing vehicle waiting time in queues, and effectively alleviating congestion problems at toll stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those skilled in the art by reading the following detailed description of the preferred embodiment.The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0040] Figure 1 This is a schematic diagram of a first embodiment of a method for automatic highway toll collection;

[0041] Figure 2 Schematic diagram of a second embodiment of a method for automatic highway toll collection;

[0042] Figure 3 This is a schematic diagram of a first embodiment of an automated highway toll collection license plate recognition system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1As shown, an automated highway toll license plate recognition method includes the following steps:

[0046] Step 101: Capture vehicle images using multiple image sensors, perform noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and use image fusion technology to synthesize images of different light sources in the initial vehicle image to obtain a synthesized vehicle image.

[0047] Specifically, in this embodiment, a vehicle image is captured by multiple image sensors, pixel weights are calculated based on the two dimensions of spatial distance and grayscale difference using a bilateral filtering algorithm, and the filtering strength is adjusted according to the image noise level to perform noise reduction processing on the vehicle image to obtain a noise-reduced vehicle image.

[0048] CLAHE adaptive histogram equalization is used to divide the denoised vehicle image into 8x8 small grids, and the contrast of each grid is independently enhanced to obtain an enhanced vehicle image;

[0049] Statistically enhance the pixels in the grayscale value range of 128-192 in the vehicle image, calculate the average RGB value, establish a dynamic correction library for different light sources, and use the correction library to perform correction to obtain a corrected vehicle image;

[0050] Using the Laplace pyramid fusion technology, five rectified vehicle images of the same vehicle at different times and under different illumination are combined into one image to obtain a composite vehicle image.

[0051] Specifically:

[0052] (1) Image sensor deployment and parameter configuration;

[0053] Multiple high dynamic range (HDR) image sensors are strategically deployed at key locations on highways, such as toll booth entrances and exits, and above each lane. The sensor resolution is set to at least 2 megapixels to ensure clear capture of vehicle details, including license plate characters. The frame rate is set to at least 30 frames per second. This ensures that at high speeds (120 km / h), at least 10 images are captured for each vehicle passing through the sensor coverage area to avoid missed images. The sensor's dynamic range should be greater than 120 dB to accommodate complex lighting conditions on highways, ranging from strong direct sunlight to low light levels in tunnels.

[0054] (2) Image preprocessing process;

[0055] Noise Reduction: A bilateral filtering algorithm is used to reduce noise in the captured vehicle images. Bilateral filtering not only effectively removes Gaussian and salt-and-pepper noise, but also preserves edge information, which is crucial for subsequent license plate feature extraction. The algorithm sets the standard deviation σ_s of the spatial Gaussian kernel to 3-5, and the standard deviation σ_r of the grayscale Gaussian kernel to 15-25. These parameters are automatically adjusted based on the image noise level. Specifically, for each pixel in the image, a weighted average of the pixels in its surrounding neighborhood is calculated, with the weight determined by both spatial distance and grayscale difference. Contrast Enhancement: A CLAHE algorithm is used to enhance image contrast. CLAHE divides the image into multiple small regions (8x8 blocks) and performs histogram equalization on each region. The contrast enhancement is also limited to avoid noise amplification. A clipping limit of 2.0 is used to balance contrast enhancement and noise suppression. This process enhances the contrast between the characters in the license plate area and the background, facilitating subsequent detection and recognition. White balance adjustment: This white balance algorithm is based on the gray world hypothesis, assuming that the average RGB values ​​of the prevalent gray areas in the image are equal. The image's average R, G, and B values ​​are first calculated, and then the image is adjusted based on the ratio of these three values ​​to make the image's color reproduction closer to the real scene. For images under different light sources (daytime sunlight, nighttime street lights, tunnel lights), the average RGB values ​​are calculated, and a light source classification model is established to automatically select appropriate white balance adjustment parameters. Image fusion: The preprocessed images with different light sources are synthesized using the Laplacian pyramid fusion algorithm. First, each initial vehicle image is constructed into a Laplacian pyramid. The number of pyramid layers is determined by the image resolution, ranging from 5 to 7 layers. At each layer, the corresponding frequency bands of the different images are then weighted and fused. The weights are automatically calculated based on indicators such as image clarity and contrast.

[0056] Step 102: Based on the improved YOLOv7 model combined with the Canny edge detection algorithm, the license plate area in the synthetic vehicle image is roughly located. The Transformer module is introduced to extract long-range feature information. The OCR text detection algorithm is used to assist in determining whether it is a real license plate, and the license plate area image is obtained.

[0057] Specifically, in this embodiment, a Transformer module is embedded in the deep feature extraction stage of the backbone network of the original YOLOv7 model, and the multi-head self-attention mechanism in the Transformer module focuses on the relationship between different regions;

[0058] The feature map output by YOLOv7 is preprocessed using the Canny edge detection algorithm. The image is smoothed using a Gaussian filter, and a dual-threshold algorithm is used to distinguish between confirmed and possible edges. Edge density analysis is performed on each candidate detection frame generated by YOLOv7, and the ratio of edge pixels within the frame is calculated. If the ratio is below the threshold, the area is determined to be the vehicle body and background and excluded. The integrity of the edge contour is checked, and if the edge within the detection frame is broken, it is excluded to obtain a coarsely located license plate image. After quickly locating the text area in the image using the EAST algorithm, the license plate format rules are verified and the number of characters is checked. If the number of characters in the detected text area deviates significantly, it is directly excluded. The character arrangement characteristics are analyzed. If the Chinese character is the first character followed by letters and the overall aspect ratio meets the license plate standard, the detection area image is output. The character format of the detection area image is verified. If the detected text area meets all three conditions, it is confirmed to be the real license plate area, and the license plate area image is obtained.

[0059] Specifically:

[0060] (1) Improved YOLOv7 model design;

[0061] Based on the original YOLOv7 model, the backbone network was improved by introducing the Transformer module. Specifically, a multi-head self-attention mechanism was incorporated into the deep feature extraction stage of the backbone network. Each Transformer module consists of a multi-head self-attention layer and a feedforward neural network, and utilizes residual connections and layer normalization. The multi-head self-attention mechanism can capture long-range dependencies between different regions in the image, which is important for extracting features related to the license plate area and other vehicle components (body and lights). The improved model resizes the input image to 640x640 to balance detection speed and accuracy. During training, mixed precision training and data augmentation techniques (such as random flipping, scaling, rotation, and color dithering) are used to improve the model's generalization capabilities.

[0062] (2) Canny edge detection and region fusion;

[0063] The feature map output by the improved YOLOv7 model is input into the Canny edge detection module. The specific parameters for Canny edge detection are: Gaussian filter standard deviation σ = 1.4, low threshold set to 30, and high threshold set to 90. Using a dual-threshold algorithm and edge tracking, the image's edge contours are obtained. The edge detection results are then fused with the object detection bounding boxes of the YOLOv7 model. For each detection bounding box, the internal edge density and contour integrity are calculated to identify areas that may contain license plates. This allows the YOLOv7 model's detection results to be further refined using edge information, improving the accuracy of license plate location.

[0064] (3) OCR text detection and auxiliary discrimination;

[0065] The coarsely located license plate area is processed using an OCR text detection algorithm based on EAST (Efficient and Accurate Scene Text). The EAST model can quickly detect text regions within an image and output the coordinates and rotation angle of the text box. For each candidate license plate area, the text detection results are extracted to determine whether they conform to the license plate character format (Chinese license plates typically contain seven characters, including Chinese characters, letters, and numbers). Simultaneously, features such as the text area's aspect ratio and character spacing are calculated and matched against prior knowledge of real license plates. Only candidate areas that meet the license plate format and characteristics are confirmed as true license plate areas, eliminating interference from non-license plate areas and improving license plate detection accuracy.

[0066] Specifically, it also includes:

[0067] 1. Improved design of YOLOv7;

[0068] Based on the traditional YOLOv7, the following optimizations are made for the license plate detection task:

[0069] Multi-scale Feature Pyramid (FPN+PAN): Introducing Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) to enhance the detection capabilities of license plates of different sizes (small cars and large trucks).

[0070] Lightweight Transformer module: A lightweight Transformer module is embedded in the YOLOv7 backbone network, replacing some convolutional layers. The Transformer uses a self-attention mechanism to capture long-range dependent features of the license plate area (such as the continuity of license plate edges and the spatial relationship between characters), improving detection robustness in complex backgrounds.

[0071] Dynamic Non-Maximum Suppression (DynamicNMS): Optimizes the NMS algorithm and dynamically adjusts the suppression threshold of the detection box to avoid duplicate detection problems caused by tilted or obscured license plates.

[0072] License plate area coarse positioning process;

[0073] Input image processing: Input the synthetic vehicle image (fused with multiple light sources) generated in step 1 into the improved YOLOv7 model.

[0074] Feature extraction and target detection:

[0075] Use the improved YOLOv7 Backbone to extract image features and combine it with the Transformer module to enhance long-distance feature expression.

[0076] The detection head (DetectionHead) of YOLOv7 outputs the candidate box (BoundingBox) of the license plate area, including the license plate position (x, y, w, h) and confidence score.

[0077] Coarse positioning result output: Filter candidate boxes with confidence higher than the threshold (0.6) as the license plate area for preliminary positioning.

[0078] 2. Auxiliary positioning of Canny edge detection;

[0079] Application of Canny edge detection;

[0080] Image preprocessing: Grayscale the synthetic vehicle image to reduce the amount of calculation and highlight the edge information of the license plate.

[0081] Edge detection: Apply the Canny algorithm to detect significant edges in the image and generate an edge binary map.

[0082] Edge feature extraction:

[0083] Morphological operations (dilation, erosion) are used to enhance edge connectivity and remove noise interference.

[0084] Combined with the coarse positioning results of YOLOv7, edge features related to the license plate area (rectangular outline of the license plate) are extracted from the edge binary image.

[0085] Co-localization with YOLOv7;

[0086] Bounding box optimization: Match the YOLOv7 candidate box with the rectangular outline of the Canny edge detection, and correct the bounding box position output by YOLOv7 (adjust the offset coordinates).

[0087] License plate region verification: Edge features are used to determine whether the candidate frame conforms to the geometric characteristics of the license plate (aspect ratio close to 3:1 and good edge continuity). If not, the candidate frame is discarded.

[0088] 3. Auxiliary verification of OCR text detection;

[0089] Integration of OCR text detection;

[0090] Text region extraction: For the license plate area image jointly located by YOLOv7 and Canny, use the OCR text detection algorithm (CTPN or DBNet) to further detect the text region.

[0091] Character arrangement verification:

[0092] The OCR algorithm outputs the confidence level of the text area and the character arrangement direction (horizontal or tilted).

[0093] Determine whether the text conforms to the distribution pattern of license plate characters (character spacing is uniform and character types meet national standards).

[0094] Real license plate screening and logical reasoning: The OCR detection results and YOLOv7 confidence level are combined to comprehensively determine whether the candidate box is a real license plate. If the OCR detects a text area and the character arrangement conforms to license plate rules, the candidate box is retained. If the OCR detects no valid text or the character arrangement is abnormal (fragmented or overlapping characters), the candidate box is discarded. Output license plate area image: The final license plate area image is used as input for subsequent steps.

[0095] Step 103: Use the U-Net semantic segmentation model to segment the license plate characters in the license plate area image, use the morphological repair algorithm and contour tracking technology to correct the character adhesion and breakage, and perform end-to-end character recognition through the lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

[0096] Specifically, in this embodiment, each character in the license plate area image is separated from the background using a U-Net semantic segmentation model;

[0097] The upper layer of the semantic segmentation model is used to compress the image through multiple convolutions and pooling to extract deep semantic features;

[0098] The lower layer of the semantic segmentation model is used for upsampling and deconvolution to gradually restore the feature map to its original size. At the same time, the detailed information of the upper layer is combined with the global information of the lower layer through jump connections to obtain the license plate characters.

[0099] The license plate characters are binarized, and then the morphological opening operation is used to remove noise and small adhesion areas, and the closing operation is used to connect slightly broken characters; the contour detection is performed on the repaired image to extract the contour of each character.

[0100] Each convolutional layer is followed by a ReLU activation function and a maximum pooling layer to extract the visual features of the characters; a two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence; and a fully connected layer and a CTC loss function are used to convert the feature sequence into a character sequence.

[0101] Specifically:

[0102] (1) U-Net semantic segmentation model;

[0103] A U-Net model suitable for license plate character segmentation was constructed. The input was a license plate region image resized to 256x256. The network architecture consisted of a contracting path and an expanding path. The contracting path consisted of convolutional layers (3x3 convolution kernel, ReLU activation function, batch normalization) and max pooling layers to gradually extract deep features. The expanding path consisted of upsampling layers, convolutional layers, and skip connections to combine shallow details with deep semantic information. Finally, a 1x1 convolutional layer was used to output the segmentation result, with each pixel classified as license plate character, license plate background, or border. Training used a combination of cross-entropy loss and Dice coefficient loss to improve segmentation accuracy and the ability to detect small objects (single characters).

[0104] (2) Correction of character adhesion and breakage;

[0105] Morphological inpainting algorithm: After segmenting, the character image is first binarized. A morphological opening operation (erosion followed by dilation) is then used to remove noise and small areas of adhesion. A closing operation (dilation followed by erosion) is used to connect slightly broken characters. A 3x3 rectangular kernel is used as the structural element for the erosion and dilation operations. The number of operations is adjusted based on the size of the character and the degree of adhesion.

[0106] Contour tracking technology: Contour detection is performed on the restored image to extract the outline of each character. For contiguous characters, the shape and position of the outlines are used to determine whether they belong to the same character, and then a contour segmentation algorithm is used to separate them. For broken characters, the continuity and symmetry of the outlines are utilized to repair the broken parts through interpolation. During the processing, a priori shape models of the characters (common shapes of Chinese characters, letters, and numbers) are established to assist in determining the rationality of the outlines.

[0107] Specifically, it also includes:

[0108] 1. Character segmentation of the U-Net semantic segmentation model;

[0109] Improved design of the U-Net model;

[0110] Residual connection and attention mechanism: The residual module (ResidualBlock) and channel attention mechanism (SEBlock) are introduced into the traditional U-Net encoder-decoder structure to enhance the model's ability to capture the edge features of license plate characters.

[0111] Multi-scale feature fusion: Dilated convolution is used in the encoder stage to extract multi-scale features and retain the detailed information of characters; in the decoder stage, high-resolution features are fused with low-resolution features through skip connections to improve segmentation accuracy.

[0112] Dynamic boundary optimization: A weighted combination of DiceLoss and cross entropy loss is introduced into the loss function to dynamically adjust the model's focus on character boundary segmentation and reduce segmentation errors caused by character adhesion or breakage.

[0113] Character segmentation process;

[0114] Input image processing: Input the license plate area image output in step 2 into the improved U-Net model.

[0115] Feature extraction and segmentation:

[0116] Encoding stage: gradually downsample through the convolutional layer to extract the global features of the license plate characters.

[0117] Decoding stage: Restore the local details of the characters through upsampling and skip connections to generate a character segmentation mask map (MaskMap).

[0118] Character region extraction: Based on the segmentation mask image, the binary region of each character is extracted and the bounding box position of the character is calculated.

[0119] Optimize segmentation results;

[0120] Character alignment: Normalize the segmented character areas and adjust the character size to a standard size (32×32 pixels) to facilitate subsequent recognition.

[0121] Character sorting: Sort characters based on the horizontal position (x-coordinate) of their bounding boxes to ensure that the order of character recognition is consistent with the license plate format.

[0122] 2. Morphological restoration and contour tracking technology;

[0123] Morphological restoration algorithm;

[0124] Opening and closing operations:

[0125] Opening operation: Use a small-size rectangular structure element (3×3) to perform an opening operation on the character segmentation image to remove noise points and separate the stuck characters.

[0126] Closing operation: Perform a closing operation on the segmented characters to fill small holes inside the characters and repair broken strokes (such as the break in the middle of the "8").

[0127] Morphological dilation: For blurred character edges, an adaptive dilation algorithm is used to enhance character contrast and ensure clear character outlines.

[0128] Contour tracking technology;

[0129] Connected component analysis: Identify independent character regions in the segmented image through connected component labeling.

[0130] Contour screening and correction:

[0131] Valid characters are screened based on their aspect ratio (0.5-2.0) and area threshold (minimum area is 100 pixels).

[0132] Perform morphological repair (filling or segmentation) on characters that do not meet the standards to ensure character integrity.

[0133] 3. Lightweight CRNN end-to-end character recognition;

[0134] Lightweight design of CRNN model;

[0135] Network structure optimization:

[0136] Convolution layer compression: Depthwise Separable Convolution is used to replace traditional convolution to reduce the number of parameters.

[0137] RNN layer simplification: Use lightweight bidirectional LSTM (Bi-LSTM) to replace the fully connected layer to reduce computational complexity.

[0138] Parameter quantity control: The final model parameter quantity is controlled within 1M to meet the deployment requirements of embedded devices.

[0139] CTC loss function: The CTC algorithm is used to align character sequences, solve the problem of variable character length, and support end-to-end training.

[0140] Character recognition process;

[0141] Input preprocessing: The segmented character images are resized to a uniform size (32×32 pixels) and normalized.

[0142] Feature extraction and sequence generation:

[0143] CNN feature extraction: extract local features (edges, textures) of characters through convolutional layers.

[0144] Bi-LSTM sequence modeling: Performs time-series modeling on the feature graph to generate a probability distribution sequence of characters. Decoding and output: Uses GreedyDecoding or BeamSearch algorithms to decode the probability sequence and output the final license plate character result.

[0145] Its beneficial effects are as follows: 1. It can significantly improve image quality, ensuring clear license plate images with complete details even under extreme lighting conditions. 2. It can effectively eliminate interference from other vehicle components (headlights, logos) and complex backgrounds, and accurately lock the license plate area even if the license plate is tilted at a certain angle or partially obscured. 3. While ensuring recognition accuracy, it greatly improves recognition speed, achieving end-to-end fast and accurate character recognition. This technical solution can cope with various complex situations such as license plate aging and stain obscuration, and the accuracy of license plate recognition is improved. 4. Through algorithm optimization and model lightweighting, it meets the real-time requirements of highway toll stations for license plate recognition. Vehicles can complete license plate recognition and toll collection without having to stop for a long time, significantly improving the traffic efficiency of toll stations, reducing vehicle waiting time in queues, and effectively alleviating congestion problems at toll stations.

[0146] See also Figure 2 In an automated highway toll license plate recognition method, a vehicle image is captured by multiple image sensors, noise reduction, contrast enhancement, and white balance adjustment are performed on the vehicle image in sequence to generate an initial vehicle image, and different light source images in the initial vehicle image are synthesized using image fusion technology to obtain a synthesized vehicle image, including the following steps:

[0147] Step 201: Capture a vehicle image using multiple image sensors, calculate pixel weights based on spatial distance and grayscale difference using a bilateral filtering algorithm, adjust the filter strength based on the image noise level, and perform noise reduction on the vehicle image to obtain a noise-reduced vehicle image.

[0148] Step 202: Use CLAHE adaptive histogram equalization to divide the denoised vehicle image into 8x8 small grids, and independently enhance the contrast of each grid to obtain an enhanced vehicle image;

[0149] Step 203: Count the pixels with grayscale values ​​between 128 and 192 in the enhanced vehicle image, calculate the average RGB value, establish a dynamic correction library for different light sources, and use the correction library to perform correction to obtain a corrected vehicle image;

[0150] Step 204 : Using the Laplacian pyramid fusion technology, the five corrected vehicle images of the same vehicle at different times and under different illuminations are combined into one image to obtain a combined vehicle image.

[0151] The above is an introduction to an embodiment of the method for automatically recognizing license plates for toll collection on highways. Figure 3 In an automated highway toll license plate recognition system, the highway toll license plate recognition system includes the following modules:

[0152] An image acquisition and synthesis module is used to acquire vehicle images using multiple image sensors, perform noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and use image fusion technology to synthesize images of different light sources in the initial vehicle image to obtain a synthesized vehicle image;

[0153] The license plate region extraction module is used to roughly locate the license plate region in the synthetic vehicle image based on the improved YOLOv7 model combined with the Canny edge detection algorithm. The Transformer module is introduced to extract long-range feature information, and the OCR text detection algorithm is used to assist in determining whether it is a real license plate, thereby obtaining a license plate region image.

[0154] The license plate character recognition module is used to segment the license plate characters in the license plate area image using the U-Net semantic segmentation model, correct the adhesion and breakage of characters using the morphological repair algorithm and contour tracking technology, and perform end-to-end character recognition through the lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition results.

[0155] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatically recognizing license plates for highway toll collection, characterized in that: The highway toll license plate recognition method comprises the following steps: capturing vehicle images using multiple image sensors, performing noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and synthesizing different light source images in the initial vehicle image using image fusion technology to obtain a synthesized vehicle image; The license plate area in the synthetic vehicle image is roughly located based on the improved YOLOv7 model combined with the Canny edge detection algorithm. The Transformer module is introduced to extract long-range feature information. The OCR text detection algorithm is used to assist in determining whether it is a real license plate, thereby obtaining a license plate area image. The U-Net semantic segmentation model is used to segment the license plate characters in the license plate area image, and the morphological repair algorithm and contour tracking technology are used to correct the character adhesion and breakage. The lightweight CRNN convolutional recurrent neural network model is used to perform end-to-end character recognition to obtain the license plate recognition result.

2. The method for automatic highway toll collection license plate recognition according to claim 1, characterized in that: The method includes: collecting vehicle images by using multiple image sensors, performing noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image; and synthesizing different light source images in the initial vehicle image using image fusion technology to obtain a synthesized vehicle image. A vehicle image is captured by multiple image sensors, pixel weights are calculated based on the two dimensions of spatial distance and grayscale difference using a bilateral filtering algorithm, and the filtering strength is adjusted according to the image noise level to perform noise reduction processing on the vehicle image to obtain a noise-reduced vehicle image; Using CLAHE adaptive histogram equalization to divide the denoised vehicle image into 8x8 small grids, and independently enhancing the contrast of each grid to obtain an enhanced vehicle image; Counting pixels with grayscale values ​​between 128 and 192 in the enhanced vehicle image, calculating the average RGB value, establishing a dynamic correction library for different light sources, and performing correction using the correction library to obtain a corrected vehicle image; Using the Laplace pyramid fusion technology, five rectified vehicle images of the same vehicle at different times and under different illumination are combined into one image to obtain a composite vehicle image.

3. The method for automatic highway toll collection license plate recognition according to claim 1, characterized in that: The improved YOLOv7 model is combined with the Canny edge detection algorithm to roughly locate the license plate area in the synthetic vehicle image, and the Transformer module is introduced to extract long-range feature information, including: A Transformer module is embedded in the deep feature extraction stage of the backbone network of the original YOLOv7 model, and the multi-head self-attention mechanism in the Transformer module focuses on the correlation between different regions; The feature map output by YOLOv7 is preprocessed using the Canny edge detection algorithm, the image is smoothed using Gaussian filtering, and a double threshold algorithm is used to distinguish between definite edges and possible edges; Perform edge density analysis on each candidate detection frame generated by YOLOv7 and calculate the ratio of edge pixels within the frame. If the ratio is lower than the threshold, the area is determined to be the vehicle body and background and is excluded. Check the integrity of the edge contour. If the edge within the detection frame is broken, exclude it and obtain a roughly positioned license plate image.

4. The method for automatic highway toll collection license plate recognition according to claim 1, characterized in that: The method of using an OCR text detection algorithm to assist in determining whether the license plate is real and obtaining a license plate area image includes: After using the EAST algorithm to quickly locate the text area in the image, the license plate format rules are verified and the number of characters is checked; If the number of characters in the detected text area is significantly deviated, it will be directly excluded. The character arrangement characteristics will be analyzed. If the Chinese character is at the first position followed by letters and the overall aspect ratio meets the license plate standard, the detection area image will be output; The character format of the detection area image is verified. If the detected text area meets the three conditions at the same time, it is confirmed as the real license plate area and the license plate area image is obtained.

5. The method for automatic highway toll collection license plate recognition according to claim 1, characterized in that: The method of segmenting the license plate characters in the license plate area image using a U-Net semantic segmentation model includes: Separate each character in the license plate area image from the background using the U-Net semantic segmentation model; The upper layer of the semantic segmentation model is used to compress the image through multiple convolutions and pooling to extract deep semantic features; The lower layer of the semantic segmentation model is used for upsampling and deconvolution to gradually restore the feature map to its original size. At the same time, the detailed information of the upper layer is combined with the global information of the lower layer through jump connections to obtain the license plate characters.

6. The method for automatic highway toll collection license plate recognition according to claim 1, characterized in that: The method of using a morphological repair algorithm and contour tracking technology to correct character adhesion and breakage includes: The license plate characters are binarized, and then the morphological opening operation is used to remove noise and small adhesion areas, and the closing operation is used to connect slightly broken characters; the contour detection is performed on the repaired image to extract the contour of each character.

7. The method for automatic highway toll collection license plate recognition according to claim 1, characterized in that: The lightweight CRNN convolutional recurrent neural network model is used to perform end-to-end character recognition to obtain license plate recognition results, including: Each convolutional layer is followed by a ReLU activation function and a maximum pooling layer to extract the visual features of the characters. A two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence. A fully connected layer and a CTC loss function are used to convert the feature sequence into a character sequence.

8. An automated highway toll collection license plate recognition system, characterized in that: The highway toll license plate recognition system comprises the following steps: An image acquisition and synthesis module is configured to acquire vehicle images using multiple image sensors, perform noise reduction, contrast enhancement, and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and synthesize different light source images in the initial vehicle image using image fusion technology to obtain a synthesized vehicle image; A license plate region extraction module is used to roughly locate the license plate region in the synthetic vehicle image based on an improved YOLOv7 model combined with a Canny edge detection algorithm, introduce a Transformer module to extract long-range feature information, and use an OCR text detection algorithm to assist in determining whether it is a real license plate, thereby obtaining a license plate region image; The license plate character recognition module is used to segment the license plate characters in the license plate area image using the U-Net semantic segmentation model, correct the adhesion and breakage of characters using the morphological repair algorithm and contour tracking technology, and perform end-to-end character recognition through the lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

9. The automatic highway toll collection license plate recognition system according to claim 8, characterized in that: The license plate character recognition module includes the following units: The binarization unit is used to binarize the license plate characters, then use morphological opening operations to remove noise and small adhesion areas, and use closing operations to connect slightly broken characters; contour detection is performed on the repaired image to extract the contour of each character.

10. The automatic highway toll collection license plate recognition system according to claim 8, characterized in that: The license plate character recognition module includes the following units: Character conversion unit, which is used to extract the visual features of characters by following each convolutional layer with ReLU activation function and maximum pooling layer; A two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence, and the feature sequence is converted into a character sequence through a fully connected layer and CTC loss function.

Images