An automated method and system for license plate recognition in highway toll collection

Abstract

This invention discloses an automated license plate recognition method and system for highway toll collection. The method involves acquiring vehicle images using multiple image sensors, sequentially performing noise reduction, contrast enhancement, and white balance adjustment on these images to generate an initial vehicle image. Image fusion technology is then used to synthesize images from different light sources within the initial vehicle image. A coarse localization of the license plate region in the synthesized vehicle image is performed based on an improved YOLOv7 model combined with the Canny edge detection algorithm. The license plate characters in the image are segmented using a U-Net semantic segmentation model, and end-to-end character recognition is performed using a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result. This approach significantly improves recognition speed while maintaining accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to an automated method and system for recognizing license plates for highway toll collection. Background Technology

[0002] Traditional license plate recognition technology relies primarily on a single camera to capture images, which has numerous limitations in the complex environment of highways. Highway lighting conditions are highly variable, ranging from strong sunlight to low light in tunnels, and nighttime headlights. These factors can cause overexposure, underexposure, or glare in the captured license plate images, making it difficult to clearly display the characters and resulting in low recognition accuracy. Secondly, the high speeds of vehicles on highways mean that ordinary image acquisition equipment often has insufficient frame rates, easily causing image blurring and ghosting, leading to the loss of detail in the license plate characters and further reducing recognition efficiency. Summary of the Invention

[0003] The purpose of this invention is to solve the above-mentioned problems by designing an automated method and system for recognizing license plates for highway toll collection.

[0004] To achieve the above objectives, the technical solution of the present invention further includes the following steps in the above-mentioned automated highway toll vehicle license plate recognition method:

[0005] Vehicle images are acquired by multiple image sensors. The vehicle images are then subjected to noise reduction, contrast enhancement, and white balance adjustment in sequence to generate an initial vehicle image. Image fusion technology is then used to synthesize the different light source images in the initial vehicle image to obtain a composite vehicle image.

[0006] Based on the improved YOLOv7 model and the Canny edge detection algorithm, the license plate region in the synthesized vehicle image is coarsely located. The Transformer module is introduced to extract long-distance feature information, and the OCR text detection algorithm is used to assist in determining whether it is a real license plate, thus obtaining the license plate region image.

[0007] The license plate characters in the license plate region image are segmented using the U-Net semantic segmentation model. Morphological restoration algorithms and contour tracking techniques are used to correct character adhesion and breakage. End-to-end character recognition is performed using a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

[0008] Furthermore, in the aforementioned automated highway toll collection license plate recognition method, the step of acquiring vehicle images through multiple image sensors, sequentially performing noise reduction, contrast enhancement, and white balance adjustment on the vehicle images to generate an initial vehicle image, and then using image fusion technology to synthesize images from different light sources in the initial vehicle image to obtain a synthesized vehicle image, includes:

[0009] Vehicle images are acquired by multiple image sensors. Pixel weights are calculated based on two dimensions, spatial distance and grayscale difference, using a bilateral filtering algorithm. The filtering intensity is adjusted according to the image noise level to perform noise reduction processing on the vehicle images, resulting in a noise-reduced vehicle image.

[0010] The denoised vehicle image is divided into 8x8 grids using CLAHE adaptive histogram equalization, and the contrast of each grid is enhanced independently to obtain an enhanced vehicle image.

[0011] The pixels with grayscale values ​​in the enhanced vehicle image between 128 and 192 are counted, the average RGB value is calculated, a dynamic correction library is established for different light sources, and the correction library is used to perform correction to obtain the corrected vehicle image.

[0012] Using the Laplacian pyramid fusion technique, five corrected vehicle images of the same vehicle taken at different times and under different lighting conditions are combined into one image to obtain a synthetic vehicle image.

[0013] Furthermore, in the aforementioned automated highway toll collection license plate recognition method, the license plate region in the synthesized vehicle image is coarsely located based on the improved YOLOv7 model combined with the Canny edge detection algorithm, and a Transformer module is introduced to extract long-distance feature information, including:

[0014] A Transformer module is embedded in the deep feature extraction stage of the backbone network of the original YOLOv7 model, and the multi-head self-attention mechanism in the Transformer module is used to focus on the correlation between different regions.

[0015] The feature map output by YOLOv7 is preprocessed using the Canny edge detection algorithm, the image is smoothed by Gaussian filtering, and a double threshold algorithm is used to distinguish between confirmed edges and potential edges.

[0016] For each candidate detection box generated by YOLOv7, perform edge density analysis, calculate the percentage of edge pixels within the box, and if it is lower than the threshold, the region is determined to be the car body and the background and is excluded.

[0017] Check the integrity of the edge contour. If the edge inside the detection box is broken, it is excluded to obtain a coarsely located license plate image.

[0018] Furthermore, in the aforementioned automated highway toll collection license plate recognition method, the step of using an OCR text detection algorithm to assist in determining whether the license plate is genuine and obtaining a license plate area image includes:

[0019] After using the EAST algorithm to quickly locate the text region in the image, the license plate format rules are verified and the number of characters is checked.

[0020] If the number of characters in the detected text region deviates significantly, it is directly excluded. Analyze the character arrangement features. If the Chinese character is at the beginning and followed by a letter and the overall aspect ratio meets the license plate standard, then output the image of the detected region.

[0021] The character format of the detected area image is verified. If the detected text area meets all three conditions, it is confirmed as a real license plate area, and the license plate area image is obtained.

[0022] Furthermore, in the aforementioned automated highway toll collection license plate recognition method, the segmentation of license plate characters in the license plate region image using the U-Net semantic segmentation model includes:

[0023] The U-Net semantic segmentation model is used to separate each character in the license plate region image from the background;

[0024] By using multiple convolutions and pooling in the upper layer of the semantic segmentation model, the image is compressed and deep semantic features are extracted.

[0025] By upsampling and deconvolution in the lower layer of the semantic segmentation model, the feature map is gradually restored to its original size. At the same time, by using skip connections, the detailed information of the upper layer is combined with the global information of the lower layer to obtain the license plate characters.

[0026] Furthermore, in the aforementioned automated highway toll collection license plate recognition method, the step of using morphological restoration algorithms and contour tracking technology to correct character adhesion and breakage includes:

[0027] The license plate characters are binarized, and then morphological opening operations are used to remove noise and small adhesion areas. Closing operations are used to connect slightly broken characters. Contour detection is performed on the repaired image to extract the contour of each character.

[0028] Furthermore, in the aforementioned automated highway toll collection license plate recognition method, the step of performing end-to-end character recognition using a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result includes:

[0029] ReLU activation and max pooling layers are applied after each convolutional layer to extract visual features of the characters; a two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence; and the feature sequence is converted into a character sequence through a fully connected layer and CTC loss function.

[0030] Furthermore, in an automated highway toll collection license plate recognition system, the highway toll collection license plate recognition system includes the following modules:

[0031] The image acquisition and synthesis module is used to acquire vehicle images through multiple image sensors, perform noise reduction, contrast enhancement and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and use image fusion technology to synthesize images of different light sources in the initial vehicle image to obtain a synthesized vehicle image.

[0032] The license plate region extraction module is used to coarsely locate the license plate region in the synthesized vehicle image based on the improved YOLOv7 model combined with the Canny edge detection algorithm. It introduces the Transformer module to extract long-distance feature information and uses the OCR text detection algorithm to help determine whether it is a real license plate, thus obtaining the license plate region image.

[0033] The license plate character recognition module is used to segment license plate characters in the license plate region image using the U-Net semantic segmentation model, correct character adhesion and breakage using morphological restoration algorithms and contour tracking technology, and perform end-to-end character recognition using a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

[0034] Furthermore, in an automated highway toll collection license plate recognition system, the license plate character recognition module includes the following units:

[0035] The binarization unit is used to binarize the license plate characters. Then, morphological opening operations are used to remove noise and small adhesion areas, and closing operations are used to connect slightly broken characters. Contour detection is performed on the repaired image to extract the contour of each character.

[0036] Furthermore, in an automated highway toll collection license plate recognition system, the license plate character recognition module includes the following units:

[0037] The character conversion unit is used to extract the visual features of characters by following each convolutional layer with a ReLU activation function and a max pooling layer. A two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence, and the feature sequence is converted into a character sequence through a fully connected layer and a CTC loss function.

[0038] Its beneficial effects are as follows: 1. It significantly improves image quality, ensuring clear and detailed license plate images even under extreme lighting conditions. 2. It effectively eliminates interference from other vehicle components (lights, markings) and complex backgrounds, accurately locating the license plate area even if it is tilted or partially obscured. 3. While maintaining recognition accuracy, it greatly improves recognition speed, achieving fast and accurate end-to-end character recognition. This technical solution can handle various complex situations such as aging license plates and obscured plates, improving license plate recognition accuracy. 4. Through algorithm optimization and model lightweighting, it meets the real-time requirements of highway toll stations for license plate recognition. Vehicles can complete license plate recognition and toll collection without prolonged stops, significantly improving the efficiency of toll station traffic, reducing vehicle queuing time, and effectively alleviating toll station congestion. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0040] Figure 1 This is a schematic diagram of the first embodiment of an automated highway toll collection vehicle license plate recognition method according to the present invention;

[0041] Figure 2 This is a schematic diagram of a second embodiment of an automated highway toll collection vehicle license plate recognition method according to the present invention;

[0042] Figure 3 This is a schematic diagram of the first embodiment of an automated highway toll collection vehicle license plate recognition system according to the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0044] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0045] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1As shown, an automated method for recognizing license plates for highway toll collection includes the following steps:

[0046] Step 101: Acquire vehicle images through multiple image sensors, perform noise reduction, contrast enhancement and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and use image fusion technology to synthesize the images of different light sources in the initial vehicle image to obtain a synthesized vehicle image.

[0047] Specifically, in this embodiment, vehicle images are acquired by multiple image sensors, and pixel weights are calculated based on two dimensions, spatial distance and grayscale difference, using a bilateral filtering algorithm. The filtering intensity is adjusted according to the image noise level to perform noise reduction processing on the vehicle images, resulting in a noise-reduced vehicle image.

[0048] The CLAHE adaptive histogram equalization is used to divide the denoised vehicle image into 8x8 grids, and the contrast of each grid is enhanced independently to obtain an enhanced vehicle image.

[0049] Statistically enhance the pixels in the vehicle image with grayscale values ​​in the range of 128-192, calculate the average RGB value, establish a dynamic correction library for different light sources, and use the correction library to perform correction to obtain the corrected vehicle image.

[0050] Using the Laplacian pyramid fusion technique, five corrected vehicle images of the same vehicle taken at different times and under different lighting conditions are combined into one image to obtain a synthetic vehicle image.

[0051] Specifically:

[0052] (a) Image sensor deployment and parameter configuration;

[0053] Multiple high dynamic range (HDR) image sensors should be strategically deployed at key locations on highways, such as tollbooth entrances and exits, and above each lane. The sensors should have a resolution of at least 2 megapixels to ensure clear capture of vehicle details, including characters on license plates. The frame rate should be set to at least 30 frames per second to guarantee that at high speeds (up to 120 km / h), at least 10 images are captured for each vehicle passing through the sensor's coverage area, preventing missed images. The sensors' dynamic range should be greater than 120 dB to adapt to complex lighting environments on highways, ranging from direct sunlight to low light in tunnels.

[0054] (II) Image preprocessing workflow;

[0055] Noise Reduction: A bilateral filtering algorithm is used to denoise the acquired vehicle images. Bilateral filtering not only effectively removes Gaussian noise and salt-and-pepper noise but also preserves the image's edge information, which is crucial for subsequent license plate feature extraction. The spatial Gaussian kernel standard deviation σ_s is set to 3-5, and the gray-level Gaussian kernel standard deviation σ_r is set to 15-25, with parameters automatically adjusted according to the image's noise level. Specifically, for each pixel in the image, the weighted average of its surrounding pixels is calculated, with the weights determined by spatial distance and gray-level differences. Contrast Enhancement: Adaptive Histogram Equalization (CLAHE) is used to enhance image contrast. CLAHE divides the image into multiple small regions (8x8 blocks), performs histogram equalization on each region separately, and limits the contrast enhancement amplitude to avoid noise amplification. The cropping limit is set to 2.0 to balance contrast enhancement and noise suppression. This processing makes the contrast between the characters in the license plate area and the background more pronounced, facilitating subsequent detection and recognition. White Balance Adjustment: A white balance algorithm based on the gray world assumption, which assumes that the average RGB values ​​of the numerous gray areas in an image are equal. First, the average R, G, and B values ​​of the image are calculated. Then, the image is adjusted according to the proportions of these three values ​​to make the color reproduction closer to the real scene. For images under different light sources (sunlight during the day, streetlights at night, and tunnel lighting), their average RGB values ​​are statistically analyzed, a light source classification model is established, and appropriate white balance adjustment parameters are automatically selected. Image Fusion: A Laplacian pyramid fusion algorithm is used to synthesize preprocessed images from different light sources. First, a Laplacian pyramid is constructed for each initial vehicle image. The number of pyramid layers is determined based on the image resolution, ranging from 5 to 7 layers. Then, at each layer, corresponding frequency bands of different images are weighted and fused. The weights are automatically calculated based on indicators such as image sharpness and contrast.

[0056] Step 102: Based on the improved YOLOv7 model and the Canny edge detection algorithm, the license plate area in the synthesized vehicle image is coarsely located. The Transformer module is introduced to extract long-distance feature information. The OCR text detection algorithm is used to assist in determining whether it is a real license plate, and the license plate area image is obtained.

[0057] Specifically, in this embodiment, a Transformer module is embedded in the deep feature extraction stage of the backbone network of the original YOLOv7 model, and the multi-head self-attention mechanism in the Transformer module focuses on the correlation between different regions.

[0058] The feature map output by YOLOv7 is preprocessed using the Canny edge detection algorithm. Gaussian filtering is applied to smooth the image, and a double threshold algorithm is used to distinguish between confirmed and potential edges. Edge density analysis is performed on each candidate detection box generated by YOLOv7, calculating the percentage of edge pixels within the box. If the percentage is below the threshold, the region is considered a vehicle body or background and is excluded. The integrity of the edge contour is checked; if the edge within the detection box is broken, it is excluded, resulting in a coarsely located license plate image. The text region in the image is quickly located using the EAST algorithm. The license plate format rules are verified, and the number of characters is checked. If the number of characters in the detected text region deviates significantly, it is directly excluded. The character arrangement features are analyzed; if a Chinese character is at the beginning followed by a letter, and the overall aspect ratio conforms to the license plate standard, the detection region image is output. The character format of the detection region image is verified; if the detected text region meets all three conditions, it is confirmed as a real license plate region, resulting in the license plate region image.

[0059] Specifically:

[0060] (a) Improved YOLOv7 model design;

[0061] Based on the original YOLOv7 model, its backbone network is improved by introducing a Transformer module. Specifically, a multi-head self-attention mechanism is added to the deep feature extraction stage of the backbone network. Each Transformer module consists of a multi-head self-attention layer and a feedforward neural network, employing residual connections and layer normalization techniques. The multi-head self-attention mechanism can capture long-distance dependencies between different regions in the image, playing a crucial role in extracting the correlation features between the license plate region and other parts of the vehicle (body, lights). The improved model's input image size is adjusted to 640x640 to balance detection speed and accuracy. During training, mixed precision training and data augmentation techniques (random flipping, scaling, rotation, color jittering, etc.) are used to improve the model's generalization ability.

[0062] (ii) Canny edge detection and region fusion;

[0063] The feature map output by the improved YOLOv7 model is input into the Canny edge detection module. The specific parameters for Canny edge detection are set as follows: standard deviation σ = 1.4 for the Gaussian filter, low threshold of 30, and high threshold of 90. Edge contours of the image are obtained through a double-threshold algorithm and edge tracking. The edge detection results are then fused with the target detection boxes of the YOLOv7 model. For each detection box, its internal edge density and contour integrity are calculated to filter out regions that may contain license plates. This allows for further refinement of the YOLOv7 model's detection results using edge information, improving the accuracy of license plate region localization.

[0064] (III) OCR text detection-assisted judgment;

[0065] An OCR text detection algorithm based on EAST (Efficient and Accurate Scene Text) is used to process the coarsely located license plate regions. The EAST model can quickly detect text regions in an image and output the coordinates and rotation angle of the text box. For each candidate license plate region, the text detection result is extracted, and it is determined whether it conforms to the character format of a license plate (Chinese license plates typically have 7 characters, including Chinese characters, letters, and numbers). Simultaneously, features such as the aspect ratio and character spacing of the text region are calculated and matched with prior knowledge of real license plates. Only candidate regions that meet the license plate format and features are confirmed as real license plate regions, thus eliminating interference from non-license plate regions and improving the accuracy of license plate detection.

[0066] Specifically, it also includes:

[0067] 1. Improved design of YOLOv7;

[0068] Based on the traditional YOLOv7, the following optimizations have been made for license plate detection tasks:

[0069] Multi-scale Feature Pyramid (FPN+PAN): Introducing the Feature Pyramid Network (FPN) and Path Aggregation Network (PAN) to enhance the detection capability of license plates of different sizes (small car license plates and large truck license plates).

[0070] Lightweight Transformer Module: A lightweight Transformer module is embedded in the YOLOv7 backbone network to replace some convolutional layers. The Transformer captures long-range dependency features of the license plate region (the continuity of license plate edges and the spatial relationship between characters) through a self-attention mechanism, improving detection robustness in complex backgrounds.

[0071] Dynamic Nonmaximum Suppression (DynamicNMS): This optimizes the NMS algorithm by dynamically adjusting the suppression threshold of the detection box to avoid duplicate detection caused by license plate tilt or occlusion.

[0072] Coarse positioning process for license plate area;

[0073] Input image processing: Input the synthetic vehicle image (fused from multiple light sources) generated in step 1 into the improved YOLOv7 model.

[0074] Feature extraction and object detection:

[0075] Image features are extracted using the improved YOLOv7 Backbone, and the Transformer module is combined to enhance long-range feature representation.

[0076] The YOLOv7 detection head outputs candidate bounding boxes for the license plate region, including the license plate position (x, y, w, h) and confidence score.

[0077] Coarse localization results output: Candidate boxes with a confidence level higher than the threshold (0.6) are selected as the initial localized license plate area.

[0078] 2. Canny edge detection-assisted localization;

[0079] Applications of Canny edge detection;

[0080] Image preprocessing: The synthesized vehicle image is converted to grayscale to reduce computation and highlight the license plate edge information.

[0081] Edge detection: The Canny algorithm is applied to detect significant edges in the image and generate a binary edge map.

[0082] Edge feature extraction:

[0083] Enhance edge connectivity and remove noise interference through morphological operations (dilation, erosion).

[0084] Based on the coarse localization results of YOLOv7, edge features (rectangular outline of the license plate) related to the license plate area are extracted from the edge binary map.

[0085] Co-location with YOLOv7;

[0086] Bounding box optimization: Match the candidate boxes of YOLOv7 with the rectangular outlines of Canny edge detection, and correct the position of the bounding boxes output by YOLOv7 (adjust the offset coordinates).

[0087] License plate region verification: Determine whether the candidate bounding box conforms to the geometric characteristics of a license plate (aspect ratio close to 3:1, good edge continuity) based on edge features. If it does not conform, the candidate bounding box is removed.

[0088] 3. Auxiliary verification of OCR text detection;

[0089] Integration of OCR text detection;

[0090] Text region extraction: For the license plate region image jointly localized by YOLOv7 and Canny, OCR text detection algorithms (CTPN or DBNet) are used to further detect the text region.

[0091] Character arrangement verification:

[0092] The OCR algorithm outputs the confidence level of the text region and the character arrangement direction (horizontal or slanted).

[0093] Determine whether the text conforms to the distribution pattern of license plate characters (uniform character spacing and character types conforming to national standards).

[0094] Real license plate screening and logical reasoning: Combining OCR detection results with YOLOv7 confidence levels, a comprehensive judgment is made as to whether a candidate box represents a real license plate. If the OCR detects a text region and the character arrangement conforms to license plate rules, the candidate box is retained; if the OCR does not detect valid text or the character arrangement is abnormal (character breaks, overlaps), the candidate box is discarded. Output license plate region image: The final determined license plate region image serves as input for subsequent steps.

[0095] Step 103: Use the semantic segmentation model of U-Net to segment the license plate characters in the license plate region image, and use morphological restoration algorithm and contour tracking technology to correct character adhesion and breakage. Use a lightweight CRNN convolutional recurrent neural network model to perform end-to-end character recognition to obtain the license plate recognition result.

[0096] Specifically, in this embodiment, the U-Net semantic segmentation model is used to separate each character in the license plate area image from the background;

[0097] By using multiple convolutions and pooling in the upper layer of the semantic segmentation model, the image is compressed and deep semantic features are extracted.

[0098] By upsampling and deconvolution in the lower layer of the semantic segmentation model, the feature map is gradually restored to its original size. At the same time, by using skip connections, the detailed information of the upper layer is combined with the global information of the lower layer to obtain the license plate characters.

[0099] The license plate characters are binarized, and then morphological opening operations are used to remove noise and small adhesion areas. Closing operations are used to connect slightly broken characters. Contour detection is performed on the repaired image to extract the contour of each character.

[0100] ReLU activation and max pooling layers are applied after each convolutional layer to extract visual features of the characters; a two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence; and the feature sequence is converted into a character sequence through a fully connected layer and CTC loss function.

[0101] Specifically:

[0102] (a) U-Net semantic segmentation model;

[0103] A U-Net model suitable for license plate character segmentation is constructed, taking a license plate region image as input, resized to 256x256. The network structure includes a contraction path and an expansion path. The contraction path consists of convolutional layers (3x3 convolutional kernels, ReLU activation function, batch normalization) and max pooling layers, progressively extracting deep features. The expansion path consists of upsampling layers, convolutional layers, and skip connections, combining shallow detail information with deep semantic information. Finally, a 1x1 convolutional layer outputs the segmentation result, with each pixel categorized as license plate character, license plate background, or border. During training, a combination of cross-entropy loss and Dice coefficient loss functions is used to improve segmentation accuracy and the ability to detect small targets (single characters).

[0104] (ii) Correction of character adhesion and breakage;

[0105] Morphological restoration algorithm: For the segmented character image, binarization is first performed. Then, morphological opening operations (erosion followed by dilation) are used to remove noise and small adhered regions. Closing operations (dilation followed by erosion) are used to connect slightly broken characters. The structuring element for the erosion and dilation operations is a 3x3 rectangular kernel, and the number of operations is adjusted according to the size and adhesion of the characters.

[0106] Contour tracking technology: Contour detection is performed on the repaired image to extract the contour of each character. For connected characters, the shape and positional relationship of the contours are used to determine whether they belong to the same character, and they are separated using a contour segmentation algorithm. For broken characters, the continuity and symmetry of the contours are utilized to repair the broken parts using interpolation methods. During the processing, a priori shape model of the characters (common shapes of Chinese characters, letters, and numbers) is established to assist in judging the rationality of the contours.

[0107] Specifically, it also includes:

[0108] 1. Character segmentation in the U-Net semantic segmentation model;

[0109] Improved design of the U-Net model;

[0110] Residual Connections and Attention Mechanisms: Residual blocks and channel attention mechanisms (SEBlocks) are introduced into the traditional U-Net encoder-decoder architecture to enhance the model's ability to capture the edge features of license plate characters.

[0111] Multi-scale feature fusion: Dilated convolution is used in the encoder stage to extract multi-scale features and preserve the details of the characters; in the decoder stage, high-resolution features are fused with low-resolution features through skip connections to improve segmentation accuracy.

[0112] Dynamic boundary optimization: A weighted combination of DiceLoss and cross-entropy loss is introduced into the loss function to dynamically adjust the model's attention to character boundary segmentation and reduce segmentation errors caused by character adhesion or breakage.

[0113] Character splitting process;

[0114] Input image processing: Input the license plate area image output from step 2 into the improved U-Net model.

[0115] Feature extraction and segmentation:

[0116] Encoding stage: Global features of license plate characters are extracted by progressive downsampling through convolutional layers.

[0117] Decoding stage: Local details of characters are recovered through upsampling and skip connections to generate a character segmentation mask map.

[0118] Character region extraction: Based on the segmentation mask image, extract the binarized region of each character and calculate the bounding box position of the character.

[0119] Segmentation results optimization;

[0120] Character alignment: The segmented character regions are normalized, and the size of the characters is adjusted to the standard size (32×32 pixels) to facilitate subsequent recognition.

[0121] Character sorting: The characters are sorted based on their horizontal position (x-coordinate) of the character bounding box to ensure that the character recognition order is consistent with the license plate format.

[0122] 2. Morphological restoration and contour tracking technology;

[0123] Morphological restoration algorithm;

[0124] Opening and closing operations:

[0125] Opening operation: The character segmentation image is opened using small rectangular structuring elements (3×3) to remove noise points and separate connected characters.

[0126] Closing operation: Performs a closing operation on the segmented characters to fill in the small holes inside the characters and repair broken strokes (such as the break in the middle of the "8").

[0127] Morphological dilation: For cases where character edges are blurred, an adaptive dilation algorithm is used to enhance character contrast and ensure clear character outlines.

[0128] Contour tracking technology;

[0129] Connected component analysis: Identifying independent character regions in a segmented image using connected component labels.

[0130] Contour filtering and correction:

[0131] Valid characters are filtered based on their aspect ratio (0.5-2.0) and area threshold (minimum area is 100 pixels).

[0132] Morphological repair (filling or segmentation) is performed on non-compliant characters to ensure character integrity.

[0133] 3. Lightweight CRNN end-to-end character recognition;

[0134] Lightweight design of CRNN models;

[0135] Network structure optimization:

[0136] Convolutional layer compression: Depthwise separable convolution is used instead of traditional convolution to reduce the number of parameters.

[0137] RNN layer simplification: Lightweight bidirectional LSTM (Bi-LSTM) is used to replace the fully connected layer to reduce computational complexity.

[0138] Parameter control: The final model parameter count is controlled within 1M to meet the deployment requirements of embedded devices.

[0139] CTC loss function: The CTC algorithm aligns character sequences, solving the problem of variable character length and supporting end-to-end training.

[0140] Character recognition process;

[0141] Input preprocessing: The segmented character images are adjusted to a uniform size (32×32 pixels) and normalized.

[0142] Feature extraction and sequence generation:

[0143] CNN feature extraction: Extracts local features (edges, textures) of characters through convolutional layers.

[0144] Bi-LSTM sequence modeling: Temporal modeling is performed on the feature map to generate a probability distribution sequence of characters. Decoding and output: The probability sequence is decoded using GreedyDecoding or BeamSearch algorithms to output the final license plate character result.

[0145] Its beneficial effects are as follows: 1. It significantly improves image quality, ensuring clear and detailed license plate images even under extreme lighting conditions. 2. It effectively eliminates interference from other vehicle components (lights, markings) and complex backgrounds, accurately locating the license plate area even if it is tilted or partially obscured. 3. While maintaining recognition accuracy, it greatly improves recognition speed, achieving fast and accurate end-to-end character recognition. This technical solution can handle various complex situations such as aging license plates and obscured plates, improving license plate recognition accuracy. 4. Through algorithm optimization and model lightweighting, it meets the real-time requirements of highway toll stations for license plate recognition. Vehicles can complete license plate recognition and toll collection without prolonged stops, significantly improving the efficiency of toll station traffic, reducing vehicle queuing time, and effectively alleviating toll station congestion.

[0146] Please see Figure 2 In an automated highway toll collection license plate recognition method, vehicle images are acquired through multiple image sensors. These images are then sequentially denoised, contrast-enhanced, and white-balanced to generate an initial vehicle image. Image fusion technology is then used to synthesize images from different light sources within the initial vehicle image to obtain a composite vehicle image. The process includes the following steps:

[0147] Step 201: Acquire vehicle images through multiple image sensors, calculate pixel weights based on the two dimensions of spatial distance and grayscale difference using a bilateral filtering algorithm, and adjust the filtering intensity according to the image noise level to perform noise reduction processing on the vehicle images to obtain noise-reduced vehicle images.

[0148] Step 202: Use CLAHE adaptive histogram equalization to divide the denoised vehicle image into 8x8 grids, and enhance the contrast of each grid independently to obtain an enhanced vehicle image;

[0149] Step 203: Count the pixels with gray values ​​in the range of 128-192 in the enhanced vehicle image, calculate the average RGB value, establish a dynamic correction library for different light sources, and use the correction library to perform correction to obtain the corrected vehicle image.

[0150] Step 204: Using the Laplacian pyramid fusion technique, five corrected vehicle images of the same vehicle at different times and under different lighting conditions are combined into one image to obtain a composite vehicle image.

[0151] The above describes an embodiment of the automated highway toll collection license plate recognition method of the present invention. Please refer to [link / reference]. Figure 3 In an automated highway toll collection vehicle license plate recognition system, the system includes the following modules:

[0152] The image acquisition and synthesis module is used to acquire vehicle images through multiple image sensors, perform noise reduction, contrast enhancement and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and use image fusion technology to synthesize images of different light sources in the initial vehicle image to obtain a synthesized vehicle image.

[0153] The license plate region extraction module is used to coarsely locate the license plate region in the synthesized vehicle image based on the improved YOLOv7 model and the Canny edge detection algorithm. The Transformer module is introduced to extract long-distance feature information, and the OCR text detection algorithm is used to help determine whether it is a real license plate, so as to obtain the license plate region image.

[0154] The license plate character recognition module is used to segment license plate characters in the license plate region image using the U-Net semantic segmentation model. Morphological restoration algorithms and contour tracking technology are used to correct character adhesion and breakage. End-to-end character recognition is performed through a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

[0155] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An automated method for license plate recognition in highway toll collection, characterized in that, The method for recognizing license plates of vehicles used for highway toll collection includes the following steps: Vehicle images are acquired by multiple image sensors. The vehicle images are then subjected to noise reduction, contrast enhancement, and white balance adjustment in sequence to generate an initial vehicle image. Image fusion technology is then used to synthesize the different light source images in the initial vehicle image to obtain a composite vehicle image. A Transformer module is embedded in the deep feature extraction stage of the backbone network of the original YOLOv7 model. The multi-head self-attention mechanism in the Transformer module focuses on the correlation between different regions. The feature map output by YOLOv7 is preprocessed using the Canny edge detection algorithm. The image is smoothed by Gaussian filtering, and a double threshold algorithm is used to distinguish between confirmed edges and possible edges. Edge density analysis is performed on each candidate detection box generated by YOLOv7 to calculate the percentage of edge pixels within the box. If the percentage is lower than the threshold, the region is determined to be a vehicle body or background and is excluded. The integrity of the edge contour is checked. If the edge within the detection box is broken, it is excluded to obtain a coarsely localized license plate image. An OCR text detection algorithm is used to assist in determining whether it is a real license plate to obtain the license plate region image. The license plate characters in the license plate region image are segmented using the U-Net semantic segmentation model. Morphological restoration algorithms and contour tracking techniques are used to correct character adhesion and breakage. End-to-end character recognition is performed using a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

2. The automated license plate recognition method for highway toll collection as described in claim 1, characterized in that, The process involves acquiring vehicle images through multiple image sensors, sequentially performing noise reduction, contrast enhancement, and white balance adjustment on the vehicle images to generate an initial vehicle image, and then using image fusion technology to synthesize images from different light sources in the initial vehicle image to obtain a synthesized vehicle image, including: Vehicle images are acquired by multiple image sensors. Pixel weights are calculated based on two dimensions, spatial distance and grayscale difference, using a bilateral filtering algorithm. The filtering intensity is adjusted according to the image noise level to perform noise reduction processing on the vehicle images, resulting in a noise-reduced vehicle image. The denoised vehicle image is divided into 8x8 grids using CLAHE adaptive histogram equalization, and the contrast of each grid is enhanced independently to obtain an enhanced vehicle image. The pixels with grayscale values ​​in the enhanced vehicle image between 128 and 192 are counted, the average RGB value is calculated, a dynamic correction library is established for different light sources, and the correction library is used to perform correction to obtain the corrected vehicle image. Using the Laplacian pyramid fusion technique, five corrected vehicle images of the same vehicle taken at different times and under different lighting conditions are combined into one image to obtain a synthetic vehicle image.

3. The automated license plate recognition method for highway toll collection as described in claim 1, characterized in that, The step of using an OCR text detection algorithm to assist in determining whether a license plate is genuine, and obtaining a license plate area image, includes: After using the EAST algorithm to quickly locate the text region in the image, the license plate format rules are verified and the number of characters is checked. If the number of characters in the detected text region deviates significantly, it is directly excluded. Analyze the character arrangement features. If the Chinese character is at the beginning and followed by a letter and the overall aspect ratio meets the license plate standard, then output the image of the detected region. The character format of the detected area image is verified. If the detected text area meets all three conditions, it is confirmed as a real license plate area, and the license plate area image is obtained.

4. The automated license plate recognition method for highway toll collection as described in claim 1, characterized in that, The segmentation of license plate characters in the license plate region image using the U-Net semantic segmentation model includes: The U-Net semantic segmentation model is used to separate each character in the license plate region image from the background; By using multiple convolutions and pooling in the upper layer of the semantic segmentation model, the image is compressed and deep semantic features are extracted. By upsampling and deconvolution in the lower layer of the semantic segmentation model, the feature map is gradually restored to its original size. At the same time, by using skip connections, the detailed information of the upper layer is combined with the global information of the lower layer to obtain the license plate characters.

5. The automated license plate recognition method for highway toll collection as described in claim 1, characterized in that, The method of using morphological restoration algorithms and contour tracking technology to correct character adhesion and breakage includes: The license plate characters are binarized, and then morphological opening operations are used to remove noise and small adhesion areas. Closing operations are used to connect slightly broken characters. Contour detection is performed on the repaired image to extract the contour of each character.

6. The automated license plate recognition method for highway toll collection as described in claim 1, characterized in that, The end-to-end character recognition using a lightweight CRNN convolutional recurrent neural network model to obtain license plate recognition results includes: ReLU activation and max pooling layers are applied after each convolutional layer to extract visual features of the characters. Two layers of Bi-LSTM bidirectional long short-term memory network are used to capture the contextual information of the character sequence. The feature sequence is converted into a character sequence through a fully connected layer and CTC loss function.

7. An automated license plate recognition system for highway toll collection, characterized in that, The highway toll collection vehicle license plate recognition system includes the following steps: The image acquisition and synthesis module is used to acquire vehicle images through multiple image sensors, perform noise reduction, contrast enhancement and white balance adjustment on the vehicle images in sequence to generate an initial vehicle image, and use image fusion technology to synthesize images of different light sources in the initial vehicle image to obtain a synthesized vehicle image. The license plate region extraction module embeds a Transformer module into the deep feature extraction stage of the backbone network of the original YOLOv7 model. The Transformer module uses a multi-head self-attention mechanism to focus on the correlation between different regions. The module preprocesses the feature map output by YOLOv7 using the Canny edge detection algorithm, smooths the image with Gaussian filtering, and uses a double threshold algorithm to distinguish between confirmed and potential edges. Edge density analysis is performed on each candidate detection box generated by YOLOv7, calculating the percentage of edge pixels within the box. If the percentage is below the threshold, the region is determined to be a vehicle body or background and is excluded. The module checks the integrity of the edge contours; if the edges within the detection box are broken, they are excluded, resulting in a coarsely located license plate image. An OCR text detection algorithm is then used to assist in determining whether it is a genuine license plate, thus obtaining the license plate region image. The license plate character recognition module is used to segment license plate characters in the license plate region image using the U-Net semantic segmentation model, correct character adhesion and breakage using morphological restoration algorithms and contour tracking technology, and perform end-to-end character recognition using a lightweight CRNN convolutional recurrent neural network model to obtain the license plate recognition result.

8. The automated highway toll collection license plate recognition system as described in claim 7, characterized in that, The license plate character recognition module includes the following units: The binarization unit is used to binarize the license plate characters. Then, morphological opening operations are used to remove noise and small adhesion areas, and closing operations are used to connect slightly broken characters. Contour detection is performed on the repaired image to extract the contour of each character.

9. The automated highway toll collection license plate recognition system as described in claim 7, characterized in that, The license plate character recognition module includes the following units: The character transformation unit is used to extract the visual features of characters by following each convolutional layer with a ReLU activation function and a max pooling layer; A two-layer Bi-LSTM bidirectional long short-term memory network is used to capture the contextual information of the character sequence, and the feature sequence is converted into a character sequence through a fully connected layer and the CTC loss function.

Images