Automatic recognition processing system and method for multi-angle license plate image, electronic device and storage medium

Abstract

The present application relates to the technical field of machine vision, in particular to a multi-angle license plate image automatic recognition processing system and method, electronic equipment and storage medium, the method comprising: acquiring multi-angle license plate images, using an adaptive threshold segmentation algorithm to perform binary processing on the images and extract license plate contours; using perspective transformation to correct license plate contour images at different angles; extracting local binary pattern features and HSV color histogram features of the corrected license plate contour images, using a support vector machine (SVM) classifier to cut the license plate character regions, correcting the cutting results according to the license plate character arrangement rules to obtain accurate character cutting images, building a multi-layer convolutional neural network, inputting the cut character images into the network for recognition and classification, and mapping and outputting license plate number information. The present application comprehensively utilizes image processing, machine learning and deep learning technologies, and comprehensively improves the accuracy and robustness of license plate recognition, and has a broad application prospect.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, specifically to an automatic recognition and processing system, method, electronic device, and storage medium for multi-angle license plate images. Background Technology

[0002] License plate recognition technology, as a crucial component of intelligent transportation systems, plays a key role in vehicle management and parking management. Traditional license plate recognition methods primarily target frontal, clear, and unobstructed license plate images, which has limitations in practical applications. However, in real-world traffic scenarios, license plate images often exhibit characteristics such as multi-angle exposure, blurriness, and low resolution, posing significant challenges to license plate recognition.

[0003] Existing license plate recognition technologies mainly include template matching, feature extraction, and deep learning-based methods. Template matching methods locate and identify license plates by matching them against predefined templates, but are sensitive to license plate angles and deformations. Feature extraction methods manually design features such as color and texture features and then use a classifier for recognition, but feature selection and extraction require significant experience and trial-and-error. Deep learning-based methods automatically learn feature representations of license plate images using convolutional neural networks and perform end-to-end recognition, achieving good results. However, existing deep learning methods are mainly designed for frontal license plate images; for multi-angle, tilted, and deformed license plate images, recognition accuracy still needs improvement.

[0004] Furthermore, the image preprocessing, license plate localization, and character segmentation stages of license plate recognition also face several challenges. The quality of the license plate image directly affects recognition accuracy; accurately locating the license plate region against complex backgrounds is a crucial issue. Tilted or deformed license plates make character segmentation difficult, and designing robust character segmentation algorithms is another challenge. Traditional binarization and character segmentation algorithms are not sufficiently adaptable to multi-angle and blurred license plate images.

[0005] In view of this, the present invention proposes an automatic recognition and processing system, method, electronic device and storage medium for multi-angle license plate images. Summary of the Invention

[0006] To achieve the above objectives, the present invention provides an automatic recognition and processing system, method, electronic device, and storage medium for multi-angle license plate images, the specific technical solutions of which are as follows:

[0007] In a first aspect, the present invention provides an automatic recognition and processing method for multi-angle license plate images, comprising:

[0008] Multi-angle license plate images are acquired, and an adaptive threshold segmentation algorithm is used to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate area. The license plate image is then binarized to obtain the outline image of the license plate.

[0009] Perspective transformation is used to correct license plate outline images from different angles. By calculating the homography matrix between the license plate quadrilateral outline and the rectangular template, the deformation of the license plate outline image is eliminated.

[0010] Local binary pattern features and HSV color histogram features were extracted from the corrected license plate outline image, and a support vector machine (SVM) classifier was used to segment the character regions of the license plate.

[0011] Based on the segmented image of the license plate character region, the character segmentation results in the license plate image are corrected according to the license plate character arrangement rules;

[0012] A multi-layer convolutional neural network is constructed, which takes license plate character images as input, identifies the character image category, and maps it to license plate information as output, thereby realizing automatic license plate recognition.

[0013] Preferably, multi-angle license plate images are acquired, and any acquired original license plate image from any angle is denoted as... ,in and These represent the row and column coordinates of the original license plate image, respectively.

[0014] Adaptive thresholding segmentation is performed on the license plate image using the Sauvola adaptive thresholding algorithm. Perform binarization and calculate the adaptive threshold for each pixel. Based on adaptive threshold Binarize the image:

[0015] ;

[0016] in, This represents the binarized license plate image, where pixels with a value of 0 correspond to the license plate area, and pixels with a value of 1 correspond to the background area.

[0017] Binarized license plate image Morphological processing was performed, including:

[0018] use Rectangular structural elements Perform an opening operation to eliminate areas smaller than 10 ... Noise points of pixels; using The rectangular structuring element performs a closing operation on the opening result, filling the area within the license plate region with a size smaller than [value missing]. The holes in the pixels;

[0019] use The cross-shaped structuring element performs a dilation operation on the closing result, connecting potentially disconnected pixels within the license plate area; using The cross-shaped structural element is used to erode the expansion result to remove noise in the license plate area;

[0020] Let the morphologically processed binary image of the license plate be denoted as . Size and same;

[0021] Using the Suzuki85 boundary tracking algorithm to extract from the morphologically processed binary license plate image Extract all closed contours, and filter out the corresponding license plate image contours based on the geometric features of the contours, including area, aspect ratio, and rectangularity. ;set up ,in, This represents the number of pixels on the outline. For the outline of the first The coordinates of each pixel.

[0022] Preferably, based on the extracted license plate outline, the Douglas-Puk algorithm is used for polygon fitting to obtain an approximate polygon of the license plate outline. ,in To fit the number of vertices of the polygon, That is, the fitted shape is a quadrilateral; let the coordinates of the vertices of the fitted license plate quadrilateral be:

[0023] ;

[0024] in The upper left vertex of the license plate quadrilateral. The upper right vertex of the license plate quadrilateral. The lower right vertex of the license plate quadrilateral. It is the lower left vertex of the license plate quadrilateral;

[0025] Set a rectangular template with the same size as the license plate:

[0026] ;

[0027] in, , , , , and These are the width and height of the license plate, respectively.

[0028] Based on the correspondence of the four points, the homography matrix is ​​calculated using the least squares method. , so that:

[0029] ;

[0030] in, It is a homography matrix; through the homography matrix For the binarized license plate image Perform perspective transformation to obtain the corrected license plate image. : ;in, These are the pixel coordinates of the original license plate image. To correct the pixel coordinates on the license plate image;

[0031] Corrected license plate image The corrected license plate image is obtained by cropping according to the rectangular template.

[0032] Preferably, for the corrected license plate image, a local binary mode operator is used to extract texture features. The local binary mode operator is defined as follows:

[0033] ;

[0034] in, This indicates the number of neighboring pixels selected. It is the index of the neighboring pixels. Represents the neighborhood radius. To correct the pixel coordinates on the license plate image, The grayscale value of the center pixel. The grayscale value of the neighboring pixels. For symbolic functions:

[0035] For each pixel ,calculate Values ​​are used to obtain the LBP feature image. ;

[0036] Will Divided into For each of the non-overlapping local regions, calculate the LBP feature histogram. :

[0037] ;

[0038] in, , , , , , All are parameters. The Kronecker function;

[0039] Concatenate the LBP feature histograms of all local regions into a vector , as a local binary pattern feature of the license plate image;

[0040] Extract HSV color histogram features and then apply them to the corrected license plate image. Color images Convert from RGB color space to HSV color space to obtain an image with three channels: H, S, and V. , , Calculate the color histogram for each channel separately. , , :

[0041] ;

[0042] in, Indicates color channels, , For the color bin, The number of colors; concatenate the color histograms of the three channels into a vector. , as the HSV color histogram feature of the license plate image;

[0043] The extracted local binary pattern features and HSV color histogram features Concatenate into feature vectors , as a feature representation of the license plate image.

[0044] Preferably, the feature vectors of a batch of license plate images are labeled with the location and category information of the character regions; the feature vectors of the license plate images and the corresponding character region information and location information are used as a sample dataset to train a support vector machine (SVM) classifier.

[0045] For the license plate image to be cut, extract the feature vector. The trained SVM classifier is used to classify each pixel position of the license plate image. Classification is performed to obtain the location and category information of the character regions; the feature vectors are then processed. The input is fed into an SVM classifier to compute feature vectors. Distance to the hyperplane :

[0046] ;

[0047] in, Represents pixels The eigenvector at that location; and Let these represent the weights and bias parameters of the Support Vector Machine (SVM) classifier, respectively; if Then Mark it as a character area; otherwise mark it as a background area.

[0048] The character regions obtained from the SVM classifier are post-processed to obtain the final character segmentation result. The steps include:

[0049] Adjacent character regions are merged to obtain complete connected character regions. For each connected character region, the bounding rectangle is calculated as the initial result of character segmentation. Based on prior knowledge of the size and aspect ratio of the license plate characters, the segmentation result is adjusted to filter out pseudo-character regions with abnormal aspect ratios. Based on the position of the character region in the license plate image and the character spacing, the segmentation result is adjusted to obtain the final character segmentation image.

[0050] Preferably, a multi-layer convolutional neural network is designed, which includes multiple convolutional layers, pooling layers and fully connected layers, for license plate character recognition;

[0051] The multi-layer convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer is used to receive character segmented images. The convolutional layer includes a first convolutional layer and a second convolutional layer. The pooling layer includes a first pooling layer and a second pooling layer. The fully connected layer includes a first fully connected layer and a second fully connected layer.

[0052] The first convolutional layer includes: using The size is The convolution kernel performs a convolution operation on the input image to obtain... Each feature map;

[0053] The first pooling layer includes: performing max pooling on the output of the first convolutional layer to reduce the size of the feature map; the second convolutional layer includes: using... The size is The convolution kernel performs a convolution operation on the output of the first pooling layer to obtain... The first convolutional layer performs max pooling on the output of the second convolutional layer to further reduce the size of the feature map; the second pooling layer stretches the output of the second pooling layer into a one-dimensional vector and connects it to the first fully connected layer. One neuron; the second fully connected layer connects the output of the first fully connected layer to the neurons via a fully connected manner. neurons, of which The number of character categories is given; the output layer performs Softmax normalization on the output of the second fully connected layer to obtain the probability of each character category;

[0054] The segmented character images and their corresponding character category labels are used as training data. The convolutional neural network is trained using backpropagation and stochastic gradient descent. Let the training samples be... ,in For the first A character image, The corresponding character category label; the loss function of the convolutional neural network is the cross-entropy loss function.

[0055] Preferably, for the license plate image to be identified, a character segmentation algorithm is used to segment it into individual character images; the segmented character images are then input into a trained convolutional neural network to obtain a character category probability vector corresponding to each character image; the character category with the highest probability is taken as the recognition result. ;in, For the first The recognition results of the character image, For character index labels;

[0056] The character recognition results are arranged according to the position of the characters in the license plate to obtain the complete license plate number: ;in, To identify the obtained license plate number, This represents the number of characters in the license plate.

[0057] An automatic recognition and processing system for multi-angle license plate images, which is used to implement the automatic recognition and processing method for multi-angle license plate images, includes an image acquisition module, an image correction module, a character segmentation module, a character correction module, and a character recognition module;

[0058] The image acquisition module acquires license plate images from multiple angles, uses an adaptive threshold segmentation algorithm to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate area, performs binarization processing on the license plate image, and obtains the outline image of the license plate.

[0059] The image correction module uses perspective transformation to correct license plate outline images from different angles. By calculating the homography matrix between the license plate quadrilateral outline and the rectangular template, the deformation of the license plate outline image is eliminated.

[0060] The character segmentation module extracts local binary pattern features and HSV color histogram features from the corrected license plate outline image, and uses a support vector machine (SVM) classifier to segment the character regions of the license plate.

[0061] The character correction module corrects the character segmentation results in the license plate image based on the segmented image of the license plate character region and according to the license plate character arrangement rules.

[0062] The character recognition module constructs a multi-layer convolutional neural network, takes license plate character images as input, identifies the character image category, and maps it to license plate information for output, thereby realizing automatic license plate recognition.

[0063] An electronic device includes a processor and a memory, wherein the memory stores a computer program for the processor to call; the processor executes the automatic recognition processing method for multi-angle license plate images by calling the computer program stored in the memory.

[0064] A computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the automatic recognition processing method for multi-angle license plate images.

[0065] The beneficial effects of this invention: The adaptive threshold segmentation provided by this invention overcomes the influence of lighting and license plate aging, accurately extracts the license plate outline, and lays the foundation for subsequent processing.

[0066] This invention corrects license plate perspective distortion through perspective transformation, restoring tilted and deformed license plates to a frontal view, and providing standardized input for character segmentation and recognition.

[0067] This invention extracts local binary patterns and HSV color histogram features, and uses SVM to segment characters, accurately locating the position and range of characters.

[0068] This invention performs post-processing on the cutting results according to character arrangement rules, solving problems of adhesion and incompleteness, and improving the integrity and accuracy of cutting.

[0069] This invention constructs a multi-layer convolutional neural network to recognize character images end-to-end and directly output license plate numbers, achieving full-process automation. Attached Figure Description

[0070] Figure 1 A flowchart of the automatic recognition and processing method for multi-angle license plate images provided by the present invention;

[0071] Figure 2 This is a structural diagram of the automatic recognition and processing system for multi-angle license plate images provided by the present invention. Detailed Implementation

[0072] To better understand the invention, various aspects of the invention will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of the invention and are not intended to limit the scope of the invention in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0073] In the accompanying drawings, the size, dimensions, and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustrative purposes only and are not strictly to scale. As used herein, the terms “approximately,” “about,” and similar terms are used to indicate approximation rather than degree, and are intended to illustrate inherent deviations in measured or calculated values ​​that will be recognized by one of ordinary skill in the art. Furthermore, the order in which the steps are described in this invention does not necessarily indicate the order in which these steps occur in actual operation, unless otherwise expressly defined or deduced from the context.

[0074] It should also be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of the invention, the word "may" is used to mean "one or more embodiments of the invention." And the term "exemplary" is intended to refer to examples or illustrations.

[0075] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formalized sense.

[0076] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0077] Example 1

[0078] Reference Figure 1 The first embodiment of the present invention provides an automatic recognition and processing method for multi-angle license plate images.

[0079] S1: Acquire multi-angle license plate images, adopt an adaptive threshold segmentation algorithm, automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate area, perform binarization processing on the license plate image, and obtain the outline image of the license plate.

[0080] The target vehicle was photographed from different angles (front, side, oblique, etc.) and distances (e.g., 1 meter, 3 meters, 5 meters, etc.) to obtain multi-angle license plate images; the original license plate image obtained from any angle is denoted as . ,in and These represent the row and column coordinates of the original license plate image, respectively.

[0081] Preprocess the original license plate image. Perform the following preprocessing operations: a) Use a 5×5 Gaussian filter to... a) Smooth the image using the standard deviation of the standard deviation to remove high-frequency noise; b) Filter the smoothed image using a 3×3 median filter to further eliminate salt-and-pepper noise; c) Enhance the contrast of the filtered image using a histogram equalization algorithm to improve the contrast between the license plate area and the background; d) Use Gamma correction is applied to adjust the brightness of the enhanced image, making the brightness distribution in the license plate area more uniform; let the preprocessed license plate image be denoted as . The size is the same as the original license plate image.

[0082] Adaptive threshold segmentation is performed on the preprocessed license plate image. The Sauvola adaptive threshold segmentation algorithm is used to segment the preprocessed license plate image. Perform binarization;

[0083] Set the local window size to 15×15, which is in pixels. Calculate the mean within a 15×15 neighborhood centered on the center. and standard deviation ;Pick Image grayscale range ; Calculate the adaptive threshold for each pixel according to the Sauvola algorithm formula. : The image is then binarized based on an adaptive threshold.

[0084] ;

[0085] in, This represents the binarized license plate image, where pixels with a value of 0 correspond to the license plate area, and pixels with a value of 1 correspond to the background area.

[0086] Binarized license plate image Morphological processing was performed, including:

[0087] use Rectangular structural elements Perform an opening operation to eliminate areas smaller than 10 ... Pixel noise points; using The rectangular structuring element performs a closing operation on the opening result, filling the area within the license plate region with a size smaller than [value missing]. Pixel holes;

[0088] use The cross-shaped structuring element performs a dilation operation on the closing result, connecting potentially disconnected pixels within the license plate area; using The cross-shaped structural element is used to erode the expansion result, removing small noise blocks in the license plate area;

[0089] Let the morphologically processed binary image of the license plate be denoted as . Its size is same;

[0090] Using the Suzuki85 boundary tracking algorithm to extract from the morphologically processed binary license plate image Extract all closed contours, and based on the geometric features of the contours—area, aspect ratio, and rectangularity—select the contours corresponding to the license plate area. .

[0091] set up ,in, This represents the number of pixels on the outline. For the outline of the first The coordinates of each pixel.

[0092] Step S1 preprocesses and binarizes multi-angle license plate images using an adaptive threshold segmentation algorithm, overcoming the challenge of license plate region segmentation under different shooting conditions, improving the robustness and accuracy of license plate localization, and laying a good foundation for subsequent license plate recognition.

[0093] S2: Use perspective transformation to correct license plate outline images from different angles. By calculating the homography matrix between the license plate quadrilateral outline and the rectangular template, the deformation of the license plate outline image is eliminated.

[0094] For the extracted license plate outline The Douglas-Peucker algorithm was used for polygon fitting to obtain an approximate polygon of the license plate outline. ,in To fit the number of vertices of the polygon, That is, it fits to a quadrilateral.

[0095] Let the coordinates of the vertices of the fitted license plate quadrilateral be:

[0096] ;

[0097] in The upper left vertex of the license plate quadrilateral. The upper right vertex of the license plate quadrilateral. The lower right vertex of the license plate quadrilateral. It is the lower left vertex of the license plate quadrilateral.

[0098] Set a rectangular template with the same size as the license plate:

[0099] ;

[0100] in, , , , , and These are the width and height of the license plate, respectively.

[0101] Based on the correspondence of the four points, the homography matrix is ​​calculated using the least squares method. , so that:

[0102] ;

[0103] in, A 3×3 homography matrix can be represented as:

[0104] ;

[0105] Substituting the correspondence of the four points into the above equation yields eight equations, forming a system of linear equations. Solving this system of equations using Singular Value Decomposition (SVD) yields the homography matrix. The estimated value.

[0106] Using the estimated homography matrix For the binarized license plate image Perform perspective transformation to obtain the corrected license plate image. : ;in, These are the pixel coordinates of the original license plate image. To correct the pixel coordinates on the license plate image.

[0107] Because perspective transformation may introduce black invalid areas, the corrected license plate image needs to be processed. Cropping is performed on the corrected license plate image. The corrected license plate image is obtained by cropping according to the rectangular template.

[0108] The cropping area can be adjusted according to the width and height ratio of the license plate. Adaptive determination, for example, the size of the cropped license plate image can be set to [value missing]. ,in , Preserve the central area of ​​the license plate image. This indicates rounding down to the nearest integer.

[0109] Local binary pattern feature extraction for the corrected binary license plate image Texture features are extracted using a local binary mode operator.

[0110] Define the LBP operator as follows:

[0111] ;

[0112] in, This indicates the number of neighboring pixels selected. It is the index of the neighboring pixels. Represents the neighborhood radius. To correct the pixel coordinates on the license plate image, The grayscale value of the center pixel. The grayscale value of the neighboring pixels. This is a symbolic function, and its function expression is as follows:

[0113] ;

[0114] in, The independent variable is the symbolic function.

[0115] For each pixel ,calculate Values ​​are used to obtain the LBP feature image. .

[0116] Will Divided into For each of the non-overlapping local regions, calculate the LBP feature histogram. :

[0117] ;

[0118] in, , , , , , All are parameters. For the Kronecker function:

[0119] ;

[0120] Concatenate the LBP feature histograms of all local regions into a single vector. This serves as a local binary pattern feature of the license plate image.

[0121] Extract HSV color histogram features and then apply them to the corrected license plate image. Color images Convert from RGB color space to HSV color space to obtain an image with three channels: H, S, and V. , , Calculate the color histogram for each channel separately. , , :

[0122] ;

[0123] in, Indicates color channels, , For the color bin, The number of colors; concatenate the color histograms of the three channels into a single vector. , as the HSV color histogram feature of the license plate image.

[0124] The extracted local binary pattern features and HSV color histogram features Concatenate them into a feature vector , as a feature representation of the license plate image.

[0125] Step S2 uses perspective transformation to correct the license plate image, eliminating the distortion caused by changes in viewing angle, restoring the license plate to a standard viewing angle, and extracting local binary mode features and HSV color histogram features to comprehensively depict the license plate texture and color information, providing rich discrimination information for license plate character segmentation.

[0126] S3: Extract local binary pattern features and HSV color histogram features from the corrected license plate outline image, and use a support vector machine (SVM) classifier to segment the character regions of the license plate.

[0127] Label the feature vectors of a batch of license plate images with the location and category information of the character regions; use the feature vectors of these license plate images and the corresponding character region and location information as a sample dataset to train a support vector machine (SVM) classifier.

[0128] The goal of the SVM classifier is to find an optimal hyperplane. This ensures that positive and negative samples can be correctly classified by the hyperplane, and maximizes the distance between the hyperplane and the nearest sample point (i.e., the classification margin).

[0129] For the license plate image to be cut, extract its feature vector. The trained SVM classifier is used to classify each pixel position of the license plate image. Classification is performed to obtain the location and category information of the character regions; the feature vectors are then processed. The input is fed into an SVM classifier to compute feature vectors. Distance to the hyperplane :

[0130] ;

[0131] in, Indicates position The eigenvector at position; where, and Let these represent the weights and bias parameters of the Support Vector Machine (SVM) classifier, respectively; if Then Mark it as a character area; otherwise, mark it as a background area.

[0132] Step S3 uses an SVM classifier to segment the license plate image into character regions. The classifier is trained using manually labeled samples, effectively identifying the location and category information of the character regions. The segmentation results are then corrected by incorporating license plate character arrangement rules, further improving the accuracy and completeness of the character segmentation.

[0133] S4: Based on the segmented image of the license plate character region, the character segmentation results in the license plate image are corrected according to the license plate character arrangement rules.

[0134] The character regions obtained from the SVM classifier are post-processed to obtain the final character segmentation result. The steps include:

[0135] Adjacent character regions are merged to obtain complete connected character regions. For each connected character region, its bounding rectangle is calculated as the initial result of character segmentation. Based on prior knowledge of the size and aspect ratio of the license plate characters, the segmentation result is adjusted to filter out pseudo-character regions that are too small or have abnormal aspect ratios. Based on the position of the character region in the license plate image and the character spacing, the segmentation result is adjusted to obtain the final character segmentation image.

[0136] Step S4 employs a segmentation-based method for license plate character recognition, overcoming issues such as character adhesion and incompleteness. Through post-processing operations such as character region merging, size and spacing adjustments, a standardized and complete character segmentation image is obtained, providing high-quality input for subsequent character recognition and improving the overall performance of license plate recognition.

[0137] S5: Construct a multi-layer convolutional neural network, take license plate character images as input, identify the character image category, and map it into license plate information output to achieve automatic license plate recognition.

[0138] Design a multi-layer convolutional neural network, which includes multiple convolutional layers, pooling layers, and fully connected layers, for license plate character recognition.

[0139] The multi-layer convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer is used to receive character segmented images. The convolutional layer includes a first convolutional layer and a second convolutional layer. The pooling layer includes a first pooling layer and a second pooling layer. The fully connected layer includes a first fully connected layer and a second fully connected layer.

[0140] The first convolutional layer includes: using The size is The convolution kernel performs a convolution operation on the input image to obtain... Each feature map; the convolution operation can be represented as:

[0141] ;

[0142] in, For the first Each feature map For the input image, and The first The weights and biases of each convolutional kernel. and It is the location inside the convolution kernel. Represents the difference between the input image and the output pixel. The corresponding pixel coordinates.

[0143] The first pooling layer includes: performing max pooling on the output of the first convolutional layer to reduce the size of the feature map; the pooling operation can be expressed as:

[0144] ;

[0145] in, This is the feature map after pooling. This is the size of the pooling window.

[0146] The second convolutional layer includes: using The size is The convolution kernel performs a convolution operation on the output of the first pooling layer to obtain... Each feature map; the convolution operation is similar to that of the first convolutional layer.

[0147] The second pooling layer performs max pooling on the output of the second convolutional layer to further reduce the size of the feature map; the pooling operation is similar to that of the first pooling layer.

[0148] The first fully connected layer stretches the output of the second pooling layer into a one-dimensional vector and connects it to the first fully connected layer. There are 10 neurons; a fully connected operation can be represented as:

[0149] ;

[0150] in, This is the output of the first fully connected layer. This is the output of the second pooling layer. and These are the width and height of the feature map. and These are the weights and biases of the first fully connected layer, respectively. These are the labels for neurons in the fully connected layer.

[0151] The second fully connected layer: connects the output of the first fully connected layer to the second fully connected layer via a fully connected connection. neurons, of which The number of character categories; the full-connect operation is similar to the first full-connect layer.

[0152] The output layer includes Softmax normalization of the output of the second fully connected layer to obtain the probability of each character category:

[0153] ;

[0154] Where e is the natural constant, For the first The probability of each character category. This is the output of the second fully connected layer.

[0155] The segmented character images and their corresponding character category labels are used as training data. The convolutional neural network is trained using backpropagation and stochastic gradient descent. Let the training samples be... ,in For the first A character image, The corresponding character category label; the network's loss function is the cross-entropy loss function.

[0156] By minimizing the loss function, the gradients of the network parameters (convolutional kernel weights, biases, and fully connected layer weights) are calculated using the backpropagation algorithm, and the parameters are updated using stochastic gradient descent.

[0157] For the license plate image to be identified, a character segmentation algorithm is used to segment it into individual character images. The segmented character images are then input into a trained convolutional neural network to obtain the character category probability vector for each character image. The character category with the highest probability is taken as the recognition result. ;in, For the first The recognition results of the character image, For character index labels.

[0158] Arrange the character recognition results according to the order of the characters in the license plate to obtain the complete license plate number: ;in, To identify the obtained license plate number, This represents the number of characters in the license plate.

[0159] Step S5 constructs a multi-layer convolutional neural network for license plate character recognition, utilizing operations such as convolution and pooling to extract hierarchical features from the character image. Through end-to-end training and Softmax classification, license plate numbers can be recognized efficiently and accurately, achieving full automation of the license plate recognition process and demonstrating significant practical value.

[0160] Example 2

[0161] Reference Figure 2 The second embodiment of the present invention provides an automatic recognition and processing system for multi-angle license plate images.

[0162] The system includes an image acquisition module, an image correction module, a character segmentation module, a character correction module, and a character recognition module.

[0163] The image acquisition module acquires license plate images from multiple angles, uses an adaptive threshold segmentation algorithm to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate area, performs binarization processing on the license plate image, and obtains the outline image of the license plate.

[0164] The image correction module uses perspective transformation to correct license plate outline images from different angles. By calculating the homography matrix between the license plate quadrilateral outline and the rectangular template, it eliminates the deformation of the license plate outline image.

[0165] The character segmentation module extracts local binary pattern features and HSV color histogram features from the corrected license plate outline image, and uses a support vector machine (SVM) classifier to segment the character regions of the license plate.

[0166] The character correction module corrects the character segmentation results in the license plate image based on the segmented image of the license plate character region and according to the license plate character arrangement rules.

[0167] The character recognition module constructs a multi-layer convolutional neural network, takes license plate character images as input, identifies the character image category, and maps it to license plate information for output, thereby realizing automatic license plate recognition.

[0168] Example 3

[0169] The present invention also provides an electronic device. This electronic device may include one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, can perform the automatic recognition processing method for multi-angle license plate images as described above.

[0170] The methods or systems according to embodiments of the present invention can also be implemented using the architecture of the electronic devices of the present invention.

[0171] Electronic devices may include buses, one or more CPUs, read-only memory (ROM), random access memory (RAM), communication ports connected to a network, input / output components, hard disks, etc.

[0172] Storage devices in electronic devices, such as ROM or hard disks, can store the automatic recognition and processing method for multi-angle license plate images provided by this invention.

[0173] An automatic license plate recognition processing method for multi-angle license plate images includes: acquiring multi-angle license plate images; employing an adaptive threshold segmentation algorithm to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate region; performing binarization processing on the license plate image to obtain the license plate contour image; correcting the license plate contour image from different angles using perspective transformation; eliminating deformation of the license plate contour image by calculating the homography matrix between the license plate quadrilateral contour and the rectangular template; extracting local binary pattern features and HSV color histogram features from the corrected license plate contour image; using a support vector machine (SVM) classifier to segment the license plate character region; correcting the character segmentation results in the license plate image based on the segmented image of the license plate character region and according to the license plate character arrangement rules; and constructing a multi-layer convolutional neural network, using the license plate character image as input, identifying the character image category, and mapping it to license plate information output to achieve automatic license plate recognition.

[0174] Furthermore, the electronic device may also include a user interface. Of course, the architecture of this invention is merely exemplary; in implementing different devices, one or more components of the electronic device disclosed in this invention may be omitted according to actual needs.

[0175] Example 4

[0176] The present invention also discloses a computer-readable storage medium.

[0177] Computer-readable instructions are stored on a computer-readable storage medium.

[0178] When computer-readable instructions are executed by a processor, an automatic recognition processing method for multi-angle license plate images according to an embodiment of the present invention, as described with reference to the above figures, can be performed.

[0179] Storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. Furthermore, according to embodiments of the present invention, the processes described in the above-mentioned flowcharts can be implemented as computer software programs.

[0180] For example, this invention provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be executed by a processor to perform instructions corresponding to the method steps provided by this invention. For example: acquiring multi-angle license plate images; employing an adaptive threshold segmentation algorithm to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate region; binarizing the license plate image to obtain the license plate outline image; correcting the license plate outline images from different angles using perspective transformation; eliminating deformation of the license plate outline image by calculating the homography matrix between the license plate quadrilateral outline and the rectangular template; extracting local binary pattern features and HSV color histogram features from the corrected license plate outline image; segmenting the license plate character region using a support vector machine (SVM) classifier; correcting the character segmentation results in the license plate image based on the segmented image of the license plate character region and according to the license plate character arrangement rules; constructing a multi-layer convolutional neural network, using the license plate character image as input, identifying the character image category, and mapping it to license plate information output to achieve automatic license plate recognition.

[0181] When the computer program is executed by the central processing unit (CPU), it performs the functions defined in the method of the present invention. The method, apparatus, and device of the present invention may be implemented in many ways. For example, the method, apparatus, and device of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware.

[0182] The above-described order of steps for the method is for illustrative purposes only, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated.

[0183] Furthermore, in some embodiments, the invention may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the method according to the invention. Therefore, the invention also covers recording media storing programs for performing the method according to the invention.

[0184] In addition, the parts of the technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0185] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An automatic recognition and processing method for multi-angle license plate images, characterized in that, include: Multi-angle license plate images are acquired, and an adaptive threshold segmentation algorithm is used to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate area. The license plate image is then binarized to obtain the outline image of the license plate. The Sauvola adaptive threshold segmentation algorithm is used to segment license plate images. Binarization is performed, and an adaptive threshold for each pixel is calculated according to the Sauvola algorithm formula. : ； in, and They are pixels Mean and standard deviation within the central neighborhood; The grayscale range of the image; It refers to adjusting parameters; Based on adaptive threshold Binarize the image: ； in, This represents the binarized license plate image, where pixels with a value of 0 correspond to the license plate area, and pixels with a value of 1 correspond to the background area. Perspective transformation is used to correct the license plate outline images from different angles. Based on the extracted license plate outline, the Douglas-Puk algorithm is used for polygon fitting to obtain an approximate polygon of the license plate outline. ,in To fit the number of vertices of the polygon, That is, the fitted shape is a quadrilateral; let the coordinates of the vertices of the fitted license plate quadrilateral be: ； in The upper left vertex of the license plate quadrilateral. The upper right vertex of the license plate quadrilateral. The lower right vertex of the license plate quadrilateral. It is the lower left vertex of the license plate quadrilateral; The deformation of the license plate outline image is eliminated by calculating the homography matrix between the quadrilateral outline of the license plate and the rectangular template. Local binary pattern features and HSV color histogram features were extracted from the corrected license plate outline image, and a support vector machine (SVM) classifier was used to segment the character regions of the license plate. The feature vectors of license plate images and the corresponding character region and location information are used as sample datasets to train a support vector machine (SVM) classifier. For the license plate image to be cut, extract the feature vector. The trained SVM classifier is used to classify each pixel position of the license plate image. Classification is performed to obtain the location and category information of the character regions; the feature vectors are then processed. The input is fed into an SVM classifier to compute feature vectors. Distance to the hyperplane : ； in, Represents pixels The eigenvector at that location; and Let these represent the weights and bias parameters of the Support Vector Machine (SVM) classifier, respectively; if Then Mark it as a character area; otherwise mark it as a background area. Based on the segmented image of the license plate character region, the character segmentation results in the license plate image are corrected according to the arrangement rules of the license plate characters; adjacent character regions are merged to obtain complete character connected regions; for each character connected region, the bounding rectangle is calculated as the preliminary result of character segmentation; based on prior knowledge of the size and aspect ratio of the license plate characters, the segmentation results are adjusted to filter out pseudo-character regions with abnormal aspect ratios; based on the position of the character region in the license plate image and the character spacing, the segmentation results are adjusted to obtain the final character segmented image; A multi-layer convolutional neural network is constructed, which takes license plate character images as input, identifies the character image category, and maps it to license plate information as output, thereby realizing automatic license plate recognition.

2. The automatic recognition and processing method for multi-angle license plate images according to claim 1, characterized in that, Acquire license plate images from multiple angles, and denote any obtained original license plate image from any angle as... ,in and These represent the row and column coordinates of the original license plate image, respectively. Binarized license plate image Morphological processing was performed, including: use Rectangular structural elements Perform an opening operation to eliminate areas smaller than 10 ... Pixel noise points; using The rectangular structuring element performs a closing operation on the opening result, filling the area within the license plate region with a size smaller than [value missing]. Pixel holes; use The cross-shaped structuring element performs a dilation operation on the closing result, connecting potentially disconnected pixels within the license plate area; using The cross-shaped structural element is used to erode the expansion result to remove noise in the license plate area; Let the morphologically processed binary image of the license plate be denoted as . Size and same; Using the Suzuki85 boundary tracking algorithm to extract from the morphologically processed binary license plate image Extract all closed contours, and filter out the corresponding license plate image contours based on the geometric features of the contours, including area, aspect ratio, and rectangularity. ;set up ,in, This represents the number of pixels on the outline. For the outline of the first The coordinates of each pixel.

3. The automatic recognition and processing method for multi-angle license plate images according to claim 2, characterized in that: Set a rectangular template with the same size as the license plate: ； in, , , , , and These are the width and height of the license plate, respectively. Based on the correspondence of the four points, the homography matrix is ​​calculated using the least squares method. , so that: ； in, It is a homography matrix; through the homography matrix For the binarized license plate image Perform perspective transformation to obtain the corrected license plate image. : ;in, These are the pixel coordinates of the original license plate image. To correct the pixel coordinates on the license plate image; Corrected license plate image The corrected license plate image is obtained by cropping according to the rectangular template.

4. The automatic recognition and processing method for multi-angle license plate images according to claim 3, characterized in that, For the corrected license plate image, the local binary mode operator is used to extract texture features. The local binary mode operator is defined as follows: ； in, This indicates the number of neighboring pixels selected. It is the index of the neighboring pixels. Represents the neighborhood radius. To correct the pixel coordinates on the license plate image, The grayscale value of the center pixel. The grayscale value of the neighboring pixels. For symbolic functions: For each pixel ,calculate Values ​​are used to obtain the LBP feature image. ; Will Divided into For each of the non-overlapping local regions, calculate the LBP feature histogram. : ； in, , , , , , All are parameters. The Kronecker function; Concatenate the LBP feature histograms of all local regions into a vector , as a local binary pattern feature of the license plate image; Extract HSV color histogram features and then apply them to the corrected license plate image. Color images Convert from RGB color space to HSV color space to obtain an image with three channels: H, S, and V. , , Calculate the color histogram for each channel separately. , , : ； in, Indicates color channels, , For the color bin, The number of colors; concatenate the color histograms of the three channels into a vector. , as the HSV color histogram feature of the license plate image; The extracted local binary pattern features and HSV color histogram features Concatenate into feature vectors , as a feature representation of the license plate image.

5. The automatic recognition and processing method for multi-angle license plate images according to claim 4, characterized in that, The location and category information of the character regions in the feature vectors of a batch of license plate images are labeled; the character regions obtained by the SVM classifier are post-processed to obtain the final character segmentation results.

6. The automatic recognition and processing method for multi-angle license plate images according to claim 5, characterized in that, Design a multi-layer convolutional neural network, which includes multiple convolutional layers, pooling layers, and fully connected layers, for license plate character recognition; The multi-layer convolutional neural network includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer is used to receive character segmented images. The convolutional layer includes a first convolutional layer and a second convolutional layer. The pooling layer includes a first pooling layer and a second pooling layer. The fully connected layer includes a first fully connected layer and a second fully connected layer. The first convolutional layer includes: using The size is The convolution kernel performs a convolution operation on the input image to obtain... Each feature map; The first pooling layer includes: performing max pooling on the output of the first convolutional layer to reduce the size of the feature map; the second convolutional layer includes: using... The size is The convolution kernel performs a convolution operation on the output of the first pooling layer to obtain... The first convolutional layer performs max pooling on the output of the second convolutional layer to further reduce the size of the feature map; the second pooling layer stretches the output of the second pooling layer into a one-dimensional vector and connects it to the first fully connected layer. One neuron; the second fully connected layer connects the output of the first fully connected layer to the neurons via a fully connected manner. neurons, of which The number of character categories is given; the output layer performs Softmax normalization on the output of the second fully connected layer to obtain the probability of each character category; The segmented character images and their corresponding character category labels are used as training data. The convolutional neural network is trained using backpropagation and stochastic gradient descent. Let the training samples be... ,in For the first A character image, The corresponding character category label; the network's loss function is the cross-entropy loss function.

7. The automatic recognition and processing method for multi-angle license plate images according to claim 6, characterized in that, For the license plate image to be identified, a character segmentation algorithm is used to segment it into individual character images. The segmented character images are then input into a trained convolutional neural network to obtain the character category probability vector for each character image. The character category with the highest probability is taken as the recognition result. ;in, For the first The recognition results of the character image, For character index labels; The character recognition results are arranged according to the position of the characters in the license plate to obtain the complete license plate number: ;in, To identify the obtained license plate number, This represents the number of characters in the license plate.

8. An automatic recognition and processing system for multi-angle license plate images, used to implement the automatic recognition and processing method for multi-angle license plate images as described in any one of claims 1 to 7, characterized in that, It includes an image acquisition module, an image correction module, a character segmentation module, a character correction module, and a character recognition module; The image acquisition module acquires license plate images from multiple angles, uses an adaptive threshold segmentation algorithm to automatically determine the binarization threshold based on the local pixel distribution characteristics of the license plate area, performs binarization processing on the license plate image, and obtains the outline image of the license plate. The image correction module uses perspective transformation to correct license plate outline images from different angles. It eliminates the deformation of the license plate outline image by calculating the homography matrix between the license plate quadrilateral outline and the rectangular template. The character segmentation module extracts local binary pattern features and HSV color histogram features from the corrected license plate outline image, and uses a support vector machine (SVM) classifier to segment the character regions of the license plate. The character correction module corrects the character segmentation results in the license plate image based on the segmented image of the license plate character region and according to the license plate character arrangement rules. The character recognition module constructs a multi-layer convolutional neural network, takes license plate character images as input, identifies the character image category, and maps it to license plate information for output, thereby realizing automatic license plate recognition.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a computer program for the processor to call; the processor executes the automatic recognition processing method for multi-angle license plate images according to any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, The device stores instructions that, when executed on a computer, cause the computer to perform the automatic recognition processing method for multi-angle license plate images as described in any one of claims 1 to 7.

Images