Multi-angle license plate image automatic identification processing system and method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of machine vision, in particular to an automatic recognition processing system and method for a multi-angle license plate image, electronic equipment and a storage medium, and the method comprises the steps: obtaining the multi-angle license plate image, carrying out the binarization processing of the image through employing a self-adaptive threshold segmentation algorithm, and extracting the contour of a license plate; correcting the license plate contour images at different angles by using perspective transformation; extracting local binary pattern features and HSV color histogram features of the corrected license plate contour image, performing character region cutting on the license plate by using a support vector machine (SVM) classifier, correcting a cutting result according to a license plate character arrangement rule to obtain an accurate character cutting image, constructing a multi-layer convolutional neural network, and obtaining a final license plate contour image. And inputting the cut character image into a network for identification and classification, and mapping and outputting license plate number information. According to the method, image processing, machine learning and deep learning technologies are comprehensively utilized, the accuracy and robustness of license plate recognition are comprehensively improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a system, method, electronic equipment and storage medium for automatically identifying and processing multi-angle license plate images. Background Art

[0002] As an important part of intelligent transportation systems, license plate recognition technology plays a key role in vehicle management and parking management. Traditional license plate recognition methods mainly focus on frontal, clear, and unobstructed license plate images, which have certain limitations in practical applications. However, in actual traffic scenes, license plate images often present characteristics such as multi-angle, blur, and low resolution, which brings great challenges to license plate recognition.

[0003] Existing license plate recognition technologies mainly include methods based on template matching, feature extraction, and deep learning. The template matching-based method locates and recognizes license plates by matching with pre-defined license plate templates, but is sensitive to license plate angles and deformations. The feature extraction-based method manually designs some features, such as color features, texture features, etc., and then uses classifiers for recognition, but the selection and extraction of features requires a lot of experience and trial and error. The deep learning-based method uses convolutional neural networks to automatically learn the feature representation of license plate images and performs end-to-end recognition, achieving good results. However, the existing deep learning methods are mainly aimed at front license plate images. For license plate images with multiple angles, tilts, and deformations, the recognition accuracy still needs to be improved.

[0004] In addition, the image preprocessing, license plate positioning, and character segmentation in the license plate recognition process also face some challenges. The quality of the license plate image directly affects the recognition accuracy. How to accurately locate the license plate area under complex backgrounds is a key issue. The tilt and deformation of the license plate will make character segmentation difficult. How to design a robust character segmentation algorithm is also a difficulty. Traditional binarization algorithms and character segmentation algorithms are not adaptable enough to multi-angle and blurred license plate images.

[0005] In view of this, the present invention proposes a system, method, electronic device and storage medium for automatically recognizing and processing multi-angle license plate images. Summary of the invention

[0006] To achieve the above purpose, the present invention provides a system, method, electronic device and storage medium for automatically recognizing and processing multi-angle license plate images. The specific technical solutions are as follows: In a first aspect, the present invention provides a method for automatically recognizing and processing multi-angle license plate images, comprising: Acquire multi-angle license plate images, use an adaptive threshold segmentation algorithm, automatically determine the binarization threshold according to the local pixel distribution characteristics of the license plate area, perform binarization on the license plate image, and obtain the contour image of the license plate; Use perspective transformation to correct the license plate contour images at different angles, and calculate the homography matrix between the license plate quadrilateral contour and the rectangular template to eliminate the deformation of the license plate contour image; Extract the local binary pattern features and HSV color histogram features of the rectified license plate contour image, and use the support vector machine (SVM) classifier to cut the character area of ​​the license plate; Based on the cut image of the license plate character area, the character segmentation result in the license plate image is corrected according to the license plate character arrangement rule; Build a multi-layer convolutional neural network, take license plate character images as input, identify character image categories, and map them into license plate information output to achieve automatic license plate recognition.

[0007] Preferably, obtain the license plate image at multiple angles, and record the original license plate image at any angle as ,in and Respectively represent the row coordinates and column coordinates of the original license plate image; The license plate image is segmented by adaptive threshold and the Sauvola adaptive threshold segmentation algorithm is used to segment the license plate image. Perform binarization and calculate the adaptive threshold for each pixel ; According to the adaptive threshold Binarize the image: ; in, Represents the license plate image after binarization, where the points with a pixel value of 0 correspond to the license plate area, and the points with a pixel value of 1 correspond to the background area; For the binary license plate image Perform morphological processing, including: use The rectangular structural element pair Perform an opening operation to eliminate the The noise points of pixels; use The rectangular structural element of the closed operation is used to close the result of the opening operation, and the area of ​​the filled license plate area is less than The pixel holes; use The cross-shaped structural element of the closed operation is used to expand the closed operation result to connect the pixels that may be disconnected in the license plate area; The cross-shaped structural element of the expansion result is corroded to remove the noise in the license plate area; The binary image of the license plate after morphological processing is recorded as , size and same; The binary image of the license plate processed from morphology using the Suzuki85 boundary tracking algorithm Extract all closed contours from the image, and select the contour corresponding to the license plate image based on the geometric features of the contour, such as area, aspect ratio and rectangularity. ;set up ,in, is the number of pixels on the contour, For the outline The coordinates of the pixel points.

[0008] Preferably, based on the extracted license plate contour, a Douglas-Peucker algorithm is used to perform polygon fitting to obtain an approximate polygon of the license plate contour. ,in is the number of vertices of the fitted polygon, , that is, fitting into a quadrilateral; let the vertex coordinates of the license plate quadrilateral obtained by fitting be: ; in is the upper left vertex of the license plate quadrilateral, is the upper right vertex of the license plate quadrilateral, is the lower right vertex of the license plate quadrilateral, is the lower left vertex of the license plate quadrilateral; Set a rectangular template with the same size as the license plate: ; in, , , , , and are the width and height of the license plate respectively; According to the correspondence between the four points, the homography matrix is ​​calculated using the least squares method. , so that: ; in, is the homography matrix; through the homography matrix , for the binary license plate image Perform perspective transformation to obtain the corrected license plate image : ;in, is the pixel coordinate on the original license plate image, is the pixel coordinates on the corrected license plate image; The rectified license plate image Crop according to the rectangular template to obtain the rectified license plate image.

[0009] Preferably, for the corrected license plate image, a local binary pattern operator is used to extract texture features, and the local binary pattern operator is defined as: ; in, Indicates the number of selected neighborhood pixels, is the serial number of the neighborhood pixel, represents the neighborhood radius, is the pixel coordinates on the corrected license plate image, is the gray value of the center pixel, is the gray value of the neighborhood pixel, is a symbolic function: For each pixel ,calculate Value, get LBP feature image ; Will Divide into Non-overlapping local regions, calculate the LBP feature histogram for each local region : ; in, , , , , , All are parameters. is the Kronecker function; Concatenate the LBP feature histograms of all local regions into a vector , as the local binary pattern feature of the license plate image; Extract the HSV color histogram features and convert the corrected license plate image Color image Convert from RGB color space to HSV color space to get images with three channels: H, S, and V , , ; Calculate the color histogram for each channel separately , , : ; in, represents the color channel, , is the index of the color bin, is the number of colors; concatenate the color histograms of the three channels into a vector , as the HSV color histogram feature of the license plate image; The extracted local binary pattern features and HSV color histogram features Concatenate into feature vectors , as the feature representation of the license plate image.

[0010] Preferably, the positions and category information of the character regions of the feature vectors of a batch of license plate images are annotated; the feature vectors of the license plate images and the corresponding character region information and position information are used as sample data sets to train a support vector machine (SVM) classifier; For the license plate image to be cut, extract the feature vector , use the trained SVM classifier to classify each pixel position of the license plate image Classify and obtain the location and category information of the character area; transform the feature vector Input to SVM classifier and calculate feature vector Distance to the hyperplane : ; in, Represents pixels The eigenvector at ; and Respectively represent the weight and bias parameters of the support vector machine SVM classifier; if , then Mark as character area, otherwise mark as background area; Post-process the character area obtained by the SVM classifier to obtain the final character segmentation result. The steps include: Merge adjacent character areas to obtain complete character connected areas; calculate the circumscribed rectangle for each character connected area as the preliminary result of character segmentation; adjust the segmentation result based on the prior knowledge of the size and aspect ratio of the license plate characters, and filter out pseudo-character areas with abnormal aspect ratios; adjust the segmentation result based on the position of the character area in the license plate image and the character spacing to obtain the final character segmentation image.

[0011] Preferably, a multi-layer convolutional neural network is designed, wherein the multi-layer convolutional neural network includes multiple convolutional layers, pooling layers and fully connected layers for license plate character recognition; The multi-layer convolutional neural network includes an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer, wherein the input layer is used to receive a character cutting image; the convolution layer includes a first convolution layer and a second convolution layer; the pooling layer includes a first pooling layer and a second pooling layer; the fully connected layer includes a first fully connected layer and a second fully connected layer; The first convolutional layer includes: using The size is The convolution kernel of the input image is convolved to obtain feature maps; The first pooling layer includes: performing a maximum pooling operation on the output of the first convolutional layer to reduce the size of the feature map; the second convolutional layer includes: using The size is The convolution kernel of performs convolution operation on the output of the first pooling layer, and obtains feature maps; the second pooling layer: performs a maximum pooling operation on the output of the second convolutional layer to further reduce the size of the feature map; the first fully connected layer stretches the output of the second pooling layer into a one-dimensional vector and connects it to The second fully connected layer connects the output of the first fully connected layer to neurons through a fully connected way. neurons, of which is the number of character categories; the output layer performs Softmax normalization on the output of the second fully connected layer to obtain the probability of each character category; The cut character images and the corresponding character category labels are used as training data, and the convolutional neural network is trained using the back propagation algorithm and the stochastic gradient descent method; the training sample is ,in For the Character images, is the corresponding character category label; the loss function of the convolutional neural network is the cross entropy loss function.

[0012] Preferably, for the license plate image to be recognized, a character cutting algorithm is used to cut it into individual character images; the cut character images are input into a trained convolutional neural network to obtain a character category probability vector corresponding to each character image; and the character category with the largest probability is taken as the recognition result: ;in, For the The recognition results of character images are is the character index number; Arrange the recognition results of the character images according to the position of the characters in the license plate to obtain the complete license plate number: ;in, To identify the license plate number, is the number of license plate characters.

[0013] An automatic recognition and processing system for multi-angle license plate images, which is used to implement the automatic recognition and processing method for multi-angle license plate images, includes an image acquisition module, an image correction module, a character cutting module, a character correction module and a character recognition module; The image acquisition module acquires multi-angle license plate images, uses an adaptive threshold segmentation algorithm, automatically determines a binarization threshold according to the local pixel distribution characteristics of the license plate area, performs binarization processing on the license plate image, and acquires a contour image of the license plate; The image correction module uses perspective transformation to correct the license plate contour images at different angles, and eliminates the deformation of the license plate contour image by calculating the homography matrix between the quadrilateral contour of the license plate and the rectangular template; The character cutting module extracts the local binary pattern features and HSV color histogram features of the corrected license plate contour image, and uses a support vector machine (SVM) classifier to perform character region cutting on the license plate; The character correction module corrects the character segmentation result in the license plate image based on the cut image of the license plate character area and according to the license plate character arrangement rule; The character recognition module builds a multi-layer convolutional neural network, takes the license plate character image as input, identifies the character image category, and maps it into license plate information output to achieve automatic license plate recognition.

[0014] An electronic device comprises: a processor and a memory, wherein the memory stores a computer program for the processor to call; the processor executes the automatic recognition and processing method of multi-angle license plate images by calling the computer program stored in the memory.

[0015] A computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the automatic recognition and processing method for multi-angle license plate images.

[0016] Beneficial effects of the present invention: The adaptive threshold segmentation provided by the present invention overcomes the influence of illumination and license plate aging, accurately extracts the license plate contour, and lays a foundation for subsequent processing.

[0017] The present invention corrects the perspective deformation of the license plate through perspective transformation, restores the tilted and deformed license plate to the front perspective, and provides standardized input for character cutting and recognition.

[0018] The present invention extracts local binary patterns and HSV color histogram features, uses SVM to perform character segmentation, and accurately locates the position and range of characters.

[0019] The present invention post-processes the cutting result according to the character arrangement rule, solves the problems of adhesion and incompleteness, and improves the cutting integrity and accuracy.

[0020] The present invention constructs a multi-layer convolutional neural network to recognize character images end-to-end, directly outputs license plate numbers, and realizes full-process automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A flow chart of the automatic recognition and processing method of multi-angle license plate images provided by the present invention; Figure 2 This is a structural diagram of the automatic recognition and processing system for multi-angle license plate images provided by the present invention. DETAILED DESCRIPTION

[0022] In order to better understand the present invention, a more detailed description will be made of various aspects of the present invention with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0023] In the accompanying drawings, the size, dimensions and shapes of the elements have been slightly adjusted for ease of illustration. The accompanying drawings are for illustration only and are not strictly drawn to scale. As used herein, the terms "substantially", "approximately" and similar terms are used as terms of approximation, not as terms of degree, and are intended to illustrate the inherent deviations in measurements or calculations that will be recognized by those of ordinary skill in the art. In addition, in the present invention, the order in which the steps are described does not necessarily represent the order in which these processes occur in actual operation, unless otherwise specified or can be derived from the context.

[0024] It should also be understood that expressions such as "comprises", "including", "having", "includes" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present invention, "may" is used to mean "one or more embodiments of the present invention". And, the term "exemplary" is intended to refer to an example or illustration.

[0025] Unless otherwise defined, all terms (including engineering terms and scientific and technological terms) used in this document have the same meaning as commonly understood by ordinary technicians in the field to which the present invention belongs. It should also be understood that unless otherwise clearly stated in the present invention, words defined in commonly used dictionaries should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense.

[0026] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] Example 1

[0028] Reference Figure 1 The first embodiment of the present invention provides an automatic recognition and processing method for multi-angle license plate images.

[0029] S1: Acquire multi-angle license plate images, use an adaptive threshold segmentation algorithm, automatically determine the binarization threshold according to the local pixel distribution characteristics of the license plate area, perform binarization on the license plate image, and obtain the contour image of the license plate.

[0030] The target vehicle is photographed at different viewing angles (front, side, oblique, etc.) and distances (such as 1 meter, 3 meters, 5 meters, etc.) to obtain multi-angle license plate images; the original license plate image obtained at any angle is recorded as ,in and Represent the row and column coordinates of the original license plate image respectively.

[0031] Preprocess the original license plate image. The following preprocessing operations are performed: a) Use a 5×5 Gaussian filter to The standard deviation of the image is used to smooth the image and remove high-frequency noise; b) a median filter of size 3×3 is used to filter the smoothed image to further eliminate salt and pepper noise; c) a histogram equalization algorithm is used to enhance the contrast of the filtered image to improve the contrast between the license plate area and the background; d) the image is filtered using Gamma correction is used to adjust the brightness of the enhanced image to make the brightness distribution of the license plate area more uniform; the pre-processed license plate image is denoted as , the size is the same as the original license plate image.

[0032] The Sauvola adaptive threshold segmentation algorithm is used to segment the preprocessed license plate image. Perform binarization processing; Set the local window size to 15×15, i.e. in pixels Calculate the mean in a 15×15 neighborhood centered on and standard deviation ;Pick , image grayscale range ; Calculate the adaptive threshold of each pixel according to the Sauvola algorithm formula : ; and binarize the image according to the adaptive threshold: ; in, Represents the license plate image after binarization. The points with a pixel value of 0 correspond to the license plate area, and the points with a pixel value of 1 correspond to the background area.

[0033] For the binary license plate image Perform morphological processing, including: use The rectangular structural element pair Perform an opening operation to eliminate the Pixel noise points; use The rectangular structural element of the closed operation is used to close the result of the opening operation, and the area of ​​the filled license plate area is less than Pixel holes; use The cross-shaped structural element of the closed operation is used to expand the closed operation result to connect the pixels that may be disconnected in the license plate area; The cross-shaped structural element of the expansion result is corroded to remove small noise blocks in the license plate area; The binary image of the license plate after morphological processing is recorded as , whose size is same; The binary image of the license plate processed from morphology using the Suzuki85 boundary tracking algorithm All closed contours are extracted from the image, and the contour corresponding to the license plate area is selected based on the geometric features of the contour, such as area, aspect ratio and rectangularity. .

[0034] set up ,in, is the number of pixels on the contour, For the outline The coordinates of the pixel points.

[0035] In step S1, the multi-angle license plate images are preprocessed and binarized through an adaptive threshold segmentation algorithm, which overcomes the difficulty of license plate area segmentation under different shooting conditions, improves the robustness and accuracy of license plate positioning, and lays a good foundation for subsequent license plate recognition.

[0036] S2: Use perspective transformation to correct the license plate contour images at different angles, and eliminate the deformation of the license plate contour image by calculating the homography matrix between the license plate quadrilateral contour and the rectangular template.

[0037] For the extracted license plate contour , use the Douglas-Peucker algorithm to perform polygon fitting and obtain an approximate polygon of the license plate outline ,in is the number of vertices of the fitted polygon, , that is, fitting into a quadrilateral.

[0038] Assume that the vertex coordinates of the license plate quadrilateral obtained by fitting are: ; in is the upper left vertex of the license plate quadrilateral, is the upper right vertex of the license plate quadrilateral, is the lower right vertex of the license plate quadrilateral, It is the lower left vertex of the license plate quadrilateral.

[0039] Set a rectangular template with the same size as the license plate: ; in, , , , , and are the width and height of the license plate respectively; According to the correspondence between the four points, the homography matrix is ​​calculated using the least squares method. , so that: ; in, is a 3×3 homography matrix, which can be expressed as: ; Substituting the correspondence between the four points into the above formula, we can get 8 equations, forming a linear equation system; using the singular value decomposition (SVD) method to solve the equation system, we get the homography matrix The estimated value of .

[0040] Using the estimated homography matrix , for the binary license plate image Perform perspective transformation to obtain the corrected license plate image : ;in, is the pixel coordinate on the original license plate image, is the pixel coordinate of the corrected license plate image.

[0041] Since perspective transformation may introduce black invalid areas, it is necessary to correct the license plate image. Crop the rectified license plate image Crop according to the rectangular template to obtain the rectified license plate image.

[0042] The cropping area can be based on the width and height ratio of the license plate Adaptively determine, for example, the size of the cropped license plate image can be ,in , , retain the central area of ​​the license plate image, Indicates rounding down.

[0043] Local binary pattern feature extraction, for the rectified license plate binary image , the local binary pattern operator is used to extract texture features.

[0044] Define the LBP operator as: ; in, Indicates the number of selected neighborhood pixels, is the serial number of the neighborhood pixel, represents the neighborhood radius, is the pixel coordinates on the corrected license plate image, is the gray value of the center pixel, is the gray value of the neighborhood pixel, is a symbolic function, and the function expression is as follows: ; in, is the symbolic function argument.

[0045] For each pixel ,calculate Value, get LBP feature image .

[0046] Will Divide into Non-overlapping local regions, calculate the LBP feature histogram for each local region : ; in, , , , , , All are parameters. is the Kronecker function: ; Concatenate the LBP feature histograms of all local regions into a vector , as the local binary pattern feature of the license plate image.

[0047] Extract the HSV color histogram features and convert the corrected license plate image Color image Convert from RGB color space to HSV color space to get images with three channels: H, S, and V , , ; Calculate the color histogram for each channel separately , , : ; in, represents the color channel, , is the index of the color bin, is the number of colors; concatenate the color histograms of the three channels into a vector , as the HSV color histogram feature of the license plate image.

[0048] The extracted local binary pattern features and HSV color histogram features Concatenate into a feature vector , as the feature representation of the license plate image.

[0049] Step S2 uses perspective transformation to correct the license plate image, eliminating the deformation caused by the change of viewing angle, restoring the license plate to the standard viewing angle, and extracting local binary pattern features and HSV color histogram features, comprehensively characterizing the texture and color information of the license plate, providing rich discriminant information for license plate character cutting.

[0050] S3: Extract the local binary pattern features and HSV color histogram features of the rectified license plate contour image, and use the support vector machine (SVM) classifier to cut the character area of ​​the license plate.

[0051] Label the position and category information of the character area of ​​the feature vectors of a batch of license plate images; use the feature vectors of these license plate images and the corresponding character area information and position information as sample data sets to train a support vector machine (SVM) classifier.

[0052] The goal of the SVM classifier is to find an optimal hyperplane , so that positive and negative samples can be correctly classified by the hyperplane, and the distance between the hyperplane and the nearest sample point (i.e., the classification interval) is maximized.

[0053] For the license plate image to be cut, extract its feature vector , use the trained SVM classifier to classify each pixel position of the license plate image Classify and obtain the location and category information of the character area; transform the feature vector Input to SVM classifier and calculate feature vector Distance to the hyperplane : ; in, Indicates location The eigenvector at ; where and Respectively represent the weight and bias parameters of the support vector machine SVM classifier; if , then Marked as character area, otherwise marked as background area.

[0054] In step S3, the SVM classifier is used to cut the character area of ​​the license plate image. The classifier is trained by manually annotating samples, which can effectively identify the location and category information of the character area. The cutting result is corrected in combination with the license plate character arrangement rules, which further improves the accuracy and completeness of character cutting.

[0055] S4: Based on the cut image of the license plate character area, according to the license plate character arrangement rule, correct the character segmentation result in the license plate image.

[0056] Post-process the character area obtained by the SVM classifier to obtain the final character segmentation result. The steps include: Merge adjacent character regions to obtain complete character connected regions; calculate the circumscribed rectangle of each character connected region as the preliminary result of character segmentation; adjust the segmentation result based on the prior knowledge of the size and aspect ratio of the license plate characters, and filter out pseudo-character regions with too small size and abnormal aspect ratio; adjust the segmentation result based on the position of the character region in the license plate image and the character spacing to obtain the final character segmentation image.

[0057] Step S4 uses a cutting-based method to perform license plate character recognition, overcoming problems such as character adhesion and incompleteness. Through post-processing operations such as character region merging, size and spacing adjustment, a standardized and complete character cutting image is obtained, which provides high-quality input for subsequent character recognition and improves the overall performance of license plate recognition.

[0058] S5: Build a multi-layer convolutional neural network, take the license plate character image as input, identify the character image category, and map it into license plate information output to achieve automatic license plate recognition.

[0059] A multi-layer convolutional neural network is designed, which includes multiple convolutional layers, pooling layers and fully connected layers for license plate character recognition.

[0060] The multi-layer convolutional neural network includes an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer. The input layer is used to receive a character cutting image; the convolution layer includes a first convolution layer and a second convolution layer; the pooling layer includes a first pooling layer and a second pooling layer; the fully connected layer includes a first fully connected layer and a second fully connected layer.

[0061] The first convolutional layer includes: using The size is The convolution kernel of the input image is convolved to obtain feature maps; the convolution operation can be expressed as: ; in, For the feature map, is the input image, and Respectively The weights and biases of the convolution kernels, and is the position inside the convolution kernel, Represents the input image and the output pixel The corresponding pixel coordinates.

[0062] The first pooling layer includes: performing a maximum pooling operation on the output of the first convolutional layer to reduce the size of the feature map; the pooling operation can be expressed as: ; in, is the feature map after pooling, is the size of the pooling window.

[0063] The second convolutional layer includes: using The size is The convolution kernel of performs convolution operation on the output of the first pooling layer, and obtains feature maps; the convolution operation is similar to the first convolution layer.

[0064] The second pooling layer: performs a maximum pooling operation on the output of the second convolutional layer to further reduce the size of the feature map; the pooling operation is similar to that of the first pooling layer.

[0065] The first fully connected layer: stretches the output of the second pooling layer into a one-dimensional vector and connects it to neurons; the fully connected operation can be expressed as: ; in, is the output of the first fully connected layer, is the output of the second pooling layer, and is the width and height of the feature map, and are the weights and biases of the first fully connected layer, is the neuron number of the fully connected layer.

[0066] The second fully connected layer: connects the output of the first fully connected layer to neurons, of which is the number of character categories; the fully connected operation is similar to the first fully connected layer.

[0067] The output layer includes performing Softmax normalization on the output of the second fully connected layer to obtain the probability of each character category: ; Among them, e is a natural constant, For the The probability of character categories, is the output of the second fully connected layer.

[0068] The cut character images and the corresponding character category labels are used as training data, and the convolutional neural network is trained using the back propagation algorithm and the stochastic gradient descent method; the training sample is ,in For the Character images, is the corresponding character category label; the network loss function is the cross entropy loss function.

[0069] By minimizing the loss function, the back-propagation algorithm is used to calculate the gradient of the network parameters (convolution kernel weights, biases, and fully connected layer weights), and the stochastic gradient descent method is used to update the parameters.

[0070] For the license plate image to be recognized, use the character cutting algorithm to cut it into single character images; input the cut character images into the trained convolutional neural network to obtain the character category probability vector corresponding to each character image; take the character category with the largest probability as the recognition result: ;in, For the The recognition results of character images are The character index number.

[0071] Arrange the character recognition results in the order of the characters' positions in the license plate to get the complete license plate number: ;in, To identify the license plate number, is the number of license plate characters.

[0072] In step S5, a multi-layer convolutional neural network is built for license plate character recognition, and the hierarchical features of the character image are extracted using operations such as convolution and pooling. Through end-to-end training and Softmax classification, the license plate number can be recognized efficiently and accurately, realizing the full process automation of license plate recognition, which has good practical value.

[0073] Example 2

[0074] Reference Figure 2 The second embodiment of the present invention provides an automatic recognition and processing system for multi-angle license plate images.

[0075] The system includes an image acquisition module, an image correction module, a character cutting module, a character correction module and a character recognition module.

[0076] The image acquisition module acquires multi-angle license plate images, adopts an adaptive threshold segmentation algorithm, automatically determines a binarization threshold according to the local pixel distribution characteristics of the license plate area, performs binarization processing on the license plate image, and acquires a contour image of the license plate.

[0077] The image correction module uses perspective transformation to correct the license plate contour images at different angles, and eliminates the deformation of the license plate contour image by calculating the homography matrix between the quadrilateral contour of the license plate and the rectangular template.

[0078] The character cutting module extracts the local binary pattern features and HSV color histogram features of the corrected license plate contour image, and uses a support vector machine (SVM) classifier to perform character region cutting on the license plate; The character correction module corrects the character segmentation result in the license plate image based on the cut image of the license plate character area and according to the license plate character arrangement rule.

[0079] The character recognition module builds a multi-layer convolutional neural network, takes the license plate character image as input, identifies the character image category, and maps it into license plate information output to achieve automatic license plate recognition.

[0080] Example 3

[0081] The present invention also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method for automatically recognizing and processing multi-angle license plate images as described above may be executed.

[0082] The method or system according to the embodiment of the present invention may also be implemented with the aid of the architecture of the electronic device of the present invention.

[0083] An electronic device may include a bus, one or more CPUs, read-only memory (ROM), random access memory (RAM), a communication port for connecting to a network, input / output components, a hard disk, and the like.

[0084] The storage device in the electronic device, such as ROM or hard disk, can store the automatic recognition and processing method of multi-angle license plate images provided by the present invention.

[0085] The invention discloses an automatic recognition and processing method for multi-angle license plate images, comprising: obtaining multi-angle license plate images, using an adaptive threshold segmentation algorithm, automatically determining a binarization threshold according to the local pixel distribution characteristics of the license plate area, binarizing the license plate image, and obtaining a contour image of the license plate; correcting the license plate contour images at different angles by using perspective transformation, and eliminating the deformation of the license plate contour image by calculating the homography matrix of the license plate quadrilateral contour and the rectangular template; extracting the local binary pattern features and HSV color histogram features of the corrected license plate contour image, and using a support vector machine (SVM) classifier to perform character region cutting on the license plate; based on the cut image of the license plate character area, correcting the character segmentation result in the license plate image according to the license plate character arrangement rule; building a multi-layer convolutional neural network, taking the license plate character image as input, identifying the character image category, and mapping it into license plate information output, so as to realize automatic recognition of the license plate.

[0086] Furthermore, the electronic device may also include a user interface. Of course, the architecture of the present invention is only exemplary, and when implementing different devices, one or more components in the electronic device disclosed in the present invention may be omitted according to actual needs.

[0087] Example 4

[0088] The invention also discloses a computer-readable storage medium.

[0089] The computer-readable storage medium has computer-readable instructions stored thereon.

[0090] When the computer-readable instructions are executed by a processor, the method for automatically recognizing and processing multi-angle license plate images according to the embodiments of the present invention described with reference to the above drawings can be executed.

[0091] The storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. In addition, according to an embodiment of the present invention, the process described above with reference to the flowchart may be implemented as a computer software program.

[0092] For example, the present invention provides a non-temporary machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be run by a processor to execute instructions corresponding to the method steps provided by the present invention, for example: obtaining multi-angle license plate images, using an adaptive threshold segmentation algorithm, automatically determining a binarization threshold according to the local pixel distribution characteristics of the license plate area, binarizing the license plate image, and obtaining a contour image of the license plate; using perspective transformation to correct the license plate contour images at different angles, and eliminating the deformation of the license plate contour image by calculating the homography matrix of the license plate quadrilateral contour and the rectangular template; extracting the local binary pattern features and HSV color histogram features of the corrected license plate contour image, and using a support vector machine SVM classifier to perform character area cutting on the license plate; based on the cut image of the license plate character area, correcting the character segmentation result in the license plate image according to the license plate character arrangement rule; building a multi-layer convolutional neural network, taking the license plate character image as input, identifying the character image category, and mapping it into license plate information output, to achieve automatic recognition of the license plate.

[0093] When the computer program is executed by the central processing unit (CPU), the above functions defined in the method of the present invention are performed. The method, apparatus, and device of the present invention may be implemented in many ways. For example, the method, apparatus, and device of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware.

[0094] The above sequence for the steps of the method is for illustration only, and the steps of the method of the present invention are not limited to the sequence specifically described above unless otherwise specifically stated.

[0095] In addition, in some embodiments, the present invention can also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0096] In addition, the parts of the above technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0097] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for automatically recognizing and processing multi-angle license plate images, characterized in that: include: Acquire multi-angle license plate images, use an adaptive threshold segmentation algorithm, automatically determine the binarization threshold according to the local pixel distribution characteristics of the license plate area, perform binarization on the license plate image, and obtain the contour image of the license plate; Use perspective transformation to correct the license plate contour images at different angles, and calculate the homography matrix between the license plate quadrilateral contour and the rectangular template to eliminate the deformation of the license plate contour image; Extract the local binary pattern features and HSV color histogram features of the rectified license plate contour image, and use the support vector machine (SVM) classifier to cut the character area of ​​the license plate; Based on the cut image of the license plate character area, the character segmentation result in the license plate image is corrected according to the license plate character arrangement rule; Build a multi-layer convolutional neural network, take license plate character images as input, identify character image categories, and map them into license plate information output to achieve automatic license plate recognition.

2. The automatic recognition and processing method of multi-angle license plate images according to claim 1 is characterized in that: Get multi-angle license plate images, and remember that the original license plate image at any angle is ,in and Respectively represent the row coordinates and column coordinates of the original license plate image; The license plate image is segmented by adaptive threshold and the Sauvola adaptive threshold segmentation algorithm is used to segment the license plate image. Perform binarization and calculate the adaptive threshold for each pixel ; According to the adaptive threshold Binarize the image: ； in, Represents the license plate image after binarization, where the points with a pixel value of 0 correspond to the license plate area, and the points with a pixel value of 1 correspond to the background area; For the binary license plate image Perform morphological processing, including: use The rectangular structural element pair Perform an opening operation to eliminate the Pixel noise points; use The rectangular structural element of the closed operation is used to close the result of the opening operation, and the area of ​​the filled license plate area is less than Pixel holes; use The cross-shaped structural element of the closed operation is used to expand the closed operation result to connect the pixels that may be disconnected in the license plate area; The cross-shaped structural element of the expansion result is corroded to remove the noise in the license plate area; The binary image of the license plate after morphological processing is recorded as , size and same; The binary image of the license plate processed from morphology using the Suzuki85 boundary tracking algorithm Extract all closed contours from the image, and select the contour corresponding to the license plate image based on the geometric features of the contour, such as area, aspect ratio and rectangularity. ;set up ,in, is the number of pixels on the contour, For the outline The coordinates of the pixel points.

3. The automatic recognition and processing method of multi-angle license plate images according to claim 2 is characterized in that: Based on the extracted license plate contour, the Douglas-Peucker algorithm is used for polygon fitting to obtain an approximate polygon of the license plate contour. ,in is the number of vertices of the fitted polygon, , that is, fitting into a quadrilateral; let the vertex coordinates of the license plate quadrilateral obtained by fitting be: ； in is the upper left vertex of the license plate quadrilateral, is the upper right vertex of the license plate quadrilateral, is the lower right vertex of the license plate quadrilateral, is the lower left vertex of the license plate quadrilateral; Set a rectangular template with the same size as the license plate: ； in, , , , , and are the width and height of the license plate respectively; According to the correspondence between the four points, the homography matrix is ​​calculated using the least squares method. , so that: ； in, is the homography matrix; through the homography matrix , for the binary license plate image Perform perspective transformation to obtain the corrected license plate image : ;in, is the pixel coordinate on the original license plate image, is the pixel coordinates on the corrected license plate image; The rectified license plate image Crop according to the rectangular template to obtain the rectified license plate image.

4. The method for automatically recognizing and processing multi-angle license plate images according to claim 3 is characterized in that: For the rectified license plate image, the local binary pattern operator is used to extract texture features, and the local binary pattern operator is defined as: ； in, Indicates the number of selected neighborhood pixels, is the serial number of the neighborhood pixel, represents the neighborhood radius, is the pixel coordinates on the corrected license plate image, is the gray value of the center pixel, is the gray value of the neighborhood pixel, is a symbolic function: For each pixel ,calculate Value, get LBP feature image ; Will Divide into Non-overlapping local regions, calculate the LBP feature histogram for each local region : ； in, , , , , , All are parameters. is the Kronecker function; Concatenate the LBP feature histograms of all local regions into a vector , as the local binary pattern feature of the license plate image; Extract the HSV color histogram features and convert the corrected license plate image Color image Convert from RGB color space to HSV color space to get images with three channels: H, S, and V , , ; Calculate the color histogram for each channel separately , , : ； in, represents the color channel, , is the index of the color bin, is the number of colors; concatenate the color histograms of the three channels into a vector , as the HSV color histogram feature of the license plate image; The extracted local binary pattern features and HSV color histogram features Concatenate into feature vectors , as the feature representation of the license plate image.

5. The method for automatically recognizing and processing multi-angle license plate images according to claim 4 is characterized in that: Label the position and category information of the character regions of the feature vectors of a batch of license plate images; use the feature vectors of the license plate images and the corresponding character region information and position information as sample data sets to train a support vector machine (SVM) classifier; For the license plate image to be cut, extract the feature vector , use the trained SVM classifier to classify each pixel position of the license plate image Classify and obtain the location and category information of the character area; transform the feature vector Input to SVM classifier and calculate feature vector Distance to the hyperplane : ； in, Represents pixels The eigenvector at ; and Respectively represent the weight and bias parameters of the support vector machine SVM classifier; if , then Mark as character area, otherwise mark as background area; Post-process the character area obtained by the SVM classifier to obtain the final character segmentation result. The steps include: Merge adjacent character areas to obtain complete character connected areas; calculate the circumscribed rectangle for each character connected area as the preliminary result of character segmentation; adjust the segmentation result based on the prior knowledge of the size and aspect ratio of the license plate characters, and filter out pseudo-character areas with abnormal aspect ratios; adjust the segmentation result based on the position of the character area in the license plate image and the character spacing to obtain the final character segmentation image.

6. The method for automatically recognizing and processing multi-angle license plate images according to claim 5 is characterized in that: Design a multi-layer convolutional neural network, which includes multiple convolutional layers, pooling layers and fully connected layers for license plate character recognition; The multi-layer convolutional neural network includes an input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer, wherein the input layer is used to receive a character cutting image; the convolution layer includes a first convolution layer and a second convolution layer; the pooling layer includes a first pooling layer and a second pooling layer; the fully connected layer includes a first fully connected layer and a second fully connected layer; The first convolutional layer includes: using The size is The convolution kernel of the input image is convolved to obtain feature maps; The first pooling layer includes: performing a maximum pooling operation on the output of the first convolutional layer to reduce the size of the feature map; the second convolutional layer includes: using The size is The convolution kernel of performs convolution operation on the output of the first pooling layer, and obtains feature maps; the second pooling layer: performs a maximum pooling operation on the output of the second convolutional layer to further reduce the size of the feature map; the first fully connected layer stretches the output of the second pooling layer into a one-dimensional vector and connects it to The second fully connected layer connects the output of the first fully connected layer to neurons through a fully connected way. neurons, of which is the number of character categories; the output layer performs Softmax normalization on the output of the second fully connected layer to obtain the probability of each character category; The cut character images and the corresponding character category labels are used as training data, and the convolutional neural network is trained using the back propagation algorithm and the stochastic gradient descent method; the training sample is ,in For the Character images, is the corresponding character category label; the loss function of the network is the cross entropy loss function.

7. The method for automatically recognizing and processing multi-angle license plate images according to claim 6 is characterized in that: For the license plate image to be recognized, use the character cutting algorithm to cut it into single character images; input the cut character images into the trained convolutional neural network to obtain the character category probability vector corresponding to each character image; take the character category with the largest probability as the recognition result: ;in, For the The recognition results of character images are is the character index number; Arrange the recognition results of the character images according to the position of the characters in the license plate to obtain the complete license plate number: ;in, To identify the license plate number, is the number of license plate characters.

8. A system for automatically recognizing and processing multi-angle license plate images, which is used to implement the method for automatically recognizing and processing multi-angle license plate images according to any one of claims 1 to 7, characterized in that: It includes an image acquisition module, an image correction module, a character cutting module, a character correction module and a character recognition module; The image acquisition module acquires multi-angle license plate images, uses an adaptive threshold segmentation algorithm, automatically determines a binarization threshold according to the local pixel distribution characteristics of the license plate area, performs binarization processing on the license plate image, and acquires a contour image of the license plate; The image correction module uses perspective transformation to correct the license plate contour images at different angles, and eliminates the deformation of the license plate contour image by calculating the homography matrix between the quadrilateral contour of the license plate and the rectangular template; The character cutting module extracts the local binary pattern features and HSV color histogram features of the corrected license plate contour image, and uses a support vector machine (SVM) classifier to perform character region cutting on the license plate; The character correction module corrects the character segmentation result in the license plate image based on the cut image of the license plate character area and according to the license plate character arrangement rule; The character recognition module builds a multi-layer convolutional neural network, takes the license plate character image as input, identifies the character image category, and maps it into license plate information output to achieve automatic license plate recognition.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a computer program for the processor to call; the processor executes the automatic recognition and processing method of multi-angle license plate images according to any one of claims 1 to 7 by calling the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: Instructions are stored, and when the instructions are executed on a computer, the computer executes the automatic recognition and processing method for multi-angle license plate images as described in any one of claims 1 to 7.

Images