A license plate recognition method and device based on deep learning
Through deep learning methods, combined with convolutional neural network and feature pyramid network, end-to-end processing of license plate recognition is realized, the problems of license plate positioning and angle influence are solved, and the accuracy and efficiency of license plate recognition are improved.
Patent Information
- Application Number
- CN202110775812.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-07-09
AI Technical Summary
The existing license plate recognition technology has problems such as inaccurate positioning of license plates, the angle of license plates affects the recognition effect, and a large dependence on data volume, resulting in low recognition accuracy.
Using a license plate recognition method based on deep learning, image features are extracted and fused through convolutional neural networks, combined with feature pyramid network structure, high-level semantic features are extracted from different scales, and a predetermined target detection network is used for license plate detection and segmentation, and multi-task loss function is trained. Optimize the license plate recognition model to achieve end-to-end license plate recognition.
It improves the accuracy and efficiency of license plate recognition, effectively solves the problem of low recognition accuracy caused by inclination angle, and can achieve high-accurate license plate recognition with a small amount of labeled data.
Smart Images

Figure CN113392837B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a license plate recognition method and device based on deep learning. Background Art
[0002] License plate recognition technology plays an important role in many tasks such as urban traffic management, vehicle identification, parking fee management, and violation handling. However, due to the influence of many factors such as lighting conditions, weather conditions, image clarity, license plate shooting angle, and license plate color, license plate recognition is still a challenging task.
[0003] License plate recognition includes two tasks. One is the detection of the license plate position, that is, obtaining the position of the license plate area from the captured image; the other is the recognition of the license plate characters, that is, recognizing the visible characters in the detected license plate area. In the prior art, license plate recognition mainly includes two methods. One is a two-stage license plate recognition method. This type of method generally detects the position of the license plate first, and then recognizes the license plate characters based on the detected license plate. This type of method requires the detection and recognition modules to be trained and tuned separately, and cannot give full play to the highly related and complementary relationship between the two. The other is an end-to-end license plate recognition method, which trains the detection and recognition processes together to directly obtain the license plate recognition result. However, the existing end-to-end license plate recognition method has the following defects: On the one hand, it cannot be trained in a completely end-to-end manner, but the detector is trained first, and then the detector is trained. The recognizer is loaded in to continue training, because the training of license plate recognition requires accurate license plate position positioning, and the license plate position positioning in early iterations is often inaccurate, which also affects the effect of license plate recognition; on the other hand, due to the different shooting angles of license plates in the real world, the position of the license plate area on the image is not completely horizontal, but presented in various shapes, resulting in inaccurate license plate recognition in practical applications; on the other hand, the existing license plate recognition methods based on fully connected networks or recurrent neural networks for output are highly dependent on the amount of data and require massive license plate data for learning. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for license plate recognition based on deep learning, which realize end-to-end license plate recognition and greatly improve the accuracy of license plate recognition.
[0005] On the one hand, an embodiment of the present invention provides a license plate recognition method based on deep learning, comprising:
[0006] Acquire multiple image frames containing vehicle license plate information, extract features of each image frame through a convolutional neural network, and fuse the extracted features of each image frame to obtain a feature representation of each image frame;
[0007] Detecting feature representations of each image frame through a predetermined target detection network to obtain category and position information of a license plate detection frame of each image frame, and segmenting the license plate of each image frame according to the category and position information to obtain feature maps of each image frame after segmentation;
[0008] Obtaining original license plate label information in each image frame, and training a license plate recognition model based on the original license plate label information and the feature graphs of each image frame;
[0009] The image to be identified is input into the license plate recognition model for license plate recognition, and the license plate characters in the image to be identified are obtained.
[0010] Furthermore, the extracting features of each image frame by a convolutional neural network and fusing the extracted features of each image frame to obtain a feature representation of each image frame includes:
[0011] A feature pyramid network structure is added to the convolutional neural network, and high-level semantic features of each image frame are extracted from different scales through the feature pyramid network structure;
[0012] The extracted high-level semantic features of each image frame are fused to obtain the feature representation of each image frame.
[0013] Furthermore, segmenting the license plate of each image frame according to the category and position information to obtain the feature map of each image frame after segmentation includes:
[0014] According to the category and position information of the license plate detection frame of each image frame, each image frame of a first predetermined size is subjected to a first predetermined number of convolution operations and a second predetermined number of deconvolution operations to extract a feature map of a second predetermined size corresponding to each image frame.
[0015] Furthermore, before the step of training a license plate recognition model according to each original license plate label information and the feature map of each image frame, the method further comprises:
[0016] According to each original license plate label information, each original license plate label is converted into a global character feature map for the global characters of the license plate and a single character feature map for a single character of the license plate.
[0017] Furthermore, the license plate recognition model is trained based on the original license plate label information and the feature map of each image frame, including:
[0018] Calculating each global character feature map, each single character feature map, and the feature map of each image frame using a predetermined loss function to obtain a calculation result;
[0019] Based on each original license plate label information, the calculation result is corrected by the loss function to train a license plate recognition model.
[0020] Further, the predetermined loss function is a multi-task loss function, and the multi-task loss function includes a loss function of a license plate target detection stage and a loss function of a license plate segmentation stage;
[0021] The multi-task loss function is determined by respectively setting the weight coefficients of the loss function of the license plate target detection stage and the loss function of the license plate segmentation stage.
[0022] Furthermore, the loss function of the license plate target detection stage includes a position regression function and a classification loss function, and the loss function of the license plate target detection stage is determined by setting weight coefficients of the position regression function and the classification loss function respectively;
[0023] The loss function of the license plate segmentation stage includes a global license plate instance segmentation loss function and a semantic segmentation loss function representing each character, wherein the global license plate instance segmentation loss function is calculated using a binary cross entropy loss function, and the semantic segmentation loss function is calculated using a weighted loss function.
[0024] Furthermore, the step of inputting the image to be identified into the license plate recognition model to perform license plate recognition and identifying the license plate characters in the image to be identified includes:
[0025] Inputting the image to be recognized into the license plate recognition model to obtain a license plate detection frame of the image to be recognized;
[0026] Detecting the license plate detection frame of the image to be recognized through a predetermined target detection network in the license plate recognition model to obtain a global character feature map of the image to be recognized and a single character feature map of the image to be recognized;
[0027] According to the license plate detection frame of the image to be identified, the average pixel value of each character area in the single character feature map of the image to be identified is calculated by a predetermined algorithm to generate a license plate character sequence;
[0028] The license plate characters in the image to be recognized are obtained according to the license plate character sequence.
[0029] On the other hand, an embodiment of the present invention provides a license plate recognition device based on deep learning, comprising:
[0030] The extraction and fusion module is used to obtain multiple image frames containing vehicle license plate information, extract the features of each image frame through a convolutional neural network, and fuse the extracted features of each image frame to obtain a feature representation of each image frame;
[0031] A detection and segmentation module, configured to detect feature representations of each image frame through a predetermined target detection network, obtain category and position information of a license plate detection frame of each image frame, and segment the license plate of each image frame according to the category and position information to obtain a feature map of each image frame after segmentation;
[0032] A training module is used to obtain the original license plate label information in each image frame, and train a license plate recognition model according to the original license plate label information and the feature map of each image frame;
[0033] The recognition module is used to input the image to be recognized into the license plate recognition model for license plate recognition, and recognize the license plate characters in the image to be recognized.
[0034] Furthermore, the extraction and fusion module includes:
[0035] An extraction unit, used to add a feature pyramid network structure to the convolutional neural network, and extract high-level semantic features of each image frame from different scales through the feature pyramid network structure;
[0036] The fusion unit is used to fuse the extracted high-level semantic features of each image frame to obtain a feature representation of each image frame.
[0037] Furthermore, the detection and segmentation module includes:
[0038] The extraction unit is used to perform a first predetermined number of convolution operations and a second predetermined number of deconvolution operations on each image frame of a first predetermined size according to the category and position information of the license plate detection frame of each image frame, so as to extract a feature map of a second predetermined size corresponding to each image frame.
[0039] Further, including:
[0040] The conversion module is used to convert each original license plate label into a global character feature map for the global characters of the license plate and a single character feature map for a single character of the license plate according to each original license plate label information.
[0041] Furthermore, the training module includes:
[0042] A calculation unit, used to calculate each global character feature map, each single character feature map and the feature map of each image frame by using a predetermined loss function to obtain a calculation result;
[0043] The training unit is used to correct the calculation result through the loss function based on each original license plate label information to train a license plate recognition model.
[0044] Further, the predetermined loss function is a multi-task loss function, and the multi-task loss function includes a loss function of a license plate target detection stage and a loss function of a license plate segmentation stage;
[0045] The multi-task loss function is determined by respectively setting the weight coefficients of the loss function of the license plate target detection stage and the loss function of the license plate segmentation stage.
[0046] Furthermore, the loss function of the license plate target detection stage includes a position regression function and a classification loss function, and the loss function of the license plate target detection stage is determined by setting weight coefficients of the position regression function and the classification loss function respectively;
[0047] The loss function of the license plate segmentation stage includes a global license plate instance segmentation loss function and a semantic segmentation loss function representing each character, wherein the global license plate instance segmentation loss function is calculated using a binary cross entropy loss function, and the semantic segmentation loss function is calculated using a weighted loss function.
[0048] Furthermore, the identification module is specifically used for
[0049] Inputting the image to be recognized into the license plate recognition model to obtain a license plate detection frame of the image to be recognized;
[0050] Detecting the license plate detection frame of the image to be recognized through a predetermined target detection network in the license plate recognition model to obtain a global character feature map of the image to be recognized and a single character feature map of the image to be recognized;
[0051] According to the license plate detection frame of the image to be identified, the average pixel value of each character area in the single character feature map of the image to be identified is calculated by a predetermined algorithm to generate a license plate character sequence;
[0052] The license plate characters in the image to be recognized are obtained according to the license plate character sequence.
[0053] The above technical scheme has the following beneficial effects: through the present invention, by utilizing a small amount of more accurate character-level license plate annotation data and using a multi-layer convolutional neural network to extract features from the image, features of different scales can be extracted, and a license plate detection frame is generated through a predetermined target detection network, and the license plate detection frame is segmented to obtain accurate license plate recognition results, thereby realizing end-to-end license plate recognition and effectively solving the problem of low license plate recognition accuracy due to a large inclination angle. At the same time, the characteristics of the license plate are fully utilized, greatly improving the accuracy and efficiency of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0055] Figure 1 This is a flow chart of a method for license plate recognition based on deep learning in an embodiment of the present invention;
[0056] Figure 2 This is a schematic diagram of the structure of a device for license plate recognition based on deep learning in another embodiment of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] The above technical scheme of the embodiment of the present invention has the following beneficial effects: through the present invention, by utilizing a small amount of more accurate character-level license plate annotation data and using a multi-layer convolutional neural network to extract features from the image, features of different scales can be extracted, and a license plate detection frame is generated through a predetermined target detection network, and the license plate detection frame is segmented to obtain accurate license plate recognition results, thereby realizing end-to-end license plate recognition and effectively solving the problem of low license plate recognition accuracy due to a large inclination angle. At the same time, the characteristics of the license plate are fully utilized, greatly improving the accuracy and efficiency of license plate recognition.
[0059] The above technical solution of the embodiment of the present invention is described in detail below with reference to application examples:
[0060] The application example of the present invention aims to realize end-to-end license plate recognition, greatly improving the accuracy of license plate recognition.
[0061] In a possible implementation, during the license plate recognition process, a license plate recognition model is pre-trained; during the training of the license plate recognition model, first, multiple image frames containing vehicle license plate information are obtained, features of each image frame are extracted through a convolutional neural network, and the extracted features of each image frame are fused to obtain feature representations of each image frame; then, the feature representations of each image frame are detected through a predetermined target detection network to obtain the category and position information of the license plate detection frame of each image frame, and the license plate of each image frame is segmented according to the category and position information to obtain feature maps of each segmented image frame; the original license plate label information in each image frame is obtained, and the license plate recognition model is trained according to the original license plate label information and the feature maps of each image frame; during license plate recognition, the picture to be recognized is input into the license plate recognition model for license plate recognition, and the license plate characters in the picture to be recognized are recognized.
[0062] It should be noted that, in the embodiments of the present invention, convolutional neural networks include but are not limited to AlexNet, ResNet (Residual Network), VGG (Visual Geometry Group Network, VGG model), etc., and the predetermined target detection network includes but is not limited to Faster-RCNN (RCNN refers to Region-based Convolutional Neural Networks, region-based convolutional neural network, Fast-RCNN is an upgraded version of RCNN), YOLO (You Only LookOnce, an object recognition and positioning algorithm based on deep neural network), SSD (Single Shot Multibox Detector, target detection algorithm) and other two-stage or single-stage target detection networks; wherein, the picture containing the vehicle license plate information covers images of the same vehicle under different perspectives, different backgrounds, and different light intensities captured by real road traffic scene monitoring, and the content covered by the vehicle image includes license plate appearance information, license plate character information, etc.
[0063] In a possible implementation, features of each image frame are extracted through a convolutional neural network, and the extracted features of each image frame are fused to obtain feature representations of each image frame, including: adding a feature pyramid network structure to the convolutional neural network, extracting high-level semantic features of each image frame from different scales through the feature pyramid network structure; and feature fusion of the extracted high-level semantic features of each image frame to obtain feature representations of each image frame.
[0064] Among them, the license plate of each image frame is segmented according to the category and position information to obtain the feature map of each image frame after segmentation, including: according to the category and position information of the license plate detection frame of each image frame, each image frame of a first predetermined size is subjected to a first predetermined number of convolution operations and a second predetermined number of deconvolution operations, respectively, to extract the feature map of a second predetermined size corresponding to each image frame.
[0065] For example, in the process of license plate recognition, a license plate recognition model is pre-trained; in the process of training the license plate recognition model, first, a plurality of image frames containing vehicle license plate information are obtained, and a feature pyramid network structure (FPN) is added to the convolutional neural network, and high-level semantic features of each image frame are extracted from different scales through the feature pyramid network structure; the extracted high-level semantic features of each image frame are feature fused to obtain feature representations of each image frame; then, the feature representations of each image frame are detected through a predetermined target detection network to obtain the category and position information of the license plate detection frame of each image frame, and according to the category and position information of the license plate detection frame of each image frame, each image frame of a first predetermined size is subjected to a first predetermined number of convolution operations and a second predetermined number of deconvolution operations to extract each image frame. The corresponding feature map of the second predetermined size, for example, the feature size of the input image frame is 16×64×256 (H×W×C), where H represents height, W represents width, and C represents the number of channels; after four convolutions and one deconvolution operation for further feature extraction, the output feature size becomes 32×128×N, which means that the output channel number is N and the size is 32×128. The number of N is 86, including a global license plate instance feature map, 84 license plate character feature maps, and 1 license plate character background feature map. The global license plate instance feature map can accurately locate the license plate character area. The 84 license plate character feature maps include 10 Arabic numerals (0-9), 24 uppercase letters (AZ, except I and O) and 50 Chinese characters; the background feature map of the license plate character is a feature map of the background area that does not include the character area.
[0066] In a possible implementation, before the step of training a license plate recognition model based on each original license plate label information and the feature map of each image frame, it includes: converting each original license plate label into a global character feature map for the global characters of the license plate and a single character feature map for a single character of the license plate based on each original license plate label information.
[0067] Among them, according to each original license plate label information and the feature map of each image frame, a license plate recognition model is trained, including: calculating each global character feature map, each single character feature map and the feature map of each image frame through a predetermined loss function to obtain a calculation result; based on each original license plate label information, the calculation result is corrected through the loss function to train the license plate recognition model.
[0068] For example, before license plate recognition, multiple image frames containing vehicle license plate information are annotated with license plate label information by manual annotation. During the training of the license plate recognition model, the original license plate label in each image is converted into a global character feature map for the global characters of the license plate and a single character feature map for a single character of the license plate; wherein the original license plate label includes a global character label and a single character label for each character, wherein the global character label is represented by P = {p1, p2...p m} means that p i The position information of each license plate area is represented by C = {c1 = (cc1, cl1), c2 = (cc2, cl2), ..., c n =(cc n ,cl n )} means, where cc j and cl j Respectively represent the category and position of each license plate character. For the conversion of global character labels, first initialize a mask with the same size as the input feature map, that is, the first predetermined size, such as the 16×64×256 (H×W×C) size in the above example, with a pixel value of 0, and then set the pixel value of the area in the license plate frame to 1 according to the coordinate position of the global character label, and then obtain the global character feature map label; for the generation of single character feature map labels, first initialize a mask with the same size as the output feature map, that is, the second predetermined size, such as the 32×128×N size in the above example, with a pixel value of 0, and then set the pixel value in the bounding box to the same pixel value as the character category index value according to the bounding box position of each character, so as to obtain a character-level feature map label, wherein the character category index value is 1 to 84, a total of 84 categories; then, through a predetermined loss function, such as a multi-task loss function, each global character feature map, each single character feature map and the feature map of each image frame are calculated to obtain a calculation result; based on the original license plate label information, the calculation result is corrected by the loss function, and the license plate recognition model is trained.
[0069] It should be noted that those skilled in the art will understand that a mask is a binary image consisting of 0s and 1s. When a mask is applied to a function, the 1-valued area is processed and the masked 0-valued area is not included in the calculation. An image mask is defined by specifying data values, data ranges, finite or infinite values, regions of interest, and annotation files. Any combination of the above options can also be used as input to create a mask.
[0070] It should also be noted that, in the embodiment of the present invention, in data annotation, it is different from the traditional data annotation method. The traditional license plate annotation method usually uses a rectangular frame to represent the entire license plate, and the annotation information includes the vertex coordinates of the upper left corner of the license plate and the width and height of the license plate, as well as the license plate characters; in the embodiment of the present invention, a quadrilateral annotation method is adopted, and four vertices are used to represent the position of the entire license plate, that is, the x-coordinates and y-coordinates of the four vertices of the upper left corner, upper right corner, lower right corner, and lower left corner of the license plate are respectively annotated. This annotation method can more accurately represent the inclined license plate, and at the same time, each character in the license plate area is also annotated using a polygonal frame to obtain the license plate position, each license plate character position, and license plate character category. Although such an annotation method is more complicated than the traditional annotation method, compared with the millions of image annotation data required by the current common license plate recognition algorithm, the present invention can achieve better results when using tens of thousands of annotation data. The license plate recognition algorithm of the present invention can achieve better license plate recognition effects using less annotation data, which greatly improves the efficiency of license plate recognition.
[0071] In a possible implementation, the predetermined loss function is a multi-task loss function, which includes a loss function of a license plate target detection stage and a loss function of a license plate segmentation stage; wherein the multi-task loss function is determined by respectively setting weight coefficients of the loss function of the license plate target detection stage and the loss function of the license plate segmentation stage; the loss function of the license plate target detection stage includes a position regression function and a classification loss function, and the loss function of the license plate target detection stage is determined by respectively setting weight coefficients of the position regression function and the classification loss function; the loss function of the license plate segmentation stage includes a global license plate instance segmentation loss function and a semantic segmentation loss function representing each character, wherein the global license plate instance segmentation loss function is calculated using a binary cross entropy loss function, and the semantic segmentation loss function is calculated using a weighted loss function.
[0072] For example, in the process of training the license plate recognition model, continuing with the above example, each global character feature map, each single character feature map and the feature map of each image frame are sent to the multi-task loss function, which includes the loss function of the license plate target detection stage and the loss function of the segmentation stage. Specifically, the multi-task loss function is as follows:
[0073] L=α1L det +α2L seg ………………(Formula 1)
[0074] Among them, L det represents the loss function of the license plate target detection stage, L seg It represents the loss function of the license plate character segmentation stage, α1 and α2 are weight coefficients, which are set to 1.
[0075] The loss function L in the license plate target detection stage det The following is formula 2:
[0076] L det =λ1L reg +λ2L cls ………………(Formula 2)
[0077] Among them, L reg represents the position regression loss function, L cls represents the classification loss function, λ1 and λ2 represent weight coefficients, which are set to 1 respectively, and the regression loss function L reg And the classification loss function L cls The loss function commonly used in target detection can be used.
[0078] The loss function L in the license plate character segmentation stage seg The following is formula three:
[0079] L seg =β1L ins +β2L char ………………(Formula 3)
[0080] Among them, β1 and β2 represent weight coefficients, which are set to 1, L ins represents the global license plate instance segmentation loss function, which is calculated using the binary cross entropy loss function as shown in Formula 4:
[0081]
[0082] Where p represents the predicted category, y=0 and y=1 represent the actual category is 0 or 1; L char The semantic segmentation loss function for each character is calculated using the weighted softmax loss function as shown in Formula 5:
[0083]
[0084] in, It is the standard format of the softmax loss function, which means to find the output probability of the current element. is the output of the current element, represents the output of all elements, N represents the number of pixels in each feature map, C represents the number of license plate character categories, Y represents the label corresponding to the output feature map X, and W represents the weight, which is used to balance the loss value of the characters and the background. The pixel format of the background class is N neg Indicates that the category index value of the background class is 0, and the weight is calculated as follows:
[0085]
[0086] The difference between the value in the annotated license plate label information is calculated through the multi-task loss function, and the gradient is calculated according to the multi-task loss function to train and optimize the license plate recognition network model parameters.
[0087] It should be noted that those skilled in the art can understand that the original meaning of gradient is a vector (vector), which means that the directional derivative of a function at this point reaches the maximum value along this direction, that is, the function changes the fastest at this point along this direction (the direction of this gradient) and the rate of change is the largest (which is the modulus of the gradient).
[0088] In a possible implementation, the step of inputting a picture to be recognized into the license plate recognition model for license plate recognition and identifying the license plate characters in the picture to be recognized includes: inputting the picture to be recognized into the license plate recognition model to obtain a license plate detection frame of the picture to be recognized; detecting the license plate detection frame of the picture to be recognized by a predetermined target detection network in the license plate recognition model to obtain a global character feature map of the picture to be recognized and a single character feature map of the picture to be recognized; calculating the average pixel value of each character area in the single character feature map of the picture to be recognized by a predetermined algorithm based on the license plate detection frame of the picture to be recognized to generate a license plate character sequence; and obtaining the license plate characters in the picture to be recognized based on the license plate character sequence.
[0089] For example, when recognizing a license plate, a picture to be recognized is input, and a candidate frame of the license plate area is obtained through a convolutional neural network and a license plate detection network. The NMS (Non-Maximum Suppression) algorithm is used to remove some redundant frames, and the retained license plate detection output frame is sent to the segmentation branch to generate a global character feature map and a single character feature map; then, the contour of the text area is calculated on the global character feature map to directly obtain the predicted accurate license plate position boundary box, and a predetermined algorithm, such as a pixel voting algorithm, is used on the single character feature map to generate a license plate character sequence. The pixel voting algorithm steps are as follows: first, the background feature map is binarized, and the binarization threshold is 192. All license plate character regions are obtained according to the connected areas in the binarized feature map. Then, for each license plate character region, the average pixel value of each character region in all character feature maps is calculated, that is, the character category probability of the region, where the character category with the maximum average value will be assigned to the region. Then, the character category of each region is obtained in turn, and the complete license plate character can be obtained according to the reading and writing habits from left to right.
[0090] An embodiment of the present invention provides a license plate recognition device based on deep learning, which can implement the method embodiment provided above. For specific function implementation, please refer to the description in the method embodiment, which will not be repeated here.
[0091] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.
[0092] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.
[0093] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.
[0094] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".
[0095] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.
[0096] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.
[0097] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.
[0098] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.
[0099] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A license plate recognition method based on deep learning, characterized in that: include: Acquire multiple image frames containing vehicle license plate information, extract features of each image frame through a convolutional neural network, and fuse the extracted features of each image frame to obtain a feature representation of each image frame; The extracting features of each image frame by using a convolutional neural network includes: A feature pyramid network structure is added to the convolutional neural network, and high-level semantic features of each image frame are extracted from different scales through the feature pyramid network structure; The extracted high-level semantic features of each image frame are subjected to feature fusion to obtain feature representation of each image frame; Detecting feature representations of each image frame through a predetermined target detection network to obtain category and position information of a license plate detection frame of each image frame, and segmenting the license plate of each image frame according to the category and position information to obtain feature maps of each image frame after segmentation; Obtaining original license plate label information in each image frame, and training a license plate recognition model based on the original license plate label information and the feature graphs of each image frame; Inputting the image to be identified into the license plate recognition model to perform license plate recognition, and identifying the license plate characters in the image to be identified; The step of segmenting the license plate of each image frame according to the category and position information to obtain a feature map of each image frame after segmentation includes: According to the category and position information of the license plate detection frame of each image frame, each image frame of a first predetermined size is subjected to a first predetermined number of convolution operations and a second predetermined number of deconvolution operations to extract a feature map of a second predetermined size corresponding to each image frame; before the step of training a license plate recognition model according to each original license plate label information and the feature map of each image frame, the method includes: According to each original license plate label information, each original license plate label is converted into a global character feature map for the global characters of the license plate and a single character feature map for a single character of the license plate; The training of a license plate recognition model based on the original license plate label information and the feature map of each image frame includes: Calculating each global character feature map, each single character feature map, and the feature map of each image frame using a predetermined loss function to obtain a calculation result; Based on each original license plate label information, the calculation result is corrected by the loss function to train a license plate recognition model; The predetermined loss function is a multi-task loss function, and the multi-task loss function includes a loss function of a license plate target detection stage and a loss function of a license plate segmentation stage; The multi-task loss function is determined by respectively setting weight coefficients of the loss function of the license plate target detection stage and the loss function of the license plate segmentation stage; The loss function of the license plate target detection stage includes a position regression function and a classification loss function, and the loss function of the license plate target detection stage is determined by setting weight coefficients of the position regression function and the classification loss function respectively; The loss function of the license plate segmentation stage includes a global license plate instance segmentation loss function and a semantic segmentation loss function representing each character, wherein the global license plate instance segmentation loss function is calculated using a binary cross entropy loss function, and the semantic segmentation loss function is calculated using a weighted loss function; The step of inputting the image to be recognized into the license plate recognition model to perform license plate recognition, and recognizing the license plate characters in the image to be recognized includes: Inputting the image to be recognized into the license plate recognition model to obtain a license plate detection frame of the image to be recognized; Detecting the license plate detection frame of the image to be recognized through a predetermined target detection network in the license plate recognition model to obtain a global character feature map of the image to be recognized and a single character feature map of the image to be recognized; According to the license plate detection frame of the image to be identified, the average pixel value of each character area in the single character feature map of the image to be identified is calculated by a predetermined algorithm to generate a license plate character sequence; The license plate characters in the image to be recognized are obtained according to the license plate character sequence.
2. A license plate recognition device based on deep learning, characterized in that: include: The extraction and fusion module is used to obtain multiple image frames containing vehicle license plate information, extract the features of each image frame through a convolutional neural network, and fuse the extracted features of each image frame to obtain a feature representation of each image frame; A detection and segmentation module, configured to detect feature representations of each image frame through a predetermined target detection network, obtain category and position information of a license plate detection frame of each image frame, and segment the license plate of each image frame according to the category and position information to obtain a feature map of each image frame after segmentation; A training module is used to obtain the original license plate label information in each image frame, and train a license plate recognition model according to the original license plate label information and the feature map of each image frame; A recognition module, used for inputting the image to be recognized into the license plate recognition model to perform license plate recognition, and recognize the license plate characters in the image to be recognized; The extraction and fusion module comprises: An extraction unit, used to add a feature pyramid network structure to the convolutional neural network, and extract high-level semantic features of each image frame from different scales through the feature pyramid network structure; A fusion unit, used for fusing the extracted high-level semantic features of each image frame to obtain a feature representation of each image frame; The detection and segmentation module comprises: an extraction unit, configured to perform a first predetermined number of convolution operations and a second predetermined number of deconvolution operations on each image frame of a first predetermined size according to the category and position information of the license plate detection frame of each image frame, and extract a feature map of a second predetermined size corresponding to each image frame; A conversion module, used for converting each original license plate label into a global character feature map for the global characters of the license plate and a single character feature map for a single character of the license plate according to each original license plate label information; The training module comprises: A calculation unit, used to calculate each global character feature map, each single character feature map and the feature map of each image frame by using a predetermined loss function to obtain a calculation result; A training unit, used for correcting the calculation result by the loss function based on each original license plate label information, and training a license plate recognition model; The predetermined loss function is a multi-task loss function, and the multi-task loss function includes a loss function of a license plate target detection stage and a loss function of a license plate segmentation stage; The multi-task loss function is determined by respectively setting weight coefficients of the loss function of the license plate target detection stage and the loss function of the license plate segmentation stage; The loss function of the license plate target detection stage includes a position regression function and a classification loss function, and the loss function of the license plate target detection stage is determined by setting weight coefficients of the position regression function and the classification loss function respectively; The loss function of the license plate segmentation stage includes a global license plate instance segmentation loss function and a semantic segmentation loss function representing each character, wherein the global license plate instance segmentation loss function is calculated using a binary cross entropy loss function, and the semantic segmentation loss function is calculated using a weighted loss function; The identification module is specifically used for Inputting the image to be recognized into the license plate recognition model to obtain a license plate detection frame of the image to be recognized; Detecting the license plate detection frame of the image to be recognized through a predetermined target detection network in the license plate recognition model to obtain a global character feature map of the image to be recognized and a single character feature map of the image to be recognized; According to the license plate detection frame of the image to be identified, the average pixel value of each character area in the single character feature map of the image to be identified is calculated by a predetermined algorithm to generate a license plate character sequence; The license plate characters in the image to be recognized are obtained according to the license plate character sequence.
Citation Information
Patent Citations
A license plate recognition method based on depth learning
CN109145900A
End-to-end license plate recognition system and method based on deep learning
CN110378331A
Multi-target vehicle track real-time construction method based on traffic monitoring video
CN111429484A
License plate detection method and device, equipment and storage medium
CN111767915A
License plate recognition method based on automatic character area perception
CN112287941A
