License plate character recognition method and device

The character connection relationship matrix and feature extraction are constructed through the license plate graph convolution neural network, which solves the robustness and accuracy of the traditional license plate recognition method, and realizes the accurate recognition of license plate characters and support for multi-layer license plates.

CN114519791BActive Publication Date: 2024-11-08AI SUPER EYE TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202210127201.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-11
Publication Date
2024-11-08
Estimated Expiration
2042-02-11

AI Technical Summary

Technical Problem

Traditional license plate character recognition methods are poorly robust and have poor recognition effects, so they cannot accurately identify license plate characters.

Method used

The license plate image convolution neural network is adopted to obtain the character detection box coordinates and identification numbers in the license plate image, build a character connection relationship matrix, perform feature extraction and training, output predicted character connection relationship matrix, adjust network parameters, and achieve the accuracy of license plate characters. Identify.

Benefits of technology

It improves the accuracy and robustness of license plate recognition, and can identify multi-layer license plates. It is suitable for different types of license plates, such as two-layer truck and motorcycle license plates, improving the recognition speed and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114519791B_ABST
    Figure CN114519791B_ABST
Patent Text Reader

Abstract

The present application discloses a license plate character recognition method and device. The method comprises: matching multiple character connection pairs with connection relationships according to the coordinate position of the character detection frame and the character detection frame identification number to form a real character connection relationship matrix, and then forming a connection relationship matrix for each character detection frame; performing feature extraction transformation on the license plate image according to the coordinate position of the character detection frame to obtain an input feature vector for each character; inputting the connection relationship matrix and the input feature vector of each character detection frame into a character graph convolutional neural network to output a predicted character connection relationship matrix; constructing a loss function to adjust parameters according to the predicted character connection relationship matrix and the real character connection relationship matrix to obtain a trained character graph convolutional neural network, predicting the license plate image to be tested to obtain the character connection relationship matrix, and obtaining the license plate corresponding to the license plate image to be tested according to the character connection relationship matrix and the character category of each character.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a license plate character recognition method and device. Background Art

[0002] License plate recognition technology plays an important role in many tasks such as urban traffic management, vehicle identification, parking fee management, and violation handling. It is still a challenging task due to the influence of many factors such as lighting conditions, weather conditions, image clarity, license plate shooting angle, and license plate color. Among them, license plate recognition includes two tasks: one is the detection of license plate position, that is, locating the position of the license plate area from the captured image; the other is the recognition of license plate characters, that is, recognizing the license plate characters in the detected license plate area.

[0003] Traditional license plate character recognition methods use template connection to perform character recognition, segmenting, binarizing, and resizing the license plate recognition characters to make their resized size the same as the size of the template in the character database, and then connect the preprocessed license plate recognition characters with all templates, and select the best connection as the recognition result. However, the template connection of the traditional license plate character recognition method is simple to implement, but has poor robustness and poor recognition effect, and cannot accurately recognize each character of the license plate.

[0004] Application Contents

[0005] The purpose of this application is to solve the technical problem that the traditional license plate character recognition method has poor robustness and poor recognition effect, which leads to the inability to accurately recognize license plate characters. To achieve the above purpose, this application provides a license plate character recognition method and device.

[0006] The present application provides a license plate character recognition method, comprising:

[0007] Acquire multiple license plate images, and perform character detection on the multiple license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each of the license plate images;

[0008] Matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs;

[0009] According to the real character connection relationship matrix, forming a connection relationship matrix for each character detection frame;

[0010] According to the coordinate position of the character detection frame, the license plate image is subjected to feature extraction transformation to obtain an input feature vector of each character;

[0011] Input the connection relationship matrix of each character detection frame and the input feature vector of each character into the character graph convolutional neural network, and output the predicted character connection relationship matrix;

[0012] Constructing a loss function according to the predicted character connection relationship matrix and the real character connection relationship matrix, and adjusting the parameters of the character graph convolutional neural network according to the loss function to obtain a trained character graph convolutional neural network;

[0013] According to the trained character graph convolutional neural network, the license plate image to be tested is predicted.

[0014] A character connection relationship matrix is ​​obtained, and a license plate corresponding to the license plate image to be tested is obtained according to the character connection relationship matrix and the character category of each of the characters.

[0015] In one embodiment, matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs, comprises:

[0016] Arranging all the character detection frames in each of the license plate images according to the coordinate positions of the character detection frames and the identification numbers of the character detection frames to obtain at least one character arrangement layer;

[0017] In each of the character arrangement layers, matching the connection relationship between two adjacent character detection frames to obtain the plurality of character connection pairs;

[0018] The real character connection relationship matrix is ​​formed according to the plurality of character connection pairs.

[0019] In one embodiment, the matching of a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs, further comprises:

[0020] In the plurality of character arrangement layers, a character detection frame corresponding to the right side of a first character arrangement layer and a character detection frame corresponding to the left side of a second character arrangement layer in two adjacent character arrangement layers are matched for connection relationship to obtain the character connection pair.

[0021] In one embodiment, the connection relationship matrix of each character detection frame and the input feature vector of each character are input into a character graph convolutional neural network, and the predicted character connection relationship matrix is ​​output, including:

[0022] Construct the character graph convolutional neural network, which is:

[0023]

[0024]

[0025] Wherein, X represents the input feature vector of each character, A represents the connection relationship matrix of each character detection frame, represents the adjacency matrix of each of the character detection frames, I represents the unit matrix of each of the character detection frames, Represents the degree matrix of each character detection box, W (0) With W (1) represents the parameters of the character graph convolutional neural network, ReLU represents a nonlinear activation function, and softmax represents a normalization function;

[0026] According to the character graph convolutional neural network, the predicted character connection relationship matrix is ​​output.

[0027] In one embodiment, the performing character detection on the plurality of license plate images to obtain the character category, the character detection frame coordinate position and the character detection frame identification number of each character in each of the license plate images includes:

[0028] Performing character detection on the plurality of license plate images to obtain character categories, character detection frame coordinate positions, and character detection frame identification numbers of all characters in each of the license plate images;

[0029] According to the coordinate position of the character detection frame and the identification number of the character detection frame, determining whether the number of license plate character digits corresponding to each license plate image is greater than a preset number of license plate character digits;

[0030] If the number of license plate character digits is greater than the preset number of license plate character digits, the confidence levels of all character detection frames in each license plate image are sorted from high to low to obtain a high-confidence character detection frame corresponding to the preset number of license plate character digits;

[0031] According to the high-confidence character detection frame corresponding to the preset number of license plate character digits, the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image are obtained.

[0032] In one embodiment, the performing character detection on the plurality of license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each of the license plate images further includes:

[0033] Annotating the multiple license plate images to obtain multiple single-character annotated license plate images;

[0034] Performing data enhancement processing on the multiple single-character labeled license plate images to obtain multiple data enhanced license plate images;

[0035] Training a character detection model according to the multiple data-enhanced license plate images to obtain a trained character detection model;

[0036] According to the trained character detection model, character detection is performed on the license plate image to be tested to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in the license plate image.

[0037] In one embodiment, the present application provides a license plate character recognition device, comprising:

[0038] A data acquisition module, used to acquire a plurality of license plate images, and perform character detection on the plurality of license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each of the license plate images;

[0039] A real character connection relationship matrix acquisition module, used for matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs;

[0040] A character detection frame connection relationship matrix acquisition module is used to form a connection relationship matrix for each character detection frame according to the real character connection relationship matrix;

[0041] An input feature vector acquisition module is used to perform feature extraction transformation on the license plate image according to the coordinate position of the character detection frame to obtain an input feature vector of each character;

[0042] A model building module, used to input the connection relationship matrix of each character detection box and the input feature vector of each character into a character graph convolutional neural network, and output a predicted character connection relationship matrix;

[0043] A training module, used to construct a loss function according to the predicted character connection relationship matrix and the real character connection relationship matrix, and adjust the parameters of the character graph convolutional neural network according to the loss function to obtain a trained character graph convolutional neural network;

[0044] The prediction module is used to predict the license plate image to be tested according to the trained character graph convolutional neural network, obtain a character connection relationship matrix, and obtain the license plate corresponding to the license plate image to be tested according to the character connection relationship matrix and the character category of each character.

[0045] In one embodiment, the real character connection relationship matrix acquisition module includes:

[0046] A character arrangement layer acquisition module, used to arrange all the character detection frames in each of the license plate images according to the coordinate positions of the character detection frames and the identification numbers of the character detection frames to obtain at least one character arrangement layer;

[0047] A first character connection pair acquisition module, configured to match the connection relationship between two adjacent character detection frames in each character arrangement layer to obtain the plurality of character connection pairs;

[0048] The real character connection relationship matrix forming module is used to form the real character connection relationship matrix according to the multiple character connection pairs.

[0049] In one embodiment, the real character connection relationship matrix acquisition module further includes:

[0050] The second character connection pair acquisition module is used to match the connection relationship between the character detection frame corresponding to the right side of the first character arrangement layer and the character detection frame corresponding to the left side of the second character arrangement layer in two adjacent character arrangement layers in the multiple character arrangement layers to obtain the character connection pair.

[0051] In one embodiment, the model building module includes:

[0052] The character graph convolutional neural network construction module is used to construct the character graph convolutional neural network, and the character graph convolutional neural network is:

[0053]

[0054]

[0055] Wherein, X represents the input feature vector of each character, A represents the connection relationship matrix of each character detection frame, represents the adjacency matrix of each of the character detection frames, I represents the unit matrix of each of the character detection frames, Represents the degree matrix of each character detection box, W (0) With W (1) represents the parameters of the character graph convolutional neural network, ReLU represents a nonlinear activation function, and softmax represents a normalization function;

[0056] The predicted character connection relationship matrix acquisition module is used to output the predicted character connection relationship matrix according to the character graph convolutional neural network.

[0057] In one embodiment, the data acquisition module includes:

[0058] All character data acquisition module, used for performing character detection on the plurality of license plate images, obtaining character categories, character detection frame coordinate positions and character detection frame identification numbers of all characters in each of the license plate images;

[0059] A judgment module, used for judging whether the number of license plate character digits corresponding to each license plate image is greater than a preset number of license plate character digits according to the coordinate position of the character detection frame and the identification number of the character detection frame;

[0060] A high confidence data acquisition module is used to sort the confidences of all the character detection frames in each license plate image from high to low if the number of character digits of the license plate is greater than the preset number of character digits of the license plate, and obtain a high confidence character detection frame corresponding to the preset number of character digits of the license plate;

[0061] The first character information acquisition module is used to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image according to the high-confidence character detection frame corresponding to the preset license plate character digit number.

[0062] In one embodiment, the data acquisition module further includes:

[0063] A labeling module, used to label the multiple license plate images to obtain multiple single-character labeled license plate images;

[0064] A data enhancement module, used to perform data enhancement processing on the multiple single-character annotated license plate images to obtain multiple data enhanced license plate images;

[0065] A character detection model forming module, used to train the character detection model according to the plurality of data-enhanced license plate images to obtain a trained character detection model;

[0066] The second character information acquisition module is used to perform character detection on the license plate image to be tested according to the trained character detection model, and obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in the license plate image.

[0067] In the above license plate character recognition method, the coordinate position and identification code of the detection frame corresponding to each license plate character are obtained, and multiple character connection pairs are constructed with each coordinate position and identification code to form a real character connection relationship matrix, so that the license plate character position information is constructed to form graph data. The position relationship between each license plate character is learned through a character graph convolutional neural network to predict the position relationship of the license plate characters, output the connection relationship matrix between each license plate character, and further obtain the arrangement order of each license plate character. According to the arrangement order of each license plate character and based on the character category of each license plate character, a complete license plate can be obtained, and the vehicle license plate can be recognized.

[0068] The license plate character recognition method provided by the present application performs type recognition and relationship matrix construction based on a single license plate character, which reduces the difficulty of detection and recognition, is conducive to model training, and further improves the recognition accuracy. The license plate character recognition method provided by the present application uses a character graph convolutional neural network based on a powerful learning ability to learn the positional relationship between license plate characters, has a low reliance on license plate data and high robustness, and can more accurately predict the connection relationship of each character of the license plate. In addition, the license plate character recognition method provided by the present application, through the learning of the positional relationship between license plate characters, makes the method of the present application not limited to the recognition of one layer of license plate characters, and can realize the recognition of multiple layers of license plate characters, and can support the recognition of special types of license plates such as two-layer trucks, motorcycle license plates, etc. Therefore, through the license plate character recognition method provided by the present application, the license plate can be recognized more accurately, the accuracy and recognition speed of license plate recognition are improved, and the applicability is wider, and it can be widely used in the recognition of multiple layers of license plate characters. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the overall steps of the license plate character recognition method provided in this application.

[0070] Figure 2 It is a structural schematic diagram of the license plate character recognition device provided in this application. DETAILED DESCRIPTION

[0071] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments.

[0072] See also Figure 1 , the present application provides a license plate character recognition method, comprising:

[0073] S10, acquiring multiple license plate images, and performing character detection on the multiple license plate images to obtain the character category, character detection frame coordinate position, and character detection frame identification number of each character in each license plate image;

[0074] S20, matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the character detection frame identification number, and forming a real character connection relationship matrix according to the plurality of character connection pairs;

[0075] S30, forming a connection relationship matrix corresponding to each character detection frame according to the real character connection relationship matrix;

[0076] S40, performing feature extraction transformation on the license plate image according to the coordinate position of the character detection frame to obtain an input feature vector corresponding to each character;

[0077] S50, inputting the connection relationship matrix corresponding to each character detection frame and the input feature vector corresponding to each license plate character into the character graph convolutional neural network, and outputting the predicted character connection relationship matrix;

[0078] S60, constructing a loss function according to the predicted character connection relationship matrix and the real character connection relationship matrix, and adjusting the parameters of the character graph convolutional neural network according to the loss function to obtain a trained character graph convolutional neural network;

[0079] S70, predicting the license plate image to be tested based on the trained character graph convolutional neural network, obtaining a character connection relationship matrix, and obtaining a license plate corresponding to the license plate image to be tested based on the character connection relationship matrix and the character category of each character.

[0080] In S10, the license plate image can be understood as an image containing license plate information, which can be a vehicle license plate image captured by real road traffic scene monitoring, covering license plate images under different viewing angles, different backgrounds, and different light intensities. The license plate image is an image containing license plates of different types of vehicles such as cars, new energy vehicles, buses, motorcycles, electric vehicles, and large trucks. The license plate image can be obtained by extracting an image frame containing a vehicle license plate from a monitoring video, and using a license plate target detection algorithm to perform license plate detection to obtain the corresponding license plate position information, and intercepting the image frame according to the license plate position information and saving it as a license plate image. Alternatively, the license plate image can also be intercepted by manual means. Character detection can be understood as detecting each character of the license plate in the license plate image, and obtaining the character category, character detection frame coordinate position, and character detection frame identification number corresponding to each license plate character. The character category includes 84 license plate character categories, namely 10 Arabic numerals (0-9), 24 capital letters (AZ, except I and O) and 50 Chinese characters. The coordinate position of the character detection frame can be represented by a minimum rectangular frame. Each minimum rectangular box surrounds the character and is represented by the center point coordinates and width and height information. The character detection box identification number can be understood as a label for the coordinate position of each character detection box. The character detection box identification number is unique and corresponds to only one character detection box and only one character detection box coordinate position.

[0081] In S20, a character detection frame coordinate position corresponds to a character detection frame identification number one by one, and can represent a character detection frame. The connection relationship can be understood as whether there is a relative positional relationship between at least two license plate characters. For example: for a certain vehicle license plate, if two license plate characters are adjacent, there is a connection relationship between the two license plate characters, and the two license plate characters form a character connection pair. Two license plate characters are adjacent, which can be understood as being adjacent left and right or up and down. Through screening and matching between multiple license plate characters, multiple character connection pairs can be formed, and based on the multiple character connection pairs, the relationship between each license plate character and other license plate characters can be known to form a real character connection relationship matrix. In the real character connection relationship matrix, if two license plate characters have a connection relationship, the value in the matrix is ​​1, otherwise it is 0.

[0082] In S30, a character detection frame coordinate position and a corresponding character detection frame identification number represent a character detection frame. The real character connection relationship matrix covers the connection relationship between multiple vehicle characters. According to the real character connection relationship matrix, the connection relationship between a character detection frame and other character detection frames can be obtained to form a connection relationship matrix corresponding to each character detection frame.

[0083] In S40, the coordinate information of each character detection frame can be obtained according to the coordinate position of the character detection frame. According to the coordinate information of each character detection frame, the license plate image is extracted to obtain each character detection frame image. Feature extraction and transformation are performed on each character detection frame image to obtain the input feature vector corresponding to each license plate character, so as to further adapt the input to the character graph convolutional neural network and perform model training on the character graph convolutional neural network.

[0084] In S50, the character graph convolutional neural network includes multiple convolutional layers, nonlinear activation layers, etc. based on the graph convolutional neural network, and can be used to process unstructured data. The connection relationship matrix corresponding to each character detection frame represents the position information of each license plate character and other license plate characters, and constructs graph structure data. The input feature vector corresponding to each license plate character and the connection relationship matrix corresponding to each character detection frame are used as the input of the character graph convolutional neural network to perform feature learning on the position relationship between license plate characters and output the predicted character connection relationship matrix.

[0085] In S60, the parameters of the character graph convolutional neural network are optimized by constructing a loss function through the predicted character connection relationship matrix and the real character connection relationship matrix, and finally a trained character graph convolutional neural network is obtained for the prediction in S70.

[0086] In S70, the trained character graph convolutional neural network is used to predict the license plate characters of the license plate image to be tested, and the connection relationship between the license plate characters is obtained. The license plate characters are sorted according to the connection relationship between the license plate characters to form a license plate corresponding to the license plate image to be tested.

[0087] Through S10 to S70, the coordinate position and identification code of the detection frame corresponding to each license plate character are obtained, and multiple character connection pairs are constructed with each coordinate position and identification code to form a real character connection relationship matrix, so that the license plate character position information is constructed to form graph data. The position relationship between each license plate character is learned through the character graph convolutional neural network to predict the position relationship of the license plate characters, and the connection relationship matrix between each license plate character is output to further obtain the arrangement order of each license plate character. According to the arrangement order of each license plate character and based on the character category of each license plate character, a complete license plate can be obtained, and the vehicle license plate can be recognized.

[0088] The license plate character recognition method provided by the present application performs type recognition and relationship matrix construction based on a single license plate character, which reduces the difficulty of detection and recognition, is conducive to model training, and further improves the recognition accuracy. The license plate character recognition method provided by the present application uses a character graph convolutional neural network based on a powerful learning ability to learn the positional relationship between license plate characters, has a low reliance on license plate data and high robustness, and can more accurately predict the connection relationship of each character of the license plate. In addition, the license plate character recognition method provided by the present application, through the learning of the positional relationship between license plate characters, makes the method of the present application not limited to the recognition of one layer of license plate characters, and can realize the recognition of multiple layers of license plate characters, and can support the recognition of special types of license plates such as two-layer trucks, motorcycle license plates, etc. Therefore, through the license plate character recognition method provided by the present application, the license plate can be recognized more accurately, the accuracy and recognition speed of license plate recognition are improved, and the applicability is wider, and it can be widely used in the recognition of multiple layers of license plate characters.

[0089] In one embodiment, in S10, character detection is performed on multiple license plate images to obtain the character category, character detection frame coordinate position, and character detection frame identification number of each license plate character in each license plate image, including:

[0090] S110, annotating a plurality of license plate images to obtain a plurality of license plate images annotated with single characters;

[0091] S120, performing data enhancement processing on a plurality of single-character labeled license plate images to obtain a plurality of data enhanced license plate images;

[0092] S130, training a character detection model according to a plurality of data-enhanced license plate images to obtain a trained character detection model;

[0093] S140, performing character detection on the license plate image to be tested according to the trained character detection model, and obtaining the character category, character detection frame coordinate position and character detection frame identification number of each license plate character in the license plate image.

[0094] In S110, by using a 2D target detection and annotation tool, each license plate character in the license plate image is annotated with a rectangular frame to obtain the annotation label of the license plate image. The annotation label includes the character category, the coordinate position of the character detection frame, and the character detection frame identification number. By annotating multiple license plate images, 84 real character categories of license plates are obtained, including 10 Arabic numerals (0-9), 24 uppercase letters (AZ, except I and O) and 50 Chinese characters. After annotating multiple license plate images, a license plate image with a single character annotation can be obtained. A license plate image with a single character annotation can be understood as each character on the license plate image being annotated.

[0095] In S120, data enhancement processing includes rotation reflection transformation, flip transformation, scaling transformation, translation transformation, scale transformation, contrast transformation, noise disturbance and color transformation. By performing data enhancement processing on multiple single-character labeled license plate images, more equivalent valid data can be generated based on limited real data, that is, multiple data enhanced license plate images are obtained. Furthermore, by performing data enhancement processing on multiple data enhanced license plate images, the distribution of training data can be enriched, the number and diversity of training samples can be increased, and the generalization ability of the trained model can be stronger, thereby improving the robustness of the model. Therefore, through the data enhancement method, various situations can be simulated based on limited real data, such as image defects, occlusions, large tilt angles, etc.

[0096] In S130, multiple data-enhanced license plate images form more abundant training data based on multiple license plate images, which are used to train and form a character detection model, and output the predicted character category, predicted character detection frame coordinate position and predicted character detection frame identification number of each license plate character, which are used to construct a loss function corresponding to the character detection model to optimize the model parameters. In one embodiment, the model parameters in the character detection model are trained and optimized based on the predicted character category, predicted character detection frame coordinate position and predicted character detection frame identification number of each license plate character predicted and output by the character detection model and the actual character category, character detection frame coordinate position and character detection frame identification number in the training sample. After training to the set number of training iterations, the optimal model parameters are obtained, and a stable character detection model is formed.

[0097] In S140, after the trained character detection model is obtained, character detection is performed on the license plate image to be tested, so as to recognize the character category, character detection frame coordinate position and character detection frame identification number of each license plate character in the license plate image to be tested.

[0098] Through S110 to S140, a large amount of license plate image data can be simulated based on a small amount of real license plate image data for model training, which solves the defect of relying on massive data in traditional methods. Through S110 to S140, various situations can be simulated based on limited real data, such as image defects, occlusion, large tilt angles, etc., which can be applied to various actual situations and have wider applicability.

[0099] In one embodiment, the feature transformation is performed on the license plate image input to the character detection model, and the feature size of the license plate image is unified to 64*256*3 (H*W*C). The trained character detection model may include two-stage or single-stage target detection methods such as Faster-RCNN, YOLO, and SSD. The license plate image with unified feature size passes through multiple convolution layers, normalization layers, nonlinear activation layers, residual convolution modules, etc. of the character detection model in sequence, and outputs the character category, character detection frame coordinate position, and character detection frame identification number of each license plate character in the license plate image.

[0100] The residual convolution module includes the connection of multiple residual convolutional neural networks. The normalization layer includes but is not limited to normalization operations such as instance normalization layer and adaptive instance normalization layer. The non-linear activation layer includes but is not limited to non-linear activation functions such as ReLu and LeakyReLu.

[0101] In one embodiment, the loss function includes a classification loss function and a position regression loss function. The classification loss function includes a Cross Entropy loss function, a Focal Loss loss function, a Softmax classification loss function, etc. The position regression loss function includes an L1 Loss mean absolute error loss function, an L2 Loss mean square error loss function, an IoU Loss intersection over union loss function, etc.

[0102] In one embodiment, the classification loss function of the character category of the character detection model in this application is:

[0103]

[0104] Among them, C represents the total number of character categories output by the model, and y c Indicates the real character category, z c represents the predicted character category, f(z c ) represents the probability value of the predicted character category.

[0105] The character category loss function classifies the constructed 84 types of license plate characters according to the character category. In this embodiment, the Softmax classification loss function is used for model training. c Indicates the real character category, the value can be 0 to 83. It is the basic representation of the Softmax function, which indicates the probability value of the i-th sample belonging to a category under j samples. The predicted character category output by the model is compared with the real character category through the character category loss function to adjust the character detection model and obtain a stable character detection model for subsequent recognition and detection of the license plate image to be tested.

[0106] In one embodiment, the character detection box coordinate position regression loss function of the character detection model in this application is:

[0107]

[0108] Among them, P represents the coordinate position of the character detection box predicted by the model, and G represents the actual coordinate position of the character detection box. Represents the ratio of the intersection and union of the predicted position and the true position.

[0109] In this embodiment, the character detection frame coordinate position regression loss function regresses the position of each license plate character with respect to the character detection frame coordinate position. This embodiment uses IoU Loss to regress the license plate character position. IoU Loss is based on the IoU intersection-over-union ratio between the predicted frame and the true value annotation frame. It can also be understood as the IoU intersection-over-union ratio between the predicted character detection frame coordinate position and the real character detection frame coordinate position. The predicted character detection frame coordinate position output by the model can be compared with the real character detection frame coordinate position to adjust the vehicle character detection model and obtain a stable character detection model for subsequent recognition and detection of the character detection frame coordinate position of the license plate image to be tested.

[0110] In one embodiment, in S10, character detection is performed on a plurality of license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each license plate character in each license plate image, and further includes:

[0111] S150, performing character detection on a plurality of license plate images to obtain character categories, character detection frame coordinate positions, and character detection frame identification numbers of all characters in each license plate image;

[0112] S160, judging whether the number of license plate character digits corresponding to each license plate image is greater than a preset number of license plate character digits according to the character detection frame coordinate position and the character detection frame identification number;

[0113] S170, if the number of license plate character digits is greater than the preset number of license plate character digits, sorting the confidences of all character detection frames in each license plate image from high to low to obtain a high-confidence character detection frame corresponding to the preset number of license plate character digits;

[0114] S180, obtaining the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image according to the high-confidence character detection frame corresponding to the preset license plate character digit number.

[0115] In S150, through the methods described in S110 to S140, the character categories, character detection frame coordinate positions and character detection frame identification numbers corresponding to all characters corresponding to each license plate image can be obtained.

[0116] In S160, the preset number of license plate character digits can be understood as the number of characters corresponding to each license plate. For example, the preset number of license plate character digits can be set to 8 or other values, representing the number of characters of each license plate. After obtaining the labels of all license plate characters, the number of digits of each license plate character can be known according to the coordinate position of the character detection frame and the character detection frame identification number. By comparing and judging the number of license plate character digits corresponding to each license plate image with the preset number of license plate character digits, redundant character detection frames can be screened out.

[0117] In S170, when the number of license plate character digits is greater than the preset number of license plate character digits, the character detection frames are screened by sorting the confidence of each character detection frame, and the character detection frames with low confidence are removed, and the number of character detection frames corresponding to the preset number of license plate character digits is retained.

[0118] In S180, by screening the character detection frames of each license plate, redundant detection frames can be removed to remove redundancy for the subsequent establishment of connection relationships between detection frames, further improving recognition accuracy, and finally obtaining the character detection frame corresponding to each license plate, and then obtaining the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image.

[0119] In one embodiment, the method described in the present application may also use a non-maximum suppression algorithm (NMS) to remove some redundant frames.

[0120] In one embodiment, S20, matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the character detection frame identification number, and forming a real character connection relationship matrix according to the plurality of character connection pairs, including:

[0121] S210. Arrange all character detection boxes in each license plate image according to the coordinate positions and identification numbers of the character detection boxes to obtain at least one character arrangement layer.

[0122] S220. In each character arrangement layer, perform connection relationship matching on two adjacent character detection boxes to obtain multiple character connection pairs.

[0123] S230. Form a real character connection relationship matrix according to the multiple character connection pairs.

[0124] In S210, the coordinate positions of the character detection boxes include the central point coordinate information X and Y of the character detection box and the width and height information W and H of the detection box. According to the coordinate positions of the character detection boxes and the identification numbers of the character detection boxes, all characters corresponding to each license plate can be arranged to form the corresponding license plate. The number of layers of the license plate can be one layer, two layers or more layers, forming at least one character arrangement layer.

[0125] In S220, in each character arrangement layer, there are corresponding positional relationships between the license plate characters. For example, character A is on the left of character B, or character A is on the right of character B, etc. In this embodiment, connection relationship matching is performed on the positions of two adjacent character detection boxes, and two adjacent character detection boxes are matched into a character connection pair. For example: the license plate character sequence is "Jing A12345", then the multiple character connection pairs include (Jing, A), (A, 1), (1, 2), (2, 3), (3, 4), (4, 5).

[0126] In S230, through the multiple character connection pairs in S220, each character corresponding to each license plate has a unique corresponding positional relationship with each other. Furthermore, through the learning of the positional relationship, the unique corresponding license plate character information can be recognized. A character detection box corresponds to a character detection box coordinate position and corresponds to a character detection box identification number, and the three are unique. A connection relationship matrix is formed through multiple character detection box identification numbers, which represents the connection relationship matrix between the license plate characters.

[0127] By processing and screening the original data in S110 to S180 to obtain more accurate character categories, character detection box coordinate positions, and character detection box identification numbers corresponding to each license plate character, the data can be more accurate when constructing the real character connection relationship matrix later. Through S210 to S230, based on the character detection box coordinate positions and the character detection box identification numbers, multiple character connection pairs are constructed and further form a real character connection relationship matrix, which accurately and uniquely represents the connection relationship between the license plate characters corresponding to each license plate, providing an accurate data basis for the subsequent learning of the character graph convolutional neural network.

[0128] In one embodiment, S20, matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the character detection frame identification number, and forming a real character connection relationship matrix according to the plurality of character connection pairs, further comprising:

[0129] S240, in the plurality of character arrangement layers, matching the connection relationship between the character detection frame corresponding to the right side of the first character arrangement layer and the character detection frame corresponding to the left side of the second character arrangement layer in two adjacent character arrangement layers to obtain a character connection pair.

[0130] In S240, for multiple character arrangement layers, there is a connection relationship between two adjacent character arrangement layers, and a connection relationship can be constructed for special types of license plates, such as two-layer trucks, motorcycle license plates, etc., to form corresponding character connection pairs. In this embodiment, for multiple character arrangement layers, a character connection pair is constructed with the rightmost license plate character of the first layer and the leftmost license plate character of the second layer. The first character arrangement layer can be understood as the upper layer, and the second character arrangement layer can be understood as the lower layer. By constructing a connection relationship between the characters of two adjacent character arrangement layers, the relative position relationship between the multiple layers of characters can be introduced into the character connection pair, and then introduced into the real character connection relationship matrix, providing a data support basis for the subsequent construction of graph structure data.

[0131] According to S210 to S240, the positional relationship between each license plate character in each license plate is used to construct label data that form a connection relationship, forming a real character connection relationship matrix. The real character connection relationship matrix can be represented as a matrix of N*N dimensions. Among them, N represents the number of character detection frames. The N*N dimensional feature represents the connection relationship between each character detection frame and the other N-1 character detection frames. When the character detection frame has a connection relationship with another character detection frame, the coordinate value representing the connection relationship in the real character connection relationship matrix is ​​1, otherwise it is 0. The relationship between the character detection frame and itself is represented as 0, and an N*N dimensional feature matrix that characterizes the connection relationship between license plate characters is obtained.

[0132] In one embodiment, S30, a connection relationship matrix corresponding to each character detection frame is formed according to the real character connection relationship matrix. In this embodiment, the connection relationship matrix corresponding to each character detection frame represents the connection relationship between each character detection frame and other N-1 character detection frames. According to the real character connection relationship matrix, the connection relationship matrix of the character detection frames corresponding to all license plate characters can be generated.

[0133] In one embodiment, S40, according to the coordinate position of the character detection frame, the feature transformation of the license plate image is performed to obtain the input feature vector corresponding to each license plate character, including:

[0134] S410, dividing and resizing each license plate image according to the coordinate position of the character detection frame to obtain a plurality of character detection frame images, and obtaining a 3*H*W dimensional feature of each character detection frame image according to each character detection frame image;

[0135] S420, set a 4*H*W dimensional distance feature, and perform feature splicing on the channel dimension of the 3*H*W dimensional feature and the 4*H*W dimensional distance feature of each character detection frame image to obtain multiple 7*H*W dimensional splicing features; wherein H*W represents the width and height of the detection frame, and the number of channels 7 includes an R channel, a G channel, a B channel, an X coordinate channel, a Y coordinate channel, a W width channel, and an H height channel;

[0136] S430, inputting the multiple concatenated features into multiple convolutional layers, and outputting multiple transformed features;

[0137] S440, input the multiple conversion features into a fully connected layer network, and output input feature vectors corresponding to the multiple license plate characters.

[0138] In S410, the coordinate position of the character detection frame represents the specific position of the character detection frame. According to the coordinate position of the character detection frame, the license plate image can be cropped, and the image of the corresponding character detection frame can be obtained in the original image. The image of the corresponding character detection frame obtained is resized to obtain a character detection frame image of the same size. The characteristic size of the transformed character detection frame image is C*H*W, where H*W represents the width and height of the character detection frame, and C represents the number of channels. At this time, the number of channels C is 3, which is expressed as an RGB three-channel image. Through division and size transformation, multiple images are converted into detection frame images of the same size, which is convenient for subsequent image processing.

[0139] In S420, in the 4*H*W dimensional distance feature, 4 represents the number of channels, which respectively represent the X, Y, W, and H information of the coordinate position of the character detection frame, H*W represents the width and height information, and X and Y represent the center point coordinate information of the character detection frame. In this embodiment, the width and height information in the features of each character detection frame image is the same as the width and height information in the distance feature. In multiple 7*H*W dimensional splicing features, the number of channels represents the RGB channel, the center point coordinate information X, Y of the detection frame, and the width and height information W, H of the detection frame, respectively. The features representing the distance between the character detection frames are added to the features of each character detection frame image, and the distance features are fully integrated. By setting the distance feature, the distance between the character detection frames is considered in the features of each character detection frame image. By adding the distance feature, the connection relationship between two adjacent characters that are close to each other is easier to detect, which can further enhance the learning of the connection relationship between two adjacent characters and improve the accuracy of the connection relationship between the characters of the license plate.

[0140] In the S430, the splicing features of 7*H*W dimensions are input into multiple convolutional layers to obtain image features after feature transformation. The dimension of the conversion feature is expressed as 128*H*W. Through S440, the conversion feature data is sent to the fully connected layer network for calculation to obtain the input feature vector corresponding to the license plate character with a feature dimension of 128*1*1, which is used as the input feature vector of the character graph convolutional neural network. Through the fully connected layer network, the learned splicing features are mapped to the sample label space through distributed feature representation for further calculation of the character graph convolutional neural network.

[0141] In one embodiment, S50, the connection relationship matrix corresponding to each character detection box and the input feature vector corresponding to each character are input into the character graph convolutional neural network, and the predicted character connection relationship matrix is ​​output, including:

[0142] S510, construct a character graph convolutional neural network, the character graph convolutional neural network is:

[0143]

[0144]

[0145] Among them, X represents the input feature vector corresponding to each character, A represents the connection relationship matrix corresponding to each character detection box, represents the adjacency matrix of each character detection box, I represents the unit matrix of each character detection box, Represents the degree matrix of each character detection box, W (0) With W (1) represents the parameters of the character graph convolutional neural network, ReLU represents a nonlinear activation function, and softmax represents a normalization function;

[0146] S520: output a predicted character connection relationship matrix according to the character graph convolutional neural network.

[0147] In S510, an adjacency matrix of each character detection frame is obtained according to a connection relationship matrix corresponding to each character detection frame, a unit matrix of each character detection frame, and a degree matrix of each character detection frame.

[0148] The character graph convolutional neural network Z includes two network layers, and the output feature dimension is N*N. X represents the input feature vector corresponding to each license plate character. The input feature vector corresponding to each license plate character is used as the input feature vector of the first layer of the graph convolutional neural network, which integrates the image features and distance features of the character detection box. The input feature vector corresponding to each license plate character and the adjacency matrix of each character detection box are used as the input of the first layer of the graph convolutional neural network. A represents the connection relationship matrix corresponding to each character detection box, which can be understood as the connection relationship matrix between each character detection box and other character detection boxes. If each character detection box has a connection relationship with a certain character detection box, the value displayed in the matrix is ​​1, otherwise it is 0. I represents the unit matrix of each character detection box. By adding the unit matrix, the feature representation of each character detection box itself can be increased. The adjacency matrix of each character detection box The adjacency matrix is ​​obtained by the connection relationship matrix of each character detection frame, the unit matrix of each character detection frame, and the degree matrix. The adjacency matrix represents the matrix of the adjacent relationship between each character detection frame, which is an n-order square matrix. Represents the degree matrix of each character detection box. The degree matrix represents the degree of each character detection box as a node, which can also be understood as the number of character detection boxes that each character detection box is connected to. Represents the first layer of graph convolutional neural network, and the output feature dimension is N*N. The adjacency matrix of each character detection box The feature dimension of W is N*N. (0) W represents the learnable parameters of the first layer of graph convolutional neural network, and the feature dimension is 128*N. The output of the first layer of graph convolutional neural network is used as the input of the second layer of graph convolutional neural network. (1) Represents the learnable parameters of the second layer of graph convolutional neural network, with a feature dimension of 128*N.

[0149] In S520, the connection relationship between characters is constructed as graph structure data and input into the character graph convolutional neural network for learning. By learning the connection relationship between adjacent characters, the vehicle license plate can be recognized. In addition, the input feature vector corresponding to each character combines the original vehicle image feature and the distance feature, and further training is performed based on the spliced ​​features, which further enhances the learning of the connection relationship between characters and is conducive to obtaining a more stable training model.

[0150] The character graph convolutional neural network constructed by S510 and S520 can more accurately determine the connection relationship between characters, solve the problem of low recognition accuracy of traditional methods, and improve the accuracy of license plate recognition.

[0151] In this embodiment, the ReLU nonlinear activation function and the softmax normalization function are used in the character graph convolutional neural network. The ReLU function has stronger expressive power and does not have the problem of gradient vanishing, so that the convergence speed of the model is maintained at a stable state. The model after sparseness achieved by the ReLU function can better mine the connection relationship characteristics between characters and fit the training data. The softmax normalization function can reduce the difficulty of model training, making it easier to converge multi-classification problems, and better suitable for the recognition of the connection relationship between characters and characters by the method of this application, and further more accurately realize the recognition of license plates.

[0152] In one embodiment, S60, constructing a loss function according to the predicted character connection relationship matrix and the real character connection relationship matrix, and adjusting the parameters of the character graph convolutional neural network according to the loss function to obtain a trained character graph convolutional neural network, includes:

[0153] S610, construct a loss function, the loss function is:

[0154]

[0155] in, represents the predicted character connection relationship matrix corresponding to the i-th training sample, y (i) Represents the true character connection relationship matrix corresponding to the i-th training sample;

[0156] S620, adjusting the parameters of the character graph convolutional neural network to minimize the loss function, and obtaining a trained character graph convolutional neural network.

[0157] In S610, the i-th training sample can also be understood as the training sample data corresponding to the i-th character detection box. The model training is performed using the L2 norm loss function. The L2 norm loss function is also called the lowest square error loss function. In the formula, m represents the number of samples in the training sample set.

[0158] In S620, the objective of the loss function is to minimize the sum of squares. By adjusting the parameters of the character graph convolutional neural network, the loss function is minimized, and the optimal network parameters are finally obtained. According to the optimal network parameters, a trained character graph convolutional neural network can be obtained. According to the character graph convolutional neural network, the connection relationship between the characters in the license plate image to be tested can be identified, and then the license plate can be accurately identified to improve the recognition accuracy.

[0159] In one embodiment, S70, predicting the license plate image to be tested according to the trained character graph convolutional neural network, obtaining a character connection relationship matrix, and obtaining a license plate corresponding to the license plate image to be tested according to the character connection relationship matrix and the character category of each license plate character, including:

[0160] Obtain the license plate image to be tested, use the trained character detection model to perform character detection on the license plate image to be tested, and obtain the character category, character detection frame coordinate position and character detection frame identification number of each corresponding character;

[0161] According to the character category, the coordinate position of the character detection frame and the identification number of the character detection frame of each character, the connection relationship matrix corresponding to each character detection frame is retrieved from the real character connection relationship matrix;

[0162] According to the coordinate position of the character detection frame, the license plate image to be tested is subjected to feature extraction transformation to obtain the input feature vector corresponding to each character;

[0163] The connection relationship matrix corresponding to each character detection box and the input feature vector corresponding to each license plate character are input into the trained character graph convolutional neural network, and the character connection relationship matrix is ​​output;

[0164] According to the character connection relationship matrix and the character category of each character, the complete license plate corresponding to the license plate image to be tested is obtained.

[0165] See also Figure 2 The present application provides a license plate character recognition device 100. The license plate character recognition device 100 includes a data acquisition module 10, a real character connection relationship matrix acquisition module 20, a character detection frame connection relationship matrix acquisition module 30, an input feature vector acquisition module 40, a model construction module 50, a training module 60 and a prediction module 70. The data acquisition module 10 is used to acquire multiple license plate images, and perform character detection on the multiple license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each of the license plate images.

[0166] The real character connection relationship matrix acquisition module 20 is used to match multiple character connection pairs with a connection relationship according to the coordinate position of the character detection frame and the character detection frame identification number, and form a real character connection relationship matrix according to the multiple character connection pairs. The character detection frame connection relationship matrix acquisition module 30 is used to form a connection relationship matrix for each character detection frame according to the real character connection relationship matrix. The input feature vector acquisition module 40 is used to perform feature extraction transformation on the license plate image according to the coordinate position of the character detection frame to obtain the input feature vector of each character. The model construction module 50 is used to input the connection relationship matrix of each character detection frame and the input feature vector of each character into the character graph convolutional neural network, and output the predicted character connection relationship matrix.

[0167] The training module 60 is used to construct a loss function based on the predicted character connection relationship matrix and the real character connection relationship matrix, and adjust the parameters of the character graph convolutional neural network based on the loss function to obtain a trained character graph convolutional neural network. The prediction module 70 is used to predict the license plate image to be tested based on the trained character graph convolutional neural network to obtain a character connection relationship matrix, and obtain the license plate corresponding to the license plate image to be tested based on the character connection relationship matrix and the character category of each character.

[0168] In this embodiment, the relevant description of the data acquisition module 10 may refer to the relevant description of S10 in the above embodiment. The relevant description of the real character connection relationship matrix acquisition module 20 may refer to the relevant description of S20 in the above embodiment. The relevant description of the character detection box connection relationship matrix acquisition module 30 may refer to the relevant description of S30 in the above embodiment. The relevant description of the input feature vector acquisition module 40 may refer to the relevant description of S40 in the above embodiment. The relevant description of the model construction module 50 may refer to the relevant description of S50 in the above embodiment. The relevant description of the training module 60 may refer to the relevant description of S60 in the above embodiment. The relevant description of the prediction module 70 may refer to the relevant description of S70 in the above embodiment.

[0169] In one embodiment, the real character connection relationship matrix acquisition module 20 includes a character arrangement layer acquisition module (not marked in the figure), a first character connection pair acquisition module (not marked in the figure) and a real character connection relationship matrix formation module (not marked in the figure). The character arrangement layer acquisition module is used to arrange all the character detection frames in each of the license plate images according to the coordinate positions of the character detection frames and the identification numbers of the character detection frames to obtain at least one character arrangement layer. The first character connection pair acquisition module is used to match the connection relationship of two adjacent character detection frames in each of the character arrangement layers to obtain the multiple character connection pairs. The real character connection relationship matrix formation module is used to form the real character connection relationship matrix according to the multiple character connection pairs.

[0170] In this embodiment, the relevant description of the character arrangement layer acquisition module can refer to the relevant description of S210 in the above embodiment. The relevant description of the first character connection pair acquisition module can refer to the relevant description of S220 in the above embodiment. The relevant description of the real character connection relationship matrix formation module can refer to the relevant description of S230 in the above embodiment.

[0171] In one embodiment, the real character connection relationship matrix acquisition module 20 further includes a second character connection pair acquisition module (not shown in the figure). The second character connection pair acquisition module is used to match the connection relationship between the character detection frame corresponding to the right side of the first character arrangement layer and the character detection frame corresponding to the left side of the second character arrangement layer in two adjacent character arrangement layers in the plurality of character arrangement layers, so as to obtain the character connection pair.

[0172] In this embodiment, the relevant description of the second character connection pair obtaining module may refer to the relevant description of S240 in the above embodiment.

[0173] In one embodiment, the model building module 50 includes a character graph convolutional neural network building module (not shown in the figure) and a predicted character connection relationship matrix acquisition module (not shown in the figure). The character graph convolutional neural network building module is used to build the character graph convolutional neural network, and the character graph convolutional neural network is:

[0174]

[0175]

[0176] Wherein, X represents the input feature vector of each character, A represents the connection relationship matrix of each character detection frame, represents the adjacency matrix of each of the character detection frames, I represents the unit matrix of each of the character detection frames, Represents the degree matrix of each character detection box, W (0)With W (1) Represents the parameters of the character graph convolutional neural network, ReLU represents a nonlinear activation function, and softmax represents a normalization function.

[0177] The predicted character connection relationship matrix acquisition module is used to output the predicted character connection relationship matrix according to the character graph convolutional neural network.

[0178] In this embodiment, the description of the character graph convolutional neural network construction module can refer to the description of S510 in the above embodiment. The description of the predicted character connection relationship matrix acquisition module can refer to the description of S520 in the above embodiment.

[0179] In one embodiment, the data acquisition module 10 includes an all-character data acquisition module (not marked in the figure), a judgment module (not marked in the figure), a high-confidence data acquisition module (not marked in the figure), and a first character information acquisition module (not marked in the figure). The all-character data acquisition module is used to perform character detection on the multiple license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of all characters in each of the license plate images. The judgment module is used to determine whether the number of license plate character digits corresponding to each of the license plate images is greater than the preset number of license plate character digits based on the character detection frame coordinate position and the character detection frame identification number.

[0180] The high confidence data acquisition module is used to sort the confidence of all character detection frames in each license plate image from high to low if the number of character digits of the license plate is greater than the preset number of character digits of the license plate, and obtain the high confidence character detection frame corresponding to the preset number of character digits of the license plate. The first character information acquisition module is used to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image based on the high confidence character detection frame corresponding to the preset number of character digits of the license plate.

[0181] In this embodiment, the relevant description of all character data acquisition modules can refer to the relevant description of S150 in the above embodiment. The relevant description of the judgment module can refer to the relevant description of S160 in the above embodiment. The relevant description of the high confidence data acquisition module can refer to the relevant description of S170 in the above embodiment. The relevant description of the first character information acquisition module can refer to the relevant description of S180 in the above embodiment.

[0182] In one embodiment, the data acquisition module 10 further includes a labeling module (not shown in the figure), a data enhancement module (not shown in the figure), a character detection model forming module (not shown in the figure), and a second character information acquisition module (not shown in the figure). The labeling module is used to label the multiple license plate images to obtain multiple single-character labeled license plate images. The data enhancement module is used to perform data enhancement processing on the multiple single-character labeled license plate images to obtain multiple data enhanced license plate images.

[0183] The character detection model forming module is used to train the character detection model according to the multiple data-enhanced license plate images to obtain a trained character detection model. The second character information acquisition module is used to perform character detection on the license plate image to be tested according to the trained character detection model to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in the license plate image.

[0184] In this embodiment, the relevant description of the annotation module can refer to the relevant description of S110 in the above embodiment. The relevant description of the data enhancement module can refer to the relevant description of S120 in the above embodiment. The relevant description of the character detection model forming module can refer to the relevant description of S130 in the above embodiment. The relevant description of the second character information acquisition module can refer to the relevant description of S140 in the above embodiment.

[0185] In the above-mentioned various embodiments, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims provide the elements of various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0186] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0187] The various illustrative logic blocks or units described in the embodiments of the present application can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or the design of any combination of the above functions. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration to implement.

[0188] The steps of the method or algorithm described in the embodiments of the present application can be directly embedded in the software module executed by the hardware, the processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0189] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A license plate character recognition method, characterized in that: include: Acquire multiple license plate images, and perform character detection on the multiple license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each of the license plate images; Matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs; According to the real character connection relationship matrix, forming a connection relationship matrix for each character detection frame; According to the coordinate position of the character detection frame, the license plate image is subjected to feature extraction transformation to obtain an input feature vector of each character; Input the connection relationship matrix of each character detection frame and the input feature vector of each character into the character graph convolutional neural network, and output the predicted character connection relationship matrix; Constructing a loss function according to the predicted character connection relationship matrix and the real character connection relationship matrix, and adjusting the parameters of the character graph convolutional neural network according to the loss function to obtain a trained character graph convolutional neural network; According to the trained character graph convolutional neural network, predict the license plate image to be tested, obtain a character connection relationship matrix, and obtain the license plate corresponding to the license plate image to be tested according to the character connection relationship matrix and the character category of each character; The matching of a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs, comprises: Arranging all the character detection frames in each of the license plate images according to the coordinate positions of the character detection frames and the identification numbers of the character detection frames to obtain at least one character arrangement layer; In each of the character arrangement layers, matching the connection relationship between two adjacent character detection frames to obtain the plurality of character connection pairs; According to the plurality of character connection pairs, forming the real character connection relationship matrix; The method of matching a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and forming a real character connection relationship matrix according to the plurality of character connection pairs, further comprises: In the plurality of character arrangement layers, a character detection frame corresponding to the right side of a first character arrangement layer and a character detection frame corresponding to the left side of a second character arrangement layer in two adjacent character arrangement layers are matched for connection relationship to obtain the character connection pair.

2. The license plate character recognition method according to claim 1, characterized in that: The step of inputting the connection relationship matrix of each character detection frame and the input feature vector of each character into a character graph convolutional neural network and outputting a predicted character connection relationship matrix comprises: Construct the character graph convolutional neural network, which is: Wherein, X represents the input feature vector of each character, A represents the connection relationship matrix of each character detection frame, represents the adjacency matrix of each of the character detection frames, I represents the unit matrix of each of the character detection frames, Represents the degree matrix of each character detection box, W (0) With W (1) Represents the parameters of the character graph convolutional neural network, ReLU represents a nonlinear activation function, and softmax represents a normalization function; according to the character graph convolutional neural network, the predicted character connection relationship matrix is ​​output.

3. The license plate character recognition method according to claim 1, characterized in that: The performing character detection on the plurality of license plate images to obtain the character category, the character detection frame coordinate position and the character detection frame identification number of each character in each of the license plate images comprises: Performing character detection on the plurality of license plate images to obtain character categories, character detection frame coordinate positions, and character detection frame identification numbers of all characters in each of the license plate images; According to the coordinate position of the character detection frame and the identification number of the character detection frame, determining whether the number of license plate character digits corresponding to each license plate image is greater than a preset number of license plate character digits; If the number of license plate character digits is greater than the preset number of license plate character digits, the confidence levels of all character detection frames in each license plate image are sorted from high to low to obtain a high-confidence character detection frame corresponding to the preset number of license plate character digits; According to the high-confidence character detection frame corresponding to the preset number of license plate character digits, the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image are obtained.

4. The license plate character recognition method according to claim 3, characterized in that: The performing of character detection on the plurality of license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image also includes: Annotating the multiple license plate images to obtain multiple single-character annotated license plate images; Performing data enhancement processing on the multiple single-character labeled license plate images to obtain multiple data enhanced license plate images; Training a character detection model according to the multiple data-enhanced license plate images to obtain a trained character detection model; According to the trained character detection model, character detection is performed on the license plate image to be tested to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in the license plate image.

5. A license plate character recognition device, characterized in that: include: A data acquisition module, used to acquire multiple license plate images, and perform character detection on the multiple license plate images to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each of the license plate images; A real character connection relationship matrix acquisition module is used to match a plurality of character connection pairs having a connection relationship according to the coordinate position of the character detection frame and the identification number of the character detection frame, and form a real character connection relationship matrix according to the plurality of character connection pairs; A character detection frame connection relationship matrix acquisition module is used to form a connection relationship matrix for each character detection frame according to the real character connection relationship matrix; An input feature vector acquisition module is used to perform feature extraction transformation on the license plate image according to the coordinate position of the character detection frame to obtain an input feature vector of each character; A model building module, used to input the connection relationship matrix of each character detection box and the input feature vector of each character into a character graph convolutional neural network, and output a predicted character connection relationship matrix; A training module, used to construct a loss function according to the predicted character connection relationship matrix and the real character connection relationship matrix, and adjust the parameters of the character graph convolutional neural network according to the loss function to obtain a trained character graph convolutional neural network; A prediction module is used to predict the license plate image to be tested according to the trained character graph convolutional neural network, obtain a character connection relationship matrix, and obtain a license plate corresponding to the license plate image to be tested according to the character connection relationship matrix and the character category of each character; The real character connection relationship matrix acquisition module includes: A character arrangement layer acquisition module, used to arrange all the character detection frames in each of the license plate images according to the coordinate positions of the character detection frames and the identification numbers of the character detection frames to obtain at least one character arrangement layer; A first character connection pair acquisition module, configured to match the connection relationship between two adjacent character detection frames in each character arrangement layer to obtain the plurality of character connection pairs; A real character connection relationship matrix forming module, used for forming the real character connection relationship matrix according to the plurality of character connection pairs; The real character connection relationship matrix acquisition module also includes: The second character connection pair acquisition module is used to match the connection relationship between the character detection frame corresponding to the right side of the first character arrangement layer and the character detection frame corresponding to the left side of the second character arrangement layer in two adjacent character arrangement layers in the multiple character arrangement layers to obtain the character connection pair.

6. The license plate character recognition device according to claim 5, characterized in that: The model building module includes: The character graph convolutional neural network construction module is used to construct the character graph convolutional neural network, and the character graph convolutional neural network is: Wherein, X represents the input feature vector of each character, A represents the connection relationship matrix of each character detection frame, represents the adjacency matrix of each of the character detection frames, I represents the unit matrix of each of the character detection frames, Represents the degree matrix of each character detection box, W (0) With W (1) Represents the parameters of the character graph convolutional neural network, ReLU represents a nonlinear activation function, and softmax represents a normalization function; a predicted character connection relationship matrix acquisition module is used to output the predicted character connection relationship matrix according to the character graph convolutional neural network.

7. The license plate character recognition device according to claim 5, characterized in that: The data acquisition module comprises: All character data acquisition module, used for performing character detection on the plurality of license plate images, obtaining character categories, character detection frame coordinate positions and character detection frame identification numbers of all characters in each of the license plate images; A judgment module, used for judging whether the number of license plate character digits corresponding to each license plate image is greater than a preset number of license plate character digits according to the coordinate position of the character detection frame and the identification number of the character detection frame; A high confidence data acquisition module is used to sort the confidences of all the character detection frames in each license plate image from high to low if the number of character digits of the license plate is greater than the preset number of character digits of the license plate, and obtain a high confidence character detection frame corresponding to the preset number of character digits of the license plate; The first character information acquisition module is used to obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in each license plate image according to the high-confidence character detection frame corresponding to the preset license plate character digit number.

8. The license plate character recognition device according to claim 6, characterized in that: The data acquisition module also includes: A labeling module, used to label the multiple license plate images to obtain multiple single-character labeled license plate images; A data enhancement module, used to perform data enhancement processing on the multiple single-character annotated license plate images to obtain multiple data enhanced license plate images; A character detection model forming module, used to train the character detection model according to the plurality of data-enhanced license plate images to obtain a trained character detection model; The second character information acquisition module is used to perform character detection on the license plate image to be tested according to the trained character detection model, and obtain the character category, character detection frame coordinate position and character detection frame identification number of each character in the license plate image.

Citation Information

Patent Citations

  • Method and device for inputting mathematical formula

    CN101859186A

  • License plate recognition method, device and system

    CN106407981A

  • Character detection method and apparatus

    CN106845530A

  • License plate recognition model establishing method and device

    CN106874907A

  • Multi-microgrid coordination control method for electric vehicle

    CN111985811A