A method, device, computer device and storage medium for single and double license plate recognition

Abstract

The present invention discloses a method, device, computer device and storage medium for single and double license plate recognition, which relates to the technical field of license plate recognition. After obtaining a sample training set, the method inputs each image sample and the corresponding license plate number sequence information in the sample training set into a preset single and double license plate recognition model capable of performing license plate number sequence modeling and license plate character positioning modeling for model training. Then, the single license plate image or double license plate image to be recognized is input into the trained single and double license plate recognition model, so as to output the predicted probability values of the corresponding license plate characters at each character position in the license plate number sequence through the license plate number sequence modeling unit. Finally, the license plate number sequence is obtained according to the output data. In this way, it is not necessary to classify the input license plate image to be recognized as a single or double license plate, and thus the license plate can be directly recognized and the recognition result can be output, greatly simplifying the recognition process and reducing the computational resource requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of license plate recognition, and particularly relates to a method, device, computer device and storage medium for recognizing single and double-line license plates. Background Art

[0002] License plate recognition has a wide range of applications in scenarios such as highway toll booths, parking lot entrances and exits, vehicle tracking, and traffic supervision. Traditional license plate recognition solutions can either only recognize single-line license plates or double-line license plates separately, or they need to first classify the license plate images. If it is a double-line license plate image, further segmentation and recombination processing are required, or the position of the upper and lower layer segmentation lines needs to be determined on the double-line license plate image, and then license plate recognition is carried out. As a result, the recognition process is relatively complex and requires more computing resources. Summary of the Invention

[0003] The purpose of the present invention is to provide a method, device, computer device and computer-readable storage medium for recognizing single and double-line license plates, so as to solve the problems of complex process and large computing resources required in the existing license plate recognition solutions when recognizing double-line license plates.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] In the first aspect, a method for recognizing single and double-line license plates is provided, including:

[0006] Obtain a sample training set, where multiple image samples in the sample training set include single-line license plate images and double-line license plate images, and each image sample in the multiple image samples is labeled with corresponding license plate number sequence information. The license plate number sequence information includes multiple character coordinate data arranged in sequence according to the license plate number sequence. The character coordinate data includes a digital unique identifier corresponding to the character and the normalized abscissa, ordinate, width, and height of the corresponding character on the license plate image where it is located;

[0007] Input each image sample and the corresponding license plate number sequence information into a preset single and double-line license plate recognition model for training the single and double-line license plate recognition model. The single and double-line license plate recognition model includes an image feature extraction unit, a spatial pyramid pooling unit, a license plate number sequence modeling unit, a license plate character positioning modeling unit, and a loss function calculation unit;

[0008] The image feature extraction unit is used to extract an image feature matrix with dimensions of K×M×N from the image sample and send the image feature matrix to the spatial pyramid pooling unit, where K represents the first least common multiple of 2, 3, and 6, M represents the second least common multiple of 2, 3, and 6 and is greater than the first least common multiple, and N represents a positive integer greater than the second least common multiple;

[0009] The spatial pyramid pooling unit is used to perform maximum pooling processing on the image feature matrix with sizes of 2×2, 3×3, and 6×6 respectively, and obtain three feature matrices with dimensions of and Then, the three feature matrices are processed in the following ways (A) and (B):

[0010] (A) Change the first two dimensions of each of the three feature matrices to one dimension, and obtain three first new feature matrices with dimensions of and Then, splice the three first new feature matrices to obtain a second new feature matrix with dimensions of Then, perform average pooling processing on the second new feature matrix with to obtain a third new feature matrix with dimensions of Len×N. Finally, input the third new feature matrix into the license plate number sequence modeling unit, where Len represents the length of the license plate number sequence;

[0011] (B) Perform padding processing of 1×2 and 2×4 on the two feature matrices among the three feature matrices with dimensions of and respectively, to obtain two fourth new feature matrices with dimensions of Then, splice the two fourth new feature matrices and the feature matrix among the three feature matrices with dimensions of to obtain a fifth new feature matrix with dimensions of [[ID=3l]]Then, perform convolution processing of 1×1×(3×(4 + 1)) on the fifth new feature matrix to obtain a sixth new feature matrix with dimensions of Finally, input the sixth new feature matrix into the license plate character positioning modeling unit;

[0012] The license plate number sequence modeling unit is used to model the license plate number sequence according to the third new feature matrix, and output the predicted values of the occurrence probabilities of each license plate character at each character position in the license plate number sequence;

[0013] The license plate number character positioning and modeling unit is used to model the positions of license plate number characters according to the sixth new feature matrix, and output the predicted values of the abscissa, ordinate, width, height and confidence of each character grid prediction box;

[0014] The loss function calculation unit is used to calculate the loss function value according to the output data of the license plate number sequence modeling unit and the license plate number character positioning and modeling unit;

[0015] Input the single-line license plate image or double-line license plate image to be recognized into the trained single / double-line license plate recognition model, so as to output the corresponding output data through the license plate number sequence modeling unit;

[0016] For the single-line license plate image or double-line license plate image to be recognized, determine the corresponding license plate number characters at each character position according to the corresponding output data, and obtain the corresponding license plate number sequence.

[0017] Based on the above invention content, a new solution for single / double-line license plate recognition that only needs to be viewed once is provided. That is, after obtaining the sample training set, input each image sample and the corresponding license plate number sequence information in the sample training set into a preset single / double-line license plate recognition model that can perform license plate number sequence modeling and license plate number character positioning modeling for model training. Then, input the single-line license plate image or double-line license plate image to be recognized into the trained single / double-line license plate recognition model, so as to output the predicted values of the appearance probabilities of the corresponding license plate number characters at each character position in the license plate number sequence through the license plate number sequence modeling unit. Finally, obtain the license plate number sequence according to the output data. In this way, it is not necessary to classify the single-line or double-line license plate for the input license plate image to be recognized, and thus the license plate can be directly recognized and the recognition result can be output, greatly simplifying the recognition process and reducing the computational resource requirements.

[0018] In a possible design, the image feature extraction unit includes a first convolutional layer, a second convolutional layer, a first convolutional residual sub-unit, a second convolutional residual sub-unit, a third convolutional residual sub-unit, a fourth convolutional residual sub-unit, an eleventh convolutional layer, a cross-stage local network CSPNet layer, and a fifth convolutional residual sub-unit arranged in sequence along the input-to-output direction. Among them, the first convolutional residual sub-unit, the second convolutional residual sub-unit, the third convolutional residual sub-unit, the fourth convolutional residual sub-unit, and the fifth convolutional residual sub-unit each include two convolutional layers and a residual block arranged in sequence along the input-to-output direction;

[0019] The cross-stage partial network CSPNet layer is used to divide the input into two parts: one part is forwarded into the internal dense block, and the other part is sent to the spatial pyramid pooling unit as an image feature matrix.

[0020] In a possible design, the license plate number sequence modeling unit uses a Transformer language model to model the license plate number sequence;

[0021] The input data of the Transformer language model consists of two parts: one part is the third new feature matrix, and the other part is a feature vector with a dimension of Len×128 obtained by converting the license plate number through word embedding. The third new feature matrix and the feature vector are added to form the input data for inputting into the Transformer language model.

[0022] In a possible design, the word embedding method uses a word embedding method based on the Word2Vec model.

[0023] In a possible design, the last fully connected layer of the Transformer language model uses the Softmax function as the activation function.

[0024] In a possible design, according to the output data of the license plate number sequence modeling unit and the license plate character positioning modeling unit, the loss function value is calculated, including:

[0025] According to the output data of the license plate number sequence modeling unit and the license plate character positioning modeling unit, the loss function value Loss is calculated according to the following formula:

[0026]

[0027] In the formula, L1 represents the first loss value of the bounding box coordinates, L2 represents the second loss value of the bounding box coordinates, L3 represents the loss value of the bounding box confidence, CTC loss represents the connectionist temporal classification CTC loss value, λ coord represents the first hyperparameter, λ noobj represents the second hyperparameter, i and j are natural numbers respectively, x i represents the predicted value of the abscissa of the prediction box of the i-th character grid, represents the true value of the abscissa of the prediction box of the i-th character grid, y i represents the predicted value of the ordinate of the prediction box of the i-th character grid, represents the true value of the ordinate of the prediction box of the i-th character grid, w i represents the predicted value of the width of the prediction box of the i-th character grid, Denote the true value of the width of the i-th character grid prediction box, h i Denote the predicted value of the height of the i-th character grid prediction box, Denote the true value of the height of the i-th character grid prediction box, C i Denote the predicted value of the confidence of the i-th character grid prediction box, Denote the true value of the confidence of the i-th character grid prediction box, Denote whether there is a target character in the i-th character grid prediction box: if there is, the value is 1, Denote whether there is no target character in the i-th character grid prediction box: if there is no, the value is 1, t represents the output data of the license plate number sequence modeling unit, z represents the correct license plate number sequence, R represents the sample training set, and p(z / t) represents the probability of outputting sequence z given the input t.

[0028] In a possible design, the first hyperparameter takes the value of 5, and the second hyperparameter takes the value of 0.5.

[0029] In a second aspect, a single and double line license plate recognition device is provided, including a training set acquisition module, an identification model training module, an identification model application module, and a license plate number determination module that are sequentially communicatively connected;

[0030] The training set acquisition module is used to acquire a sample training set. Among them, multiple image samples in the sample training set include single-line license plate images and double-line license plate images. Each image sample in the multiple image samples is labeled with corresponding license plate number sequence information. The license plate number sequence information includes multiple character coordinate data arranged in sequence according to the license plate number sequence. The character coordinate data includes the digital unique identifier of the corresponding character and the normalized abscissa, ordinate, width, and height of the corresponding character on the license plate image where it is located;

[0031] The identification model training module is used to input each image sample and the corresponding license plate number sequence information into a preset single and double line license plate recognition model to train the single and double line license plate recognition model. Among them, the single and double line license plate recognition model includes an image feature extraction unit, a spatial pyramid pooling unit, a license plate number sequence modeling unit, a license plate character positioning modeling unit, and a loss function calculation unit;

[0032] The image feature extraction unit is used to extract an image feature matrix with dimensions of K×M×N from the image sample and send the image feature matrix to the spatial pyramid pooling unit. Among them, K represents the first least common multiple of 2, 3, and 6, M represents the second least common multiple of 2, 3, and 6 and is greater than the first least common multiple, and N represents a positive integer greater than the second least common multiple;

[0033] The spatial pyramid pooling unit is used to perform maximum pooling operations of 2×2, 3×3, and 6×6 on the image feature matrix respectively, obtaining three feature matrices with dimensions of and Then, the following processing methods (A) and (B) are performed on the three feature matrices:

[0034] (A) Change the first two dimensions of each of the three feature matrices into one dimension, obtaining three first new feature matrices with dimensions of and Then, splice the three first new feature matrices, obtaining a second new feature matrix with a dimension of Then, perform average pooling operation of on the second new feature matrix, obtaining a third new feature matrix with a dimension of Len×N. Finally, input the third new feature matrix into the license plate number sequence modeling unit, where Len represents the length of the license plate number sequence;

[0035] (B) Perform padding operations of 1×2 and 2×4 on two of the three feature matrices with dimensions of and respectively, obtaining two fourth new feature matrices with dimensions of Then, splice the two fourth new feature matrices and the feature matrix with a dimension of among the three feature matrices, obtaining a fifth new feature matrix with a dimension of Then, perform a convolution operation of 1×1×(3×(4 + 1)) on the fifth new feature matrix, obtaining a sixth new feature matrix with a dimension of Finally, input the sixth new feature matrix into the license plate number character positioning modeling unit;

[0036] The license plate number sequence modeling unit is used to model the license plate number sequence according to the third new feature matrix, and output the predicted values of the occurrence probabilities of each license plate number character at each character position in the license plate number sequence;

[0037] The license plate number character positioning modeling unit is used to model the license plate number character positions according to the sixth new feature matrix, and output the predicted values of the abscissa, ordinate, width, height, and confidence of each character grid prediction box;

[0038] The loss function calculation unit is used to calculate the loss function value according to the output data of the license plate number sequence modeling unit and the license plate number character positioning modeling unit;

[0039] The recognition model application module is configured to input a single-line license plate image or a double-line license plate image to be recognized into the trained single / double-line license plate recognition model, so as to output corresponding output data through the license plate number sequence modeling unit;

[0040] The license plate number determination module is configured to, for the single-line license plate image or the double-line license plate image to be recognized, determine the license plate characters corresponding thereto at each character position according to the corresponding output data, so as to obtain a corresponding license plate number sequence.

[0041] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a transceiver that are communicatively connected in sequence. Among them, the memory is configured to store a computer program, the transceiver is configured to send and receive messages, and the processor is configured to read the computer program and execute the single / double-line license plate recognition method as described in the first aspect or any possible design in the first aspect.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium, on which instructions are stored. When the instructions run on a computer, the single / double-line license plate recognition method as described in the first aspect or any possible design in the first aspect is executed.

[0043] In a fifth aspect, the present invention provides a computer program product containing instructions. When the instructions run on a computer, the computer is made to execute the single / double-line license plate recognition method as described in the first aspect or any possible design in the first aspect.

[0044] Beneficial effects:

[0045] (1) The present invention provides a new solution for single / double-line license plate recognition that only requires one look. That is, after obtaining a sample training set, each image sample and the corresponding license plate number sequence information in the sample training set are input into a preset single / double-line license plate recognition model that can perform license plate number sequence modeling and license plate character positioning modeling for model training. Then, a single-line license plate image or a double-line license plate image to be recognized is input into the trained single / double-line license plate recognition model, so as to output the predicted probability values of the occurrences of the corresponding license plate characters at each character position in the license plate number sequence through the license plate number sequence modeling unit. Finally, the license plate number sequence is obtained according to the output data. In this way, it is not necessary to classify the input license plate image to be recognized into single-line or double-line license plates, and thus license plate recognition can be directly performed and the recognition result can be output, greatly simplifying the recognition process and reducing the computational resource requirements;

[0046] (2) Since the license plate image features are extracted through the backbone network, and during the training process, the license plate number recognition and character localization modeling are respectively based on these license plate image features, and only a convolution operation is involved in the character localization modeling process, which does not consume too much computing resources and is convenient for practical application and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is a schematic flowchart of the single and double license plate recognition method provided in this embodiment.

[0049] Figure 2 It is a schematic structural diagram of the single and double license plate recognition model provided in this embodiment.

[0050] Figure 3 It is a schematic layer structure diagram of the image feature extraction unit and the spatial pyramid pooling unit in the single and double license plate recognition model provided in this embodiment.

[0051] Figure 4 It is a schematic structural diagram of the single and double license plate recognition device provided in this embodiment.

[0052] Figure 5 It is a schematic structural diagram of the computer device provided in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the present invention in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the drawing structure is only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. It should be noted here that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation to the present invention.

[0054] It should be understood that although terms such as first and second may be used herein to describe various objects, these objects should not be limited by these terms. These terms are only used to distinguish one object from another. For example, the first object can be called the second object, and similarly, the second object can be called the first object, without departing from the scope of the exemplary embodiments of the present invention.

[0055] It should be understood that for the term "and / or" that may appear in this article, it is merely a relational description of associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, B exists alone, or both A and B exist simultaneously. Another example, A, B, and / or C can represent any one of A, B, and C or any combination of them. For the term " / and" that may appear in this article, it is a description of another associated object relationship, indicating that there can be two relationships. For example, A / and B can represent two situations: A exists alone or both A and B exist simultaneously. Additionally, for the character " / " that may appear in this article, it generally indicates that the associated objects before and after are in an "or" relationship.

[0056] As Figure 1 shown, the single / double license plate recognition method provided in the first aspect of this embodiment can be, but is not limited to, executed by a computer device with certain computing resources, such as a platform server, a personal computer (Personal Computer, PC, referring to a multi-purpose computer suitable for personal use in terms of size, price, and performance; desktop computers, laptops, small laptops, tablet computers, and ultrabooks all belong to personal computers), a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), or a wearable device, etc. As Figure 1 shown, the single / double license plate recognition method can be, but is not limited to, including the following steps S1 to S4.

[0057] S1. Obtain a sample training set. Among them, the multiple image samples in the sample training set include single license plate images and double license plate images. Each image sample in the multiple image samples is labeled with corresponding license plate number sequence information. The license plate number sequence information includes multiple character coordinate data arranged in sequence according to the license plate number sequence. The character coordinate data includes the digitalized unique identifier of the corresponding character and the normalized abscissa, ordinate, width, and height of the corresponding character on the license plate image where it is located.

[0058] In step S1, the number of single license plate images and the number of double license plate images in the sample training set are preferably not very different. An example of the multiple character coordinate data is shown in Table 1 below:

[0059] Table 1. Example of multiple character coordinate data

[0060]

[0061] As shown in Table 1, it is the seven-character coordinate data of a single-lane license plate image (which corresponds one-to-one with the seven character positions), and the corresponding license plate number sequence is "Wan A38519". In addition, since the license plate number sequence is generally composed of 24 letters, 10 digits and 42 Chinese characters (including provincial and special license plate Chinese characters, such as "gua" and "jing", etc.), so there are 76 license plate characters required.

[0062] S2. Input the respective image samples and the corresponding license plate number sequence information into a preset single / double-lane license plate recognition model to train the single / double-lane license plate recognition model. Among them, the single / double-lane license plate recognition model includes, but is not limited to, an image feature extraction unit, a spatial pyramid pooling unit, a license plate number sequence modeling unit, a license plate character positioning modeling unit, a loss function calculation unit, etc.

[0063] In the step S2, the image feature extraction unit is used to extract an image feature matrix with dimensions of K×M×N from the image sample and send the image feature matrix to the spatial pyramid pooling unit, where K represents the first least common multiple of 2, 3 and 6, M represents the second least common multiple of 2, 3 and 6 and is greater than the first least common multiple, and N represents a positive integer greater than the second least common multiple. Preferably, as Figure 3As shown, the image feature extraction unit includes, but is not limited to, a first convolutional layer, a second convolutional layer, a first convolutional residual sub-unit, a second convolutional residual sub-unit, a third convolutional residual sub-unit, a fourth convolutional residual sub-unit, an eleventh convolutional layer, a cross-stage partial network CSPNet layer, and a fifth convolutional residual sub-unit arranged in sequence along the input-to-output direction. Among them, the first convolutional residual sub-unit, the second convolutional residual sub-unit, the third convolutional residual sub-unit, the fourth convolutional residual sub-unit, and the fifth convolutional residual sub-unit each include, but are not limited to, two convolutional layers and a residual block arranged in sequence along the input-to-output direction; the cross-stage partial network CSPNet layer is used to divide the input into two parts: one part is forwarded into the internal dense block, and the other part is sent as an image feature matrix to the spatial pyramid pooling unit. That is, considering the simplicity of the license plate scenario, if the structure is too complex, overfitting will occur. Therefore, based on the reference of the CSPDarknet53 network (i.e., the backbone network CSPDarknet53 of YOLOv4), only 13 convolutional layers are used in the first half of the image feature extraction unit, which not only omits most of the convolutional layers in the original CSPDarknet53 network but also saves a large amount of computing resources. At the same time, by introducing the cross-stage partial network CSPNet (Cross Stage Partial DenseNet) layer after the eleventh convolutional layer, features can be retained through propagation, encouraging the network to reuse features, reducing the number of network parameters, helping to preserve fine-grained features, and thus enabling more efficient forward entry into deeper layers. In addition, as Figure 3 shown, the value of the first least common multiple K is exemplified as 12, the value of the second least common multiple M is exemplified as 24, and N is exemplified as 128.

[0064] In the step S2, the spatial pyramid pooling (SPP) unit is used to perform max-pooling operations on the image feature matrix with sizes of 2×2, 3×3, and 6×6 respectively, to obtain three feature matrices with dimensions of and Then, the three feature matrices are processed in the following ways (A) and (B): (A) The first two dimensions of each of the three feature matrices are changed to one dimension, obtaining three first new feature matrices with dimensions of and Then, the three first new feature matrices are concatenated to obtain a second new feature matrix with a dimension of Then, the second new feature matrix is subjected to Average pooling is performed to obtain a third new feature matrix with dimensions of Len×N. Finally, the third new feature matrix is input into the license plate number sequence modeling unit, where Len represents the length of the license plate number sequence (usually 7); (B) Padding processing of 1×2 and 2×4 is performed on two of the three feature matrices with dimensions of and respectively to obtain two fourth new feature matrices with dimensions of Then, the two fourth new feature matrices and the feature matrix of the three feature matrices with dimensions of are concatenated to obtain a fifth new feature matrix with dimensions of Then, convolution processing of 1×1×(3×(4 + 1)) is performed on the fifth new feature matrix to obtain a sixth new feature matrix with dimensions of Finally, the sixth new feature matrix is input into the license plate character localization modeling unit.

[0065] The aforementioned method (A) corresponds to license plate number sequence modeling, and the aforementioned method (B) corresponds to license plate character localization modeling. Although the two modelings have different ways of concatenating feature matrices, the source data is the same, which also means that the two modelings share the feature matrix. In this way, no matter whether the input license plate image is a single-line license plate image or a double-line license plate image, there will be more accurate localization and recognition effects. In addition, the aforementioned max pooling, average pooling, and padding processing are all existing conventional processing methods. For example, 2×2 max pooling (maxpooling) means taking the maximum value in a 2×2 grid; average pooling (avgpooling) means taking the average value; 1×2 padding (padding) means adding 1 0 to the left and right of the matrix and adding 2 0s to the top and bottom for padding.

[0066] In the step S2, the license plate number sequence modeling unit is used to model the license plate number sequence according to the third new feature matrix, and output the predicted values of the occurrence probabilities of each license plate number character at each character position in the license plate number sequence. Specifically, the license plate number sequence modeling unit uses a Transformer language model (which is a seq2seq model proposed in the paper Attention Is All You Need published by Google Brain at the end of 2017. It abandons the traditional CNN and RNN, and the entire network structure is completely composed of the Attention mechanism. It has surpassed RNN and LSTM in many tasks in NLP, and has good algorithm parallelism and can use GPU to accelerate training) to model the license plate number sequence; the input data of the Transformer language model consists of two parts: one part is the third new feature matrix (with a dimension of Len×128), and the other part is to convert the license plate number into a feature vector with a dimension of Len×128 through the word embedding method. After adding the third new feature matrix and the feature vector, the input data is formed (that is, the encoding of the entire license plate number) to be input into the Transformer language model. Specifically, the aforementioned word embedding method can be but is not limited to the word embedding method based on the Word2Vec model (which is a tool for generating word vectors and is also a model for learning semantic knowledge in an unsupervised manner from a large amount of text corpora); the last fully connected layer of the Transformer language model can be but is not limited to using the Softmax function as the activation function, so as to output the predicted values of the occurrence probabilities of each license plate number character at each character position in the license plate number sequence through the last fully connected layer, that is, the output data is a matrix with a dimension of Len×76, where 76 represents 76 license plate number characters. In addition, the specific process of modeling the license plate number sequence is prior art and will not be elaborated here.

[0067] In the step S2, the license plate number character positioning modeling unit is used to model the license plate number character position according to the sixth new feature matrix, and output the predicted values of the abscissa, ordinate, width, height and confidence of each character grid prediction box. The specific process of modeling the license plate number character position is prior art and will not be elaborated here.

[0068] In the step S2, the loss function calculation unit is used to calculate the loss function value according to the output data of the license plate number sequence modeling unit and the license plate number character positioning modeling unit. Specifically, according to the output data of the license plate number sequence modeling unit and the license plate number character positioning modeling unit, the loss function value Loss is calculated according to the following formula:

[0069]

[0070] In the formula, L1 represents the first loss value of the bounding box coordinates, L2 represents the second loss value of the bounding box coordinates, L3 represents the loss value of the bounding box confidence, CTC loss represents the loss value of Connectionist Temporal Classification (CTC), λ coord represents the first hyperparameter, λ noobj represents the second hyperparameter, i and j are natural numbers respectively, and x i represents the predicted value of the abscissa of the i-th character grid prediction box, represents the true value of the abscissa of the i-th character grid prediction box, y i represents the predicted value of the ordinate of the i-th character grid prediction box, represents the true value of the ordinate of the i-th character grid prediction box, w i represents the predicted value of the width of the i-th character grid prediction box, represents the true value of the width of the i-th character grid prediction box, h i represents the predicted value of the height of the i-th character grid prediction box, represents the true value of the height of the i-th character grid prediction box, C i represents the predicted value of the confidence of the i-th character grid prediction box, represents the true value of the confidence of the i-th character grid prediction box, represents whether there is a target character in the i-th character grid prediction box: if there is, the value is 1, represents whether there is no target character in the i-th character grid prediction box: if there is no, the value is 1. t represents the output data of the license plate number sequence modeling unit, z represents the correct license plate number sequence, R represents the sample training set, and p(z / t) represents the probability of the output sequence z given the input t. Based on the foregoing formula, it can be seen that the loss function is jointly composed of the bounding box parameters of each character and the CTC loss. Among them, the CTC loss constructs a table, and each grid in the table represents the probability that the current position is a certain character. CTC will find all possible paths with the output sequence z from the initial position to the end position according to the given input t, that is, p(z / t) is one of the paths. After taking the product of the path probabilities of all output sequences z and taking the negative logarithm, minimizing the loss function can make the probability of the output sequence z reach the maximum. The foregoing first hyperparameter and second hyperparameter can both be set artificially. Specifically, the first hyperparameter takes the value of 5, and the second hyperparameter takes the value of 0.5. In addition, the foregoing predicted values and true values can all be obtained routinely during model training, and the specific application of the loss function value is a conventional method for existing model training, which will not be elaborated here.

[0071] S3. Input the single-row license plate image or double-row license plate image to be recognized into the trained single-row and double-row license plate recognition model, so as to obtain corresponding output data through the output of the license plate number sequence modeling unit.

[0072] In step S3, the output data is a matrix with a dimension of Len×76, so as to represent the predicted value of the occurrence probability of each license plate number character at each character position in the license plate number sequence.

[0073] S4. For the single-row license plate image or double-row license plate image to be identified, determine the corresponding license plate number characters at the respective character positions according to the corresponding output data, and obtain the corresponding license plate number sequence.

[0074] In step S4, specifically, for each character position, the corresponding license plate character with the maximum predicted probability of occurrence can be used as the corresponding character, thereby combining the characters at all positions to obtain the license plate sequence. For example, a typical license plate sequence has seven character positions. For each of the seven character positions, if the corresponding license plate characters with the maximum predicted probability of occurrence are "環", "A", "3", "8", "5", "1", and "9", respectively, then the license plate sequence "環A38519" can be obtained by concatenating them to serve as the final license plate recognition result.

[0075] Therefore, based on the single-row and double-row license plate recognition method described in the aforementioned steps S1 to S4, a new solution is provided for single-row and double-row license plate recognition that only requires one look, that is, after obtaining a sample training set, each image sample and the corresponding license plate number sequence information in the sample training set are input into a preset single-row and double-row license plate recognition model that can perform license plate number sequence modeling and license plate number character positioning modeling for model training, and then the single-row license plate image or double-row license plate image to be recognized is input into the trained single-row and double-row license plate recognition model so that the corresponding license plate number sequence modeling unit outputs the predicted probability of occurrence of each license plate number character at each character position in the license plate number sequence, and finally the license plate number sequence is obtained based on the output data. In this way, there is no need to classify the input license plate image to be recognized into single-row and double-row license plates, and the license plate recognition can be directly performed and the recognition result can be output, which greatly simplifies the recognition process and reduces the computing resource requirements. In addition, since the license plate image features are extracted through the backbone network, and the license plate number recognition and character positioning are modeled based on the license plate image features during the training process, and the character positioning modeling process is only a convolution operation, it will not take up too many computing resources, which is convenient for practical application and promotion.

[0076] like Figure 4As shown in the figure, in the second aspect of this embodiment, a virtual device for implementing the single and double license plate recognition method described in the first aspect is provided, including a training set acquisition module, an identification model training module, an identification model application module, and a license plate number determination module that are communicatively connected in sequence;

[0077] The training set acquisition module is used to acquire a sample training set. Among them, multiple image samples in the sample training set include single-line license plate images and double-line license plate images. Each image sample in the multiple image samples is labeled with corresponding license plate number sequence information. The license plate number sequence information includes multiple character coordinate data arranged in sequence according to the license plate number sequence. The character coordinate data includes the digital unique identifier of the corresponding character and the normalized horizontal coordinate, vertical coordinate, width, and height of the corresponding character on the license plate image where it is located;

[0078] The identification model training module is used to input each image sample and the corresponding license plate number sequence information into a preset single and double license plate recognition model to train the single and double license plate recognition model. Among them, the single and double license plate recognition model includes an image feature extraction unit, a spatial pyramid pooling unit, a license plate number sequence modeling unit, a license plate character positioning modeling unit, and a loss function calculation unit;

[0079] The image feature extraction unit is used to extract an image feature matrix with a dimension of K×M×N from the image sample and send the image feature matrix to the spatial pyramid pooling unit. Among them, K represents the first least common multiple of 2, 3, and 6, M represents the second least common multiple of 2, 3, and 6 and is greater than the first least common multiple, and N represents a positive integer greater than the second least common multiple;

[0080] The spatial pyramid pooling unit is used to perform maximum pooling processing on the image feature matrix with 2×2, 3×3, and 6×6 respectively, and obtain three feature matrices with dimensions of and Then, the three feature matrices are processed in the following ways (A) and (B):

[0081] (A) Change the first two dimensions of each of the three feature matrices to one dimension, and obtain three first new feature matrices with dimensions of and Then splice the three first new feature matrices to obtain a second new feature matrix with a dimension of Then, perform average pooling processing on the second new feature matrix to obtain a third new feature matrix with a dimension of Len×N. Finally, input the third new feature matrix into the license plate number sequence modeling unit, where Len represents the length of the license plate number sequence;

[0082] (B) For two feature matrices among the three feature matrices with dimensions of and respectively, perform padding processing of 1×2 and 2×4 to obtain two fourth new feature matrices with dimensions of Then, splice the two fourth new feature matrices and the feature matrix among the three feature matrices with dimension of to obtain a fifth new feature matrix with dimension of Then, perform convolution processing of 1×1×(3×(4 + 1)) on the fifth new feature matrix to obtain a sixth new feature matrix with dimension of Finally, input the sixth new feature matrix into the license plate character positioning modeling unit;

[0083] The license plate sequence modeling unit is used to model the license plate sequence according to the third new feature matrix and output the predicted values of the occurrence probabilities of each license plate character at each character position in the license plate sequence;

[0084] The license plate character positioning modeling unit is used to model the license plate character position according to the sixth new feature matrix and output the predicted values of the abscissa, ordinate, width, height, and confidence of each character grid prediction box;

[0085] The loss function calculation unit is used to calculate the loss function value according to the output data of the license plate sequence modeling unit and the license plate character positioning modeling unit;

[0086] The recognition model application module is used to input a single-line license plate image or a double-line license plate image to be recognized into the trained single / double-line license plate recognition model, so as to output corresponding output data through the license plate sequence modeling unit;

[0087] The license plate determination module is used to determine the license plate characters corresponding to the single-line license plate image or the double-line license plate image to be recognized at each character position according to the corresponding output data, and obtain the corresponding license plate sequence.

[0088] For the working process, working details, and technical effects of the foregoing device provided in the second aspect of this embodiment, reference may be made to the single / double-line license plate recognition method described in the first aspect, which will not be elaborated herein.

[0089] As Figure 5As shown in the figure, in the third aspect of this embodiment, a computer device for executing the single and double license plate recognition method described in the first aspect is provided, including a memory, a processor, and a transceiver that are communicatively connected in sequence. Among them, the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the single and double license plate recognition method described in the first aspect. Specifically, for example, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first input first output (FIFO), and / or first input last output (FILO), etc.; the processor may include, but is not limited to, a microprocessor of the STM32F105 series. In addition, the computer device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0090] For the working process, working details, and technical effects of the foregoing computer device provided in the third aspect of this embodiment, reference may be made to the single and double license plate recognition method described in the first aspect, which will not be elaborated herein.

[0091] In the fourth aspect of this embodiment, a computer-readable storage medium storing instructions for the single and double license plate recognition method described in the first aspect is provided, that is, instructions are stored on the computer-readable storage medium, and when the instructions are run on a computer, the single and double license plate recognition method described in the first aspect is executed. Among them, the computer-readable storage medium refers to a carrier for storing data, and may include, but is not limited to, floppy disks, optical discs, hard disks, flash memories, USB flash drives, and / or memory sticks, etc. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices.

[0092] For the working process, working details, and technical effects of the foregoing computer-readable storage medium provided in the fourth aspect of this embodiment, reference may be made to the single and double license plate recognition method described in the first aspect, which will not be elaborated herein.

[0093] In the fifth aspect of this embodiment, a computer program product containing instructions is provided, and when the instructions are run on a computer, the computer is made to execute the single and double license plate recognition method described in the first aspect. Among them, the computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices.

[0094] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A single and double row license plate recognition method, characterized in that: include: Obtaining a sample training set, wherein the plurality of image samples in the sample training set include single-row license plate images and double-row license plate images, each image sample in the plurality of image samples is annotated with corresponding license plate number sequence information, the license plate number sequence information includes a plurality of character coordinate data sequentially arranged in license plate number sequence order, the character coordinate data including a digital unique identifier of a corresponding character and a normalized horizontal coordinate, vertical coordinate, width, and height of the corresponding character on the license plate image; Inputting each of the image samples and the corresponding license plate number sequence information into a preset single-row and double-row license plate recognition model to train the single-row and double-row license plate recognition model, wherein the single-row and double-row license plate recognition model includes an image feature extraction unit, a spatial pyramid pooling unit, a license plate number sequence modeling unit, a license plate number character positioning modeling unit, and a loss function calculation unit; The image feature extraction unit is configured to extract an image feature matrix of dimension K×M×N from the image sample, and input the image feature matrix into the spatial pyramid pooling unit, where K represents a first common multiple of 2, 3, and 6, M represents a second common multiple of 2, 3, and 6 and is greater than the first common multiple, and N represents a positive integer greater than the second common multiple; The spatial pyramid pooling unit is used to perform 2×2, 3×3 and 6×6 maximum pooling processing on the image feature matrix, and the dimensions are and Then, the three characteristic matrices are processed as follows (A) and (B): (A) The first two dimensions of each of the three feature matrices are changed to one dimension, and the dimensions are and Then concatenate the three first new feature matrices to obtain a dimension of A second new feature matrix, and then the second new feature matrix is Average pooling is performed to obtain a third new feature matrix with a dimension of Len×N, and finally the third new feature matrix is input into the license plate number sequence modeling unit, where Len represents the length of the license plate number sequence; (B) For the three feature matrices whose dimensions are and The two feature matrices of are filled with 1×2 and 2×4 respectively, and the dimensions are Then, the two fourth new feature matrices and the three feature matrices are The feature matrix of is spliced to obtain a dimension of A fifth new feature matrix is obtained, and then the fifth new feature matrix is subjected to a 1×1×(3×(4+1)) convolution process to obtain a dimension of Finally, the sixth new feature matrix is input into the license plate number character positioning modeling unit; The license plate number sequence modeling unit is used to model the license plate number sequence according to the third new feature matrix, and output a predicted value of the occurrence probability of each license plate number character at each character position in the license plate number sequence; The license plate number character positioning modeling unit is used to model the license plate number character position according to the sixth new feature matrix, and output the horizontal coordinate prediction value, vertical coordinate prediction value, width prediction value, height prediction value and confidence prediction value of each character grid prediction box; The loss function calculation unit is used to calculate the loss function value based on the output data of the license plate number sequence modeling unit and the license plate number character positioning modeling unit; Inputting the single-row license plate image or the double-row license plate image to be recognized into the trained single-row and double-row license plate recognition model so as to obtain corresponding output data through the license plate number sequence modeling unit; For the single-row license plate image or double-row license plate image to be identified, the corresponding license plate number characters at the respective character positions are determined according to the corresponding output data to obtain a corresponding license plate number sequence.

2. The single and double row license plate recognition method according to claim 1, characterized in that: The image feature extraction unit includes a first convolutional layer, a second convolutional layer, a first convolutional residual subunit, a second convolutional residual subunit, a third convolutional residual subunit, a fourth convolutional residual subunit, an eleventh convolutional layer, a cross-stage local network CSPNet layer and a fifth convolutional residual subunit, wherein the first convolutional residual subunit, the second convolutional residual subunit, the third convolutional residual subunit, the fourth convolutional residual subunit and the fifth convolutional residual subunit respectively include two convolutional layers and a residual block arranged in sequence from the input to the output direction; The cross-stage local network CSPNet layer is used to divide the input into two parts: one part is forwarded into the internal dense block, and the other part is sent to the spatial pyramid pooling unit as an image feature matrix.

3. The single and double row license plate recognition method according to claim 1, characterized in that: The license plate number sequence modeling unit uses a transformer language model to model the license plate number sequence; The input data of the transformer language model consists of two parts: one part is the third new feature matrix, and the other part is the license plate number converted into a feature vector with a dimension of Len×128 through the word embedding method. The third new feature matrix and the feature vector are added together to form the input data for input into the transformer language model.

4. The single and double row license plate recognition method according to claim 3, characterized in that: The word embedding method adopts a word embedding method based on the Word2Vec model.

5. The single and double row license plate recognition method according to claim 3, characterized in that: The last fully connected layer of the transformer language model uses the Softmax function as the activation function.

6. The single and double row license plate recognition method according to claim 1, characterized in that: The loss function value is calculated based on the output data of the license plate sequence modeling unit and the license plate character positioning modeling unit, including: According to the output data of the license plate sequence modeling unit and the license plate character positioning modeling unit, the loss function value Loss is calculated according to the following formula: Where L1 represents the first loss value of the bounding box coordinates, L2 represents the second loss value of the bounding box coordinates, L3 represents the loss value of the bounding box confidence, and CTC loss represents the connectionist temporal classification CTC loss value, λ coord represents the first hyperparameter, λ noobj represents the second hyperparameter, i and j are natural numbers, x i Represents the horizontal coordinate prediction value of the i-th character grid prediction box, Represents the true value of the horizontal coordinate of the i-th character grid prediction box, y i represents the predicted value of the ordinate of the i-th character grid prediction box, represents the true value of the vertical coordinate of the i-th character grid prediction box, w i represents the predicted value of the width of the i-th character grid prediction box, represents the true value of the width of the i-th character grid prediction box, h i represents the height prediction value of the i-th character grid prediction box, represents the true value of the height of the i-th character grid prediction box, C i represents the confidence prediction value of the i-th character grid prediction box, represents the true value of the confidence of the i-th character grid prediction box, Indicates whether there is a target character in the i-th character grid prediction box: if yes, the value is 1, Indicates whether there is no target character in the i-th character grid prediction box: if not, the value is 1, t represents the output data of the license plate number sequence modeling unit, z represents the correct license plate number sequence, R represents the sample training set, and p(z / t) represents the probability of outputting sequence z when given input t.

7. The single and double row license plate recognition method according to claim 6, characterized in that: The first hyperparameter value is 5, and the second hyperparameter value is 0.

5.

8. A single and double row license plate recognition device, characterized in that: It includes a training set acquisition module, a recognition model training module, a recognition model application module and a license plate number determination module which are sequentially connected in communication; The training set acquisition module is used to acquire a sample training set, wherein the plurality of image samples in the sample training set include single-row license plate images and double-row license plate images, each image sample in the plurality of image samples is annotated with corresponding license plate number sequence information, the license plate number sequence information includes a plurality of character coordinate data arranged in sequence according to the license plate number sequence order, the character coordinate data includes a digital unique identifier of the corresponding character and a normalized horizontal coordinate, vertical coordinate, width, and height of the corresponding character on the license plate image; The recognition model training module is used to input the image samples and the corresponding license plate number sequence information into a preset single-row and double-row license plate recognition model to train the single-row and double-row license plate recognition model, wherein the single-row and double-row license plate recognition model includes an image feature extraction unit, a spatial pyramid pooling unit, a license plate number sequence modeling unit, a license plate number character positioning modeling unit, and a loss function calculation unit; The image feature extraction unit is configured to extract an image feature matrix of dimension K×M×N from the image sample, and input the image feature matrix into the spatial pyramid pooling unit, where K represents a first common multiple of 2, 3, and 6, M represents a second common multiple of 2, 3, and 6 and is greater than the first common multiple, and N represents a positive integer greater than the second common multiple; The spatial pyramid pooling unit is used to perform 2×2, 3×3 and 6×6 maximum pooling processing on the image feature matrix, and the dimensions are and Then, the three characteristic matrices are processed as follows (A) and (B): (A) The first two dimensions of each of the three feature matrices are changed to one dimension, and the dimensions are and Then concatenate the three first new feature matrices to obtain a dimension of A second new feature matrix, and then the second new feature matrix is Average pooling is performed to obtain a third new feature matrix with a dimension of Len×N, and finally the third new feature matrix is input into the license plate number sequence modeling unit, where Len represents the length of the license plate number sequence; (B) For the three feature matrices whose dimensions are and The two feature matrices of are filled with 1×2 and 2×4 respectively, and the dimensions are Then, the two fourth new feature matrices and the three feature matrices are The feature matrix of is spliced to obtain a dimension of A fifth new feature matrix is obtained, and then the fifth new feature matrix is subjected to a 1×1×(3×(4+1)) convolution process to obtain a dimension of Finally, the sixth new feature matrix is input into the license plate number character positioning modeling unit; The license plate number sequence modeling unit is used to model the license plate number sequence according to the third new feature matrix, and output a predicted value of the occurrence probability of each license plate number character at each character position in the license plate number sequence; The license plate number character positioning modeling unit is used to model the license plate number character position according to the sixth new feature matrix, and output the horizontal coordinate prediction value, vertical coordinate prediction value, width prediction value, height prediction value and confidence prediction value of each character grid prediction box; The loss function calculation unit is used to calculate the loss function value based on the output data of the license plate number sequence modeling unit and the license plate number character positioning modeling unit; The recognition model application module is used to input the single-row license plate image or the double-row license plate image to be recognized into the trained single-row and double-row license plate recognition model, so as to obtain corresponding output data through the output of the license plate number sequence modeling unit; The license plate number determination module is used to determine the corresponding license plate number characters at the respective character positions according to the corresponding output data for the single-row license plate image or the double-row license plate image to be identified, and obtain the corresponding license plate number sequence.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the single-row and double-row license plate recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are run on a computer, the single-row and double-row license plate recognition method as described in any one of claims 1 to 7 is executed.

Images