Single-double line license plate information recognition method and system based on transformer

Abstract

This application relates to a method and system for recognizing single and double license plate information based on Transformer. The method includes: acquiring first image data, the first image data containing a first object, which is the license plate information to be recognized; extracting features from the first image data using a first convolutional neural network to obtain a three-dimensional feature matrix of the first object; recognizing the three-dimensional feature matrix using a second convolutional neural network to obtain the single / double row category of the first object; reshaping and transposing the three-dimensional feature matrix to obtain a two-dimensional feature matrix of the first object; and obtaining multiple first vectors based on the two-dimensional feature matrix, which are used to represent the features of the first object. This method achieves single / double row classification of license plate information by obtaining the three-dimensional feature matrix of the license plate information to be recognized, avoiding errors from manual classification and complex model construction, and improving the efficiency of single / double row license plate recognition to a certain extent.

Description

Technical Field

[0001] This application relates to the field of license plate recognition technology, and in particular to a method and system for recognizing single and double license plate information based on Transformer. Background Technology

[0002] Vehicle License Plate Recognition (VLPR) is an image processing technology that integrates front-end license plate image acquisition with back-end recognition and processing. It uses on-site image acquisition cameras to capture vehicle license plate images, transmits them to the back-end recognition system for image processing, and uses visual recognition to identify the license plate number. VLPR technology has been widely applied in various scenarios, such as license plate recognition and unmanned toll collection systems in parking lots, electronic toll collection (ETC) systems on highways, speeding violation recognition systems based on traffic police cameras, road traffic flow monitoring and management systems, and automatic license plate classification and release control in special areas or units (military-controlled areas, classified units, etc.). With the further development of computer video image recognition technology, the accuracy and efficiency of VLPR technology are gradually improving, playing an important practical role in maintaining traffic safety and urban security, preventing traffic congestion, realizing automated traffic management, and the scientific and efficient management of residential areas and parking lots.

[0003] Existing license plate recognition technology, VLPR, mainly consists of a front-end and a back-end. The front-end is responsible for image acquisition and needs to be deployed at the vehicle's passageway, aligned with the vehicle entrance direction, and its angle and orientation adjusted. The back-end is responsible for image processing and processing of the recognition results. The purpose of image processing is to identify the license plate. The main recognition algorithms include license plate detection and license plate recognition. License plate detection refers to locating the license plate position from the image acquired by the front-end, which can employ grayscale detection, edge detection, or object detection algorithms (anchored or unanchored algorithms). License plate recognition is the key part, and there are many existing algorithms. Currently, the industry's methods for recognizing single-lane and dual-lane license plate information generally involve first classifying the license plate to determine whether it is a single-lane or dual-lane license plate, and then, based on the classification result, calling different OCR (Optical Character Recognition) models for character recognition (e.g., ...). Figure 1 As shown in the image, this method requires one classification model and two different OCR recognition models. The large number of models leads to reduced recognition efficiency. Another method involves classifying license plates by single / double line (e.g., ...). Figure 2 As shown in the figure, the double-line license plate is then segmented into upper and lower lines, and then stitched into a single-line image. The OCR model is then called for character recognition. Although this method only requires one OCR recognition model, it requires the double-line license plate to be segmented into upper and lower lines first. The accuracy of the segmentation directly affects the efficiency of license plate recognition.

[0004] In summary, traditional single / double license plate recognition methods have low license plate recognition efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a Transformer-based method and system for recognizing single and double license plate information with high recognition efficiency to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a method for recognizing single and double license plate information based on Transformer, the method comprising:

[0007] Acquire first image data, which contains a first object, the first object being the license plate information to be identified;

[0008] The first image data is used to extract features through a first convolutional neural network to obtain the three-dimensional feature matrix of the first object;

[0009] The three-dimensional feature matrix is ​​identified by a second convolutional neural network to obtain the single or double row category of the first object;

[0010] The three-dimensional feature matrix is ​​reshaped and transposed to obtain the two-dimensional feature matrix of the first object;

[0011] Multiple first vectors are obtained based on the two-dimensional feature matrix, and the multiple first vectors are used to express the features of the first object.

[0012] In one embodiment, the step of reshaping and transposing the three-dimensional feature matrix includes:

[0013] The encoded feature matrix corresponding to the two-dimensional feature matrix is ​​obtained through the encoding unit;

[0014] The decoding feature matrix corresponding to the encoded feature matrix is ​​obtained through the decoding unit.

[0015] In one embodiment, the step of reshaping and transposing the three-dimensional feature matrix further includes:

[0016] The feature representation vector of the first object is obtained based on the decoded feature matrix. The feature representation vector is composed of the plurality of first vectors and is used to obtain the recognition result of the first object.

[0017] In one embodiment, the encoding unit comprises multiple encoders, each encoder having a multi-head self-attention layer and a first feedforward network layer; the method further includes:

[0018] The multi-head self-attention layer transforms the two-dimensional feature matrix into a corresponding feature representation matrix.

[0019] The feature representation matrix is ​​output through the first feedforward network layer.

[0020] In one embodiment, the decoding unit comprises multiple decoders, each decoder having an attention layer and a first feedforward network layer; the method further includes:

[0021] The attention layer transforms the feature representation matrix into a corresponding feature representation vector.

[0022] The feature representation vector is output as the corresponding recognition result through the second feedforward network layer.

[0023] In one embodiment, the identification result of the first object includes a first predicted symbol, a second predicted symbol, and a third predicted symbol, and the method further includes:

[0024] The first predicted symbol is obtained based on the feature representation vector, and the first predicted symbol is the first recognition result of the first object;

[0025] The second predicted symbol is obtained based on the first predicted symbol, and the second predicted symbol is the recognition result obtained based on the first predicted symbol;

[0026] The third prediction symbol is obtained based on the second prediction symbol. The third prediction symbol is a termination symbol used to indicate that the first object identification is complete.

[0027] In one embodiment, the single / double row category of the first object includes a single row category and a double row category, and the method further includes:

[0028] When the first object is the single-row category, the first image data is directly reshaped and transposed to obtain a two-dimensional feature matrix corresponding to the first image data.

[0029] Multiple second vectors are obtained based on the two-dimensional feature matrix, and the multiple second vectors are used to express the features of the first object;

[0030] The feature representation vector of the first object is obtained based on the plurality of second vectors.

[0031] Secondly, this application provides a Transformer-based single / double license plate information recognition system, the system comprising:

[0032] An image acquisition module is used to acquire first image data, wherein the first image data contains a first object, and the first object is license plate information to be identified;

[0033] The feature extraction module is used to extract features from the first image data through a first convolutional neural network to obtain a three-dimensional feature matrix of the first object.

[0034] The feature recognition module is used to identify the three-dimensional feature matrix through a second convolutional neural network to obtain the single or double row category of the first object;

[0035] The reshaping and transposing module is used to reshape and transpose the three-dimensional feature matrix to obtain the two-dimensional feature matrix of the first object.

[0036] The feature representation module is used to obtain multiple first vectors based on the two-dimensional feature matrix, and the multiple first vectors are used to represent the features of the first object.

[0037] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0038] Acquire first image data, which contains a first object, the first object being the license plate information to be identified;

[0039] The first image data is used to extract features through a first convolutional neural network to obtain the three-dimensional feature matrix of the first object;

[0040] The three-dimensional feature matrix is ​​identified by a second convolutional neural network to obtain the single or double row category of the first object;

[0041] The three-dimensional feature matrix is ​​reshaped and transposed to obtain the two-dimensional feature matrix of the first object;

[0042] Multiple first vectors are obtained based on the two-dimensional feature matrix, and the multiple first vectors are used to express the features of the first object.

[0043] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0044] Acquire first image data, which contains a first object, the first object being the license plate information to be identified;

[0045] The first image data is used to extract features through a first convolutional neural network to obtain the three-dimensional feature matrix of the first object;

[0046] The three-dimensional feature matrix is ​​identified by a second convolutional neural network to obtain the single or double row category of the first object;

[0047] The three-dimensional feature matrix is ​​reshaped and transposed to obtain the two-dimensional feature matrix of the first object;

[0048] Multiple first vectors are obtained based on the two-dimensional feature matrix, and the multiple first vectors are used to express the features of the first object.

[0049] The aforementioned Transformer-based method and system for recognizing single / double license plate information acquires image data of the license plate to be recognized, and then uses a convolutional neural network to extract features from the image data to obtain a three-dimensional feature matrix of the license plate. Based on these three-dimensional features, a newly constructed convolutional neural network is then used to recognize the license plate, thereby determining its single / double row category. After classifying the license plate as single / double row, the three-dimensional feature matrix of the license plate is reshaped and transposed to obtain a two-dimensional feature matrix, which is then divided into multiple feature vectors. This method achieves single / double row classification of license plate information by obtaining the three-dimensional feature matrix, avoiding errors from manual classification and complex model construction, and thus improving the efficiency of single / double row license plate recognition to a certain extent. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the traditional multi-OCR model license plate recognition process in this application;

[0051] Figure 2 This is a schematic diagram of the traditional single OCR model license plate recognition process in this application;

[0052] Figure 3 This is one of the flowcharts for the Transformer-based single / double license plate information recognition method of this application;

[0053] Figure 4 This is the second flowchart of the Transformer-based method for recognizing single and double license plate information in this application;

[0054] Figure 5 This is the third flowchart of the Transformer-based single / double license plate information recognition method of this application;

[0055] Figure 6 This is the fourth flowchart of the Transformer-based method for recognizing single and double license plate information in this application;

[0056] Figure 7 This is the fifth flowchart of the Transformer-based method for recognizing single and double license plate information in this application;

[0057] Figure 8This is a schematic diagram illustrating the single / double license plate recognition process of a Transformer-based single / double license plate information recognition scheme according to an embodiment of this application.

[0058] Figure 9 This is a schematic diagram of the Transformer model structure of the single / double license plate information recognition scheme based on Transformer in this embodiment;

[0059] Figure 10 This is a schematic diagram of the double-lane license plate feature sequence of the single and double-lane license plate information recognition scheme based on Transformer in this embodiment;

[0060] Figure 11 This is a schematic diagram showing the character positions of single-lane and double-lane license plates in the Transformer-based single / double-lane license plate information recognition scheme of this embodiment.

[0061] Figure 12 This is a schematic diagram of the license plate recognition model structure of the Transformer-based single and double license plate information recognition scheme in this embodiment;

[0062] Figure 13 This is a schematic diagram of the structure of a Transformer-based single / double license plate information recognition system according to an embodiment of this application;

[0063] Figure 14 This is an internal structural diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] like Figure 3 As shown in one embodiment, a method for recognizing single and double license plate information based on Transformer includes the following steps:

[0066] Step S310: Obtain first image data, which contains a first object, namely the license plate information to be identified.

[0067] Specifically, the server acquires image data containing the license plate information to be identified.

[0068] Step S320: Extract features from the first image data using a first convolutional neural network to obtain a three-dimensional feature matrix of the first object.

[0069] Specifically, the server uses a convolutional neural network to extract features from the image data obtained in step S310 to obtain a three-dimensional feature matrix of the license plate information to be identified.

[0070] Step S330: The three-dimensional feature matrix is ​​identified by the second convolutional neural network to obtain the single or double row category of the first object.

[0071] Specifically, the server uses a newly constructed convolutional neural network to identify the three-dimensional feature matrix obtained in step S320, thereby obtaining the single or double row category of the license plate information to be identified.

[0072] Step S340: Reshape and transpose the three-dimensional feature matrix to obtain the two-dimensional feature matrix of the first object.

[0073] Specifically, the server reshapes and transposes the three-dimensional feature matrix of the license plate information to be identified based on the results of single and double line recognition in step S330, so as to obtain the two-dimensional feature matrix of the license plate information to be identified.

[0074] It should be noted that after the convolutional neural network performs single / double recognition on the license plate information to be identified, the server will mark the license plate as single / double line type according to the single / double line of the license plate information. After marking, the license plate is reshaped and transposed. The single / double line type marking reduces the probability of error in the license plate recognition process.

[0075] Step S350: Obtain multiple first vectors based on the two-dimensional feature matrix. The multiple first vectors are used to express the features of the first object.

[0076] Specifically, the server obtains multiple vectors based on the two-dimensional feature matrix of the license plate information to be identified, and uses these multiple vectors to realize the feature representation of the vehicle information to be identified.

[0077] The aforementioned Transformer-based method for recognizing single / double license plate information acquires image data of the license plate to be recognized and extracts features from this image data using a convolutional neural network (CNN) to obtain a three-dimensional feature matrix. Then, a newly constructed CNN is used to recognize the license plate information in this three-dimensional feature state to determine its single / double row category. After classifying the license plate as single / double row, the three-dimensional feature matrix is ​​reshaped and transposed to obtain a two-dimensional feature matrix, which is then divided into multiple feature vectors. This method achieves single / double row classification by obtaining the three-dimensional feature matrix of the license plate, avoiding errors from manual classification and complex model construction, thus improving the efficiency of single / double row license plate recognition to a certain extent.

[0078] like Figure 4 As shown, in one embodiment, the Transformer-based single / double license plate information recognition method of this application reshapes and transposes the three-dimensional feature matrix, and then includes the following steps:

[0079] Step S410: Obtain the encoded feature matrix corresponding to the two-dimensional feature matrix through the encoding unit.

[0080] Specifically, the server obtains the encoded feature matrix corresponding to the two-dimensional feature matrix of the license plate information to be identified through the encoding unit.

[0081] Step S420: Obtain the decoding feature matrix corresponding to the encoding feature matrix through the decoding unit.

[0082] Specifically, the server obtains the decoded feature matrix corresponding to the encoded feature matrix obtained in step S410 through the decoding unit.

[0083] Step S430: Obtain the feature representation vector of the first object based on the decoded feature matrix. The feature representation vector is composed of multiple first vectors and is used to obtain the recognition result of the first object.

[0084] Specifically, the server obtains the feature representation vector of the license plate information to be identified based on the decoding feature matrix obtained in step S420. The feature representation vector is composed of multiple feature vectors and is used to obtain the recognition result of the license plate information to be identified.

[0085] like Figure 5As shown, in one embodiment, the Transformer-based single / double license plate information recognition method of this application includes an encoding unit composed of multiple encoders, each encoder having a multi-head self-attention layer and a first feedforward network layer; a decoding unit composed of multiple decoders, each decoder having an attention layer and a second feedforward network layer; and further includes the following steps:

[0086] Step S510: The two-dimensional feature matrix is ​​transformed into the corresponding feature representation matrix through a multi-head self-attention layer.

[0087] Specifically, the server uses a multi-head self-attention layer in the encoder to transform the two-dimensional feature matrix of the license plate information to be identified into a corresponding feature representation matrix.

[0088] Step S520: Output the feature representation matrix through the first feedforward network layer.

[0089] Specifically, the server outputs the feature representation matrix obtained in step S510 through the first feedforward network layer in the encoder.

[0090] Step S530: The feature representation matrix is ​​transformed into the corresponding feature representation vector through the attention layer.

[0091] Specifically, the server-side uses the attention layer in the decoder to transform the feature representation matrix into the corresponding feature representation vector.

[0092] Step S540: The feature representation vector is output as the corresponding recognition result through the second feedforward network layer.

[0093] Specifically, the server outputs the feature representation vector obtained in step S530 as the corresponding recognition result through the second feedforward network layer in the decoder.

[0094] like Figure 6 As shown, in one embodiment, the Transformer-based single / double license plate information recognition method of this application further includes the following steps:

[0095] Step S610: Obtain the first predicted symbol based on the feature representation vector. The first predicted symbol is the first recognition result of the first object.

[0096] Specifically, the server obtains the first recognition result of the license plate information based on the feature representation vector of the license plate information to be recognized.

[0097] Step S620: Obtain a second predicted symbol based on the first predicted symbol. The second predicted symbol is the recognition result obtained based on the first predicted symbol.

[0098] Specifically, the server obtains other recognition results related to the first recognition result of the license plate information based on the first recognition result, and repeats this step to achieve the recognition of complete characters in the license plate information.

[0099] Step S630: Obtain the third prediction symbol based on the second prediction symbol. The third prediction symbol is a termination symbol used to indicate that the first object recognition is complete.

[0100] Specifically, the server repeats step S620 until it obtains the termination prediction symbol for the license plate information, thus completing the recognition of the license plate information.

[0101] It should be noted that the complete recognition result of the license plate information is composed of the first prediction symbol, the second prediction symbol, and the third prediction symbol.

[0102] like Figure 7 As shown, in one embodiment, the Transformer-based single / double license plate information recognition method of this application, where the single / double license plate information includes single-line and double-line categories, further includes the following steps:

[0103] Step S710: When the first object is a single-row category, directly reshape and transpose the first image data to obtain a two-dimensional feature matrix corresponding to the first image data.

[0104] Specifically, when the license plate information to be identified is of the single-line category, the server does not need to use a convolutional neural network to perform single or double-line category recognition on the license plate image. Instead, it directly reshapes and transposes the single-line license plate image to obtain the two-dimensional feature matrix of the single-line license plate image.

[0105] Step S720: Obtain multiple second vectors based on the two-dimensional feature matrix. These multiple second vectors are used to express the features of the first object.

[0106] Specifically, the server obtains feature vectors of multiple single-lane license plates based on the two-dimensional feature matrix of the single-lane license plate image.

[0107] Step S730: Obtain the feature representation vector of the first object based on multiple second vectors.

[0108] Specifically, the server obtains the feature representation vector of the single-lane license plate based on the feature vector obtained in step S720, so as to realize the recognition of the single-lane license plate information.

[0109] like Figures 8 to 9As shown in one embodiment, a Transformer-based single / double license plate information recognition scheme supports the recognition of single-line and double-line license plate characters and directly outputs the single / double line category and character recognition results of the license plate, without the need for multiple models or multiple processes to be cascaded, so as to achieve end-to-end recognition of single / double line license plate information.

[0110] First, an image of the license plate to be identified is acquired using a camera. This image is then scaled to a fixed height of H and width of W (H and W are not fixed), resulting in an image size of 3×H×W. The scaled image is then input into a feature extraction module, which contains a convolutional neural network (CNN) layer 1 (one of the following: ResNet, VGGNet, DenseNet, etc.). After passing through CNN 1, the license plate image outputs a three-dimensional feature matrix M with dimensions K×(H / p)×(W / q), where the values ​​of p and q are determined by the CNN's parameter settings. Two branches then emerge: branch 1 and branch 2. Branch 1 inputs the feature matrix M into a newly constructed CNN layer 2, a fully connected layer, and a softmax layer to predict the single or double row category of the license plate. Branch 2 reshapes and transposes matrix M to obtain a two-dimensional feature matrix Z of (H*W / (p*q))×K, which is used as input to the Transformer model. After adding position embedding to the two-dimensional feature matrix Z, we get a W×K feature matrix, where K is the dimension of the feature vector.

[0111] It should be noted that when the license plate image is a single-line license plate, the dimensionality of the license plate image with size 3×H×W is directly reshaped to obtain a two-dimensional input matrix of size W×3H, and a linear transformation is performed on the two-dimensional input matrix of size W×3H to add position embedding.

[0112] Next, the W×K feature matrix is ​​input into the encoding module, which consists of N identical encoders. Each encoder contains a multi-head self-attention layer and a feedforward network layer. After passing through the encoding module, each feature vector pays attention to other feature vectors in the input sequence, forming a better feature representation vector. At this point, the encoding module outputs a better W×M feature matrix, where M is the dimension of the encoded feature vector, determined by the network parameters of the encoding module. Decoding then begins, using an initial symbol as input to the decoding module, which consists of N identical decoders. The second attention layer in each decoder uses the encoded feature matrix output by the encoder to obtain a feature representation vector related to the current symbol to be identified. After passing through the feedforward network layer, it outputs the currently predicted symbol.

[0113] Finally, the predicted output symbols are used as input to the decoding module to predict the next symbol. This process is repeated cyclically using the predicted output symbols to predict the next symbol until a specified termination symbol is predicted. At this point, all the predicted symbols are the license plate information character recognition results.

[0114] In this embodiment, assuming the input license plate image is a double-line license plate with H / p = 4, the feature matrix output by the convolutional neural network is mapped onto the original image, such as... Figure 10 As shown, each K-dimensional feature vector represents the features expressed by a small block in the license plate image. After rearranging all the blocks, a new feature sequence is formed. Then, through the self-attention mechanism of the Transformer encoding module, the expressive features of the characters in the upper and lower lines are distinguished, thus forming a better feature representation vector for all characters. Even assuming the input license plate image is a single-line license plate, the self-attention mechanism of the Transformer encoding module can still fuse the relevant sequence features after introducing the feature extraction module, forming the correct feature representation vector, so it does not affect the character recognition of single-line license plates. During the first decoding, the decoding module uses this symbol information to start searching for sequence features related to the first character.

[0115] The first character of a single-lane license plate and a double-lane license plate are not in the same position, such as... Figure 11 As shown, the first character of a single-row license plate starts from sequence 1, while the first character of a double-row license plate starts from sequence 9 in the rearranged feature sequence. Two initial special symbols are defined in the OCR dictionary to represent the "single" and "double" license plate categories, respectively. If the input image is a single-row license plate, the initial input to the decoding module is a symbol representing "single"; if the input is a double-row license plate, the initial input for the first decoding is a symbol representing "double". Whether the initial input to the decoding module is a "single" or "double" symbol is determined by the single / double license plate category predicted in branch 1 of the preceding feature extraction module. The entire license plate recognition model network structure is as follows: Figure 12 As shown, output1 represents the single or double line type of the license plate, and output2 represents the license plate character information.

[0116] In the training process of the license plate information recognition model, the training data consists of a set of license plate region images and corresponding label files indicating the single / double row category of the license plate numbers. First, using the known single / double row license plate images and corresponding license plate number labels, the network parameters of Convolutional Neural Network Layer 1 and Transformer encoder / decoder can be trained. At this point, the initial input to the decoding module is the license plate single / double row category label. Next, locking the already trained network parameters in Convolutional Neural Network Layer 1, the network parameters of Convolutional Neural Network Layer 2 and the fully connected layer can be trained using the license plate images and corresponding single / double row category labels, ultimately completing the training of the entire license plate information recognition model.

[0117] The aforementioned Transformer-based single / double license plate recognition scheme obtains a W×K feature matrix, maps it onto the original image, and makes each K-dimensional feature vector represent the features expressed in each row of the license plate image. Through the self-attention mechanism of the Transformer encoding module, each sequence feature is fused with other related sequence features to form a better feature representation vector. The decoding module obtains the sequence features related to each license plate character to be recognized, predicts each character in one go, and finally achieves the recognition of the entire license plate character set. Furthermore, this scheme uses a single model to complete both single / double license plate classification and character recognition, eliminating the need for multiple model cascades and conditional multi-process execution, achieving end-to-end license plate recognition. Based on the Transformer model, it has been optimized for double-row license plate character recognition, enabling the model to simultaneously support the recognition of both single-row and double-row license plate characters, thus improving the efficiency of single / double license plate recognition to a certain extent.

[0118] like Figure 13 As shown, in one embodiment, a Transformer-based single / double license plate information recognition system includes an image acquisition module 1310, a feature extraction module 1320, a feature recognition module 1330, a reshaping and transposition module 1340, and a feature expression module 1350.

[0119] The image acquisition module 1310 is used to acquire first image data, which contains a first object, namely, the license plate information to be identified.

[0120] The feature extraction module 1320 is used to extract features from the first image data through a first convolutional neural network to obtain a three-dimensional feature matrix of the first object.

[0121] The feature recognition module 1330 is used to recognize the three-dimensional feature matrix through the second convolutional neural network to obtain the single or double row category of the first object.

[0122] The reshaping and transpose module 1340 is used to reshape and transpose the three-dimensional feature matrix to obtain the two-dimensional feature matrix of the first object.

[0123] The feature representation module 1350 is used to obtain multiple first vectors based on the two-dimensional feature matrix, and the multiple first vectors are used to represent the features of the first object.

[0124] The aforementioned Transformer-based single / double lane license plate recognition system acquires image data of the license plate information to be recognized and extracts features from the image data using a convolutional neural network to obtain a three-dimensional feature matrix of the license plate information. Then, a newly constructed convolutional neural network is used to recognize the license plate information in the three-dimensional feature state to determine its single / double lane category. After classifying the license plate information as single / double lane, the three-dimensional feature matrix of the license plate information is reshaped and transposed to obtain a two-dimensional feature matrix, which is then divided into multiple feature vectors. This system achieves single / double lane classification of license plate information by obtaining the three-dimensional feature matrix, avoiding errors from manual classification and complex model construction, and thus improving the efficiency of single / double lane license plate recognition to a certain extent.

[0125] In one embodiment, the Transformer-based single / double license plate information recognition system of this application further includes an encoding module, a decoding module, and a vector acquisition module.

[0126] The encoding module is used to obtain the encoded feature matrix corresponding to the two-dimensional feature matrix through the encoding unit.

[0127] The decoding module is used to obtain the decoded feature matrix corresponding to the encoded feature matrix through the decoding unit.

[0128] The vector acquisition module is used to obtain the feature representation vector of the first object based on the decoded feature matrix. The feature representation vector is composed of multiple first vectors and is used to obtain the recognition result of the first object.

[0129] In one embodiment, the Transformer-based single / double license plate information recognition system of this application further includes a multi-head self-attention module, a first feedforward network module, an attention module, and a second feedforward network module.

[0130] The multi-head self-attention module is used to transform a two-dimensional feature matrix into a corresponding feature representation matrix through a multi-head self-attention layer.

[0131] The first feedforward network module is used to output the feature representation matrix through the first feedforward network layer.

[0132] The attention module is used to transform the feature representation matrix into the corresponding feature representation vector through the attention layer.

[0133] The second feedforward network module is used to output the feature representation vector as the corresponding recognition result through the second feedforward network layer.

[0134] In one embodiment, the Transformer-based single / double license plate information recognition system of this application further includes a first recognition module, a second recognition module, and a third recognition module.

[0135] The first recognition module is used to obtain the first predicted symbol based on the feature representation vector. The first predicted symbol is the first recognition result of the first object.

[0136] The second recognition module is used to obtain a second prediction symbol based on the first prediction symbol. The second prediction symbol is the recognition result obtained based on the first prediction symbol.

[0137] The third recognition module is used to obtain the third prediction symbol based on the second prediction symbol. The third prediction symbol is a termination symbol used to indicate that the first object recognition is complete.

[0138] In one embodiment, the Transformer-based single / double lane license plate information recognition system of this application further includes a first single lane license plate recognition module, a second single lane license plate recognition module, and a third single lane license plate recognition module.

[0139] The first single-line license plate recognition module is used to directly reshape and transpose the first image data when the first object is of the single-line category, so as to obtain a two-dimensional feature matrix corresponding to the first image data.

[0140] The second single-line license plate recognition module is used to obtain multiple second vectors based on the two-dimensional feature matrix. These multiple second vectors are used to express the features of the first object.

[0141] The third single-line license plate recognition module is used to obtain the feature representation vector of the first object based on multiple second vectors.

[0142] In one embodiment, a computer device is provided, which may be a smart terminal, and its internal structure diagram may be as follows: Figure 14 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a Transformer-based method for recognizing single and double license plate information.

[0143] Those skilled in the art will understand that Figure 14The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0144] In one embodiment, a computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above-described method embodiments. In another embodiment, a computer storage medium stores a computer program, the computer program being executed by a processor to implement the steps in the above-described method embodiments.

[0145] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0146] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for recognizing single and double line license plate information based on a Transformer, characterized in that, The method includes: Acquire first image data, which contains a first object, the first object being the license plate information to be identified; The first image data is used to extract features through a first convolutional neural network to obtain the three-dimensional feature matrix of the first object; The three-dimensional feature matrix is ​​identified by a second convolutional neural network to obtain the single or double row category of the first object; The three-dimensional feature matrix is ​​reshaped and transposed to obtain the two-dimensional feature matrix of the first object; Multiple first vectors are obtained based on the two-dimensional feature matrix, and the multiple first vectors are used to express the features of the first object; The first object's single / double row categories include single row and double row categories, and the method further includes: When the first object is the single-row category, the first image data is directly reshaped and transposed to obtain a two-dimensional feature matrix corresponding to the first image data. Multiple second vectors are obtained based on the two-dimensional feature matrix, and the multiple second vectors are used to express the features of the first object; The feature representation vector of the first object is obtained based on the plurality of second vectors; The step of reshaping and transposing the three-dimensional feature matrix includes: The encoded feature matrix corresponding to the two-dimensional feature matrix is ​​obtained through the encoding module; The decoding module obtains the decoding feature matrix corresponding to the encoded feature matrix. If the input image is a single-line license plate, the initial input to the decoding module is an initial symbol representing "single". If the input is a double-line license plate, the initial input for the first decoding is an initial symbol representing "double". An initial symbol is used as the input to the decoding module. The decoding module consists of N identical decoders. The second attention layer in each decoder will use the encoded feature matrix output by the encoder to obtain the feature representation vector related to the current symbol to be identified. After passing through the feedforward network layer, the currently predicted symbol is output. The predicted output symbol is used as the input of the decoding module to predict the next symbol. The next symbol is predicted repeatedly using the predicted output symbol until a specified termination symbol is predicted. At this point, all the predicted symbols are the license plate information character recognition results.

2. The method of claim 1, wherein, The step of reshaping and transposing the three-dimensional feature matrix further includes: The feature representation vector of the first object is obtained based on the decoded feature matrix. The feature representation vector is composed of the plurality of first vectors and is used to obtain the recognition result of the first object.

3. The method of claim 1, wherein, The encoding module consists of multiple encoders, each encoder having a multi-head self-attention layer and a first feedforward network layer. The method further includes: The multi-head self-attention layer transforms the two-dimensional feature matrix into a corresponding feature representation matrix. The feature representation matrix is ​​output through the first feedforward network layer.

4. The method of claim 3, wherein, The decoding module consists of multiple decoders, each decoder having an attention layer and a second feedforward network layer. The method further includes: The attention layer transforms the feature representation matrix into a corresponding feature representation vector. The feature representation vector is output as the corresponding recognition result through the second feedforward network layer.

5. The method of claim 4, wherein, The identification result of the first object includes a first predicted symbol, a second predicted symbol, and a third predicted symbol. The method further includes: The first predicted symbol is obtained based on the feature representation vector, and the first predicted symbol is the first recognition result of the first object; The second predicted symbol is obtained based on the first predicted symbol, and the second predicted symbol is the recognition result obtained based on the first predicted symbol; The third prediction symbol is obtained based on the second prediction symbol. The third prediction symbol is a termination symbol used to indicate that the first object identification is complete.

6. A single-double line license plate information recognition system based on a Transformer, used to implement the method of any one of claims 1-5, characterized in that, The system includes: An image acquisition module is used to acquire first image data, wherein the first image data contains a first object, and the first object is license plate information to be identified; The feature extraction module is used to extract features from the first image data through a first convolutional neural network to obtain a three-dimensional feature matrix of the first object. The feature recognition module is used to identify the three-dimensional feature matrix through a second convolutional neural network to obtain the single or double row category of the first object; The reshaping and transposing module is used to reshape and transpose the three-dimensional feature matrix to obtain the two-dimensional feature matrix of the first object. The feature representation module is used to obtain multiple first vectors based on the two-dimensional feature matrix, and the multiple first vectors are used to represent the features of the first object.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images