End-to-end Laotian license plate recognition method and device based on image skeleton features

Abstract

The present invention relates to an end-to-end Laos license plate recognition method and device integrating image skeleton features, and belongs to the field of natural language processing. The method of the present invention comprises four parts: construction and preprocessing of Laos license plate image data, construction of an end-to-end Laos license plate recognition model integrating image skeleton features, training of an end-to-end Laos license plate model integrating image skeleton features, and an end-to-end Laos license plate recognition model integrating image skeleton features. An end-to-end Laos license plate recognition device integrating image skeleton features is manufactured according to the functional modularization of these four parts, and information recognition is performed on the input Laos license plate image. The present invention improves the accuracy of Laos license plate image recognition, and effectively solves the problem of easy confusion in the recognition of similar character groups in Laos license plates.

Description

Technical Field

[0001] The present invention relates to an end-to-end Laos license plate recognition method and device integrating image skeleton features, and belongs to the technical field of natural language processing. Background Art

[0002] The end-to-end Lao license plate recognition method that integrates the dual attention mechanism has been able to initially complete the recognition of Laos' special double-row license plate structure. In particular, it uses a classification method to identify the province information of Lao license plates, avoiding the problem of overall recognition errors in the province part of Lao license plates due to the misrecognition of the phonetic symbols and tones of individual characters, thereby improving the recognition accuracy of the province information of Lao license plates. However, since the character information of Lao license plates includes two Lao consonant characters, and there are multiple groups of similar pairs of Lao consonant characters in structure, it is challenging to obtain accurate Lao license plate character information. Figure 1 As shown in the figure, some consonant characters in the Lao license plate image have high similarity in image representation, which is easy to be misidentified during the recognition process, resulting in a decrease in the accuracy of license plate recognition. In real life, since the license plate serial number is an important identification information of the vehicle, once the recognition error occurs, it is easy to bring more serious consequences to the field related to traffic safety. Therefore, how to improve the recognition accuracy of similar characters in Lao license plates is of great significance to the promotion of the practical task of Lao license plate recognition.

[0003] After statistics and analysis of Laotian consonant characters, the 23 Laotian consonant characters that may appear on Laotian license plates were divided into similar categories, resulting in a total of 6 groups of similar characters, such as Figure 2 As shown. At present, the research on Lao text image recognition is still in the exploratory stage. The recognition of similar characters is more about combining contextual word sentence information or post-processing error correction. The character information of Lao license plates is random probability and independent, lacking word-level and sentence-level information. Therefore, it is difficult to use such methods. While sorting out the methods of predecessors in Chinese and English license plate recognition, the present invention proposes an end-to-end Lao license plate recognition method and device that integrates image skeleton features. Summary of the invention

[0004] The present invention provides an end-to-end Laos license plate recognition method and device integrating image skeleton features, which is used to solve the problem that similar character groups on Laos license plates are easily confused; the present invention extracts the image skeleton diagram of license plate characters, which can more accurately distinguish different characters, and is conducive to improving the accuracy and stability of character recognition.

[0005] The technical solution of the present invention is: in the first aspect, the present invention provides an end-to-end Laos license plate recognition method integrating image skeleton features, comprising:

[0006] Step 1: Expand and preprocess the Laos license plate dataset and divide it into evaluation set, training set and test set;

[0007] Step 2, construct an end-to-end Laos license plate recognition model that integrates image skeleton features; the end-to-end Laos license plate recognition model that integrates image skeleton features includes constructing a feature extraction module based on an improved residual network, a skeleton feature fusion module, a Laos license plate province information classification module, and a Laos license plate character information recognition module that are sequentially connected;

[0008] Step 3: Train the constructed end-to-end Laos license plate recognition model that integrates image skeleton features. Specifically, the four modules in step 2 are jointly trained in an end-to-end manner, the skeleton map of the license plate is extracted through the improved residual network, and the skeleton map is integrated into the features of the original image using the cross attention network;

[0009] Step 4: Save the trained end-to-end Laos license plate recognition model that integrates image skeleton features and deploy it on the server to realize information recognition in Laos license plate images.

[0010] Furthermore, in step 1, the expansion and preprocessing include:

[0011] Step 1.1, collect Lao vehicle images and randomly select images to be used as evaluation sets and test sets;

[0012] Step 1.2: After obtaining the Laotian vehicle images, the Laotian license plate dataset is expanded by increasing the proportion of license plate images with similar character groups;

[0013] Step 1.3: Obtain the license plate area of ​​Lao vehicles through the license plate area detection tool; and after image correction technology and scaling technology, scale the pixel height of the image to 32, and the pixel width is distributed in the range of 59-65.

[0014] Furthermore, the specific steps of step 2 are:

[0015] Step 2.1, construct a feature extraction module based on the improved residual network. The network architecture of the feature extraction module based on the improved residual network is to improve the model structure on the basis of the ResNet-34 feature extraction network, introduce the activation function ReLU after the original convolution layer, and increase the nonlinearity of the network; in the original Resnet-34 architecture, replace the convolution layer with an initial convolution kernel size of 7×7 with a convolution kernel size of 3×3, and remove the first pooling layer; it is used to extract the global feature map of the license plate image;

[0016] Step 2.2, construct a skeleton feature fusion module to generate Laos license plate image feature representation and reconstruct the global feature map;

[0017] Step 2.3, positionally encode the Laos license plate image feature representation to obtain provincial feature information, build a Laos license plate provincial information classification module, and obtain provincial information by overall classification of provincial features;

[0018] Step 2.4: Positionally encode the feature representation of the Laos license plate image to obtain character feature information, build a Laos license plate character information recognition module, and use the sequence labeling method to identify the characters on the lower line of the license plate.

[0019] Furthermore, the step 2.2 includes:

[0020] Step 2.2.1: Given an original Laos license plate image First, use the image skeleton generator to generate the license plate image skeleton ; Then X and M are sent to the two improved ResNet-34 feature extraction networks in step 2.1 respectively to generate and The local relationship between the original image and the skeleton image is modeled through the improved ResNet-34 feature extraction network, and two-dimensional position encoding is added to record the position information of the uplink and downlink to enhance the characteristics of the Laos license plate image.

[0021] Step 2.2.2, after obtaining the skeleton feature map of the Lao license plate image, the skeleton feature information of the original image is fused through the skeleton guided encoder; first, the original image features of the improved ResNet-34 feature extraction network are extracted Obtaining global features of license plate images through multi-head self-attention networks ; A multi-head cross attention layer is added after the self-attention layer to convert the global features As key K and value V, the skeleton feature As the query Q is connected with Combined to generate Laos license plate image feature representation ;

[0022] The skeleton-guided encoder consists of N blocks, each of which consists of a self-attention layer, a cross-attention layer, and a feed-forward layer; the skeleton query cross-attention mechanism is expressed as:

[0023]

[0024] Where CA stands for cross attention and σ is the scaling factor.

[0025] Furthermore, the step 2.3 includes:

[0026] Step 2.3.1: Represent the Laotian license plate image features fused with skeleton features Flatten in the channel direction to obtain the flattened Laos license plate image feature representation , Respectively The height, width, and number of channels of and width The product of ; Then, the position coding method based on sine and cosine functions is used for weighting to obtain the provincial location information ,

[0027] Step 2.3.2: With provincial location information After fusion, we get the provincial characteristic information ; Then the vector matrix is ​​flattened and sent to the province classification module composed of two layers of fully connected networks to identify the province information.

[0028] Furthermore, in step 2.3.1, the province location information The specific formula for encoding is as follows:

[0029]

[0030]

[0031] in, , , f represents the dimension of encoding, It represents the value of position p and dimension 2i in the position encoding matrix, Represents the value at position p and dimension 2i+1 in the position encoding matrix.

[0032] Furthermore, the specific steps of step 2.4 are as follows:

[0033] Step 2.4.1: In the Transformer decoder, a multi-head self-attention mechanism is introduced to positionally encode the elements in the character sequence of the Lao license plate to obtain the character position information. ,

[0034] Step 2.4.2: Flatten the Laos license plate image feature representation With character position information After fusion, character feature information is obtained ,Then a Lao license plate transcription module consisting of four Tranformer decoders is used to obtain the final character recognition result.

[0035] Furthermore, the Tranformer decoder includes 4 multi-head attention mechanism layers and a feed-forward neural network; by introducing the multi-head attention mechanism, it can more effectively generate outputs with the flattened Laos license plate image feature representation. Perform association to improve the recognition performance of the Lao license plate recognition model;

[0036] During the transcription process of the Lao license plate, the Lao license plate transcription module adds the encoding vector and visual feature vector of the starting character "BOS" to the character feature information As the overall input; then, through the decoding unit, the first character is obtained, and its encoding vector and visual feature vector are used as the input of the decoding unit for the next character prediction; this process will continue until the predicted character is the end symbol "EOS" or the prediction is stopped when the maximum prediction length is reached.

[0037] In a second aspect, the present invention also provides an end-to-end Laos license plate recognition device that integrates image skeleton features, and the device includes a module for executing the method of the first aspect.

[0038] The beneficial effects of the present invention are as follows: the skeleton image, as a supplement to the original image information of the Laos license plate, can not only provide more information about the shape and structure of the characters, but also help distinguish similar characters. The present invention attempts to extract the image skeleton image of the license plate characters, and finds that the skeleton images of similar characters have relatively obvious differences. Compared with the original image from a visual aspect, there are more different edge features, and the main features of the characters can be more clearly captured, such as the thickness of the strokes, the position and shape of the connection points, etc. This rich feature representation enables the recognition system to distinguish different characters more accurately, which is conducive to improving the accuracy and stability of character recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Examples of similar character information on Lao license plates; (a) is an example of a Lao license plate image, (b) is an example of a province character adhesion image, and (c) is an example of a character similarity pair image;

[0040] Figure 2 This is an example of similar character groups for Laos license plates;

[0041] Figure 3 This is the skeleton diagram of the similar character group of Laos license plate;

[0042] Figure 4 This is a network architecture diagram of an end-to-end Laos license plate recognition device that integrates image skeleton features in the present invention;

[0043] Figure 5 It is a flowchart of the method in the present invention;

[0044] Figure 6This is the analysis result of the sample of the present invention. DETAILED DESCRIPTION

[0045] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that the execution subject of the embodiments of the present invention can be various types of terminals, such as computers, smart phones, tablet computers, wearable devices, personal digital assistants (English: Personal Digital Assistant, abbreviated: PDA), mobile Internet devices (English: Mobile Internet Device, abbreviated: MID) and other terminals that can perform text processing, and the present invention does not impose any restrictions on this.

[0047] like Figure 5 As shown, it is a flowchart of the end-to-end Laos license plate recognition method integrating image skeleton features of the present invention, including:

[0048] Step 1, expand and preprocess the Lao license plate data set, and divide it into an evaluation set, a training set, and a test set; It should be noted that the Lao license plate studied in the present invention is a dual-row license plate, which consists of the full name of the province in the upper row, consonant characters in the lower row, and Arabic numerals. The upper part of the Lao license plate shows the full name of the province in Laos. The Lao characters belong to the horizontal vowel-labeled characters. A Lao character consists of the main consonants, vowel characters, and tone characters, but the upper vowel characters, tone characters, and consonant characters are separated, which belong to the edge features of the character area and are easily disturbed by the image background noise during the recognition process; In addition, the upper provincial characters are arranged relatively compactly, which is prone to character adhesion problems, resulting in the difficulty of accurately segmenting characters using the segmentation method and the poor effect of character feature extraction using the sequence annotation method. The lower part of the Lao license plate consists of two Lao consonant characters and four Arabic numerals. Since there are multiple groups of similar character pairs in the Lao consonant characters, the recognition accuracy of the license plate is greatly affected.

[0049] This embodiment is aimed at the fact that dual-lane license plates are popular in Laos and there is little research on dual-lane license plate recognition at present, and there is a lack of Lao license plate data sets. Therefore, the present invention collects images of import and export vehicles at border checkpoints and Lao highways and vehicle images in Laos, and constructs a Lao license plate data set by annotation and verification. It contains more than 100,000 images in total, and randomly selects 2,000 images as evaluation data sets and test data sets, providing data support for the end-to-end Lao license plate recognition method integrating a two-layer attention network; and pre-processes it as follows:

[0050] Augmentation and preprocessing include:

[0051] Step 1.1, collect Lao vehicle images and randomly select images to be used as evaluation sets and test sets;

[0052] Step 1.2: After obtaining Lao vehicle images, the Lao license plate dataset is expanded by increasing the proportion of license plate images with similar character groups; providing data support for the end-to-end Lao license plate recognition method that integrates image skeleton features;

[0053] Step 1.3: Obtain the license plate area of ​​Lao vehicles through the license plate area detection tool; and after image correction technology and scaling technology, scale the pixel height of the image to 32, and the pixel width is distributed in the range of 59-65.

[0054] Step 2, construct an end-to-end Laos license plate recognition model that integrates image skeleton features; the end-to-end Laos license plate recognition model that integrates image skeleton features includes constructing a feature extraction module based on an improved residual network, a skeleton feature fusion module, a Laos license plate province information classification module, and a Laos license plate character information recognition module that are sequentially connected;

[0055] The specific steps of step 2 are:

[0056] Step 2.1, construct a feature extraction module based on the improved residual network. The network architecture of the feature extraction module based on the improved residual network is to improve the model structure on the basis of the ResNet-34 feature extraction network, introduce the activation function ReLU after the original convolution layer, and increase the nonlinearity of the network; in the original Resnet-34 architecture, replace the convolution layer with an initial convolution kernel size of 7×7 with a convolution kernel size of 3×3, and remove the first pooling layer; it is used to extract the global feature map of the license plate image;

[0057] Step 2.2, construct a skeleton feature fusion module to generate Laos license plate image feature representation and reconstruct the global feature map;

[0058] The step 2.2 includes:

[0059] Step 2.2.1: Given an original Laos license plate image First, use the image skeleton generator to generate the license plate image skeleton ; Then X and M are sent to the two improved ResNet-34 feature extraction networks in step 2.1 respectively to generate and The local relationship between the original image and the skeleton image is modeled through the improved ResNet-34 feature extraction network, and two-dimensional position encoding is added to record the position information of the uplink and downlink to enhance the characteristics of the Laos license plate image.

[0060] Step 2.2.2, after obtaining the skeleton feature map of the Lao license plate image, the skeleton feature information of the original image is fused through the skeleton guided encoder; first, the original image features of the improved ResNet-34 feature extraction network are extracted Obtaining global features of license plate images through multi-head self-attention networks ; A multi-head cross attention layer is added after the self-attention layer to convert the global features As key K and value V, the skeleton feature As the query Q is connected with Combined to generate Laos license plate image feature representation ;

[0061] The skeleton-guided encoder consists of N blocks, each of which consists of a self-attention layer, a cross-attention layer, and a feed-forward layer; the skeleton query cross-attention mechanism is expressed as:

[0062]

[0063] Where CA stands for cross attention and σ is the scaling factor.

[0064] Step 2.3, positionally encode the Laos license plate image feature representation to obtain provincial feature information, build a Laos license plate provincial information classification module, and obtain provincial information by overall classification of provincial features;

[0065] The step 2.3 includes:

[0066] Step 2.3.1: Represent the Laotian license plate image features fused with skeleton features Flatten in the channel direction to obtain the flattened Laos license plate image feature representation , Respectively The height, width, and number of channels of and width The product of ; Then, the position coding method based on sine and cosine functions is used for weighting to obtain the provincial location information ,

[0067] Province location information The specific formula for encoding is as follows:

[0068]

[0069]

[0070] in, , , f represents the dimension of encoding, It represents the value of position p and dimension 2i in the position encoding matrix, Represents the value at position p and dimension 2i+1 in the position encoding matrix.

[0071] Step 2.3.2: With provincial location information After fusion, we get the provincial characteristic information ; Then the vector matrix is ​​flattened and sent to the province classification module composed of two layers of fully connected networks to identify the province information.

[0072] Step 2.4: Positionally encode the feature representation of the Laos license plate image to obtain character feature information, build a Laos license plate character information recognition module, and use the sequence labeling method to identify the characters on the lower line of the license plate.

[0073] The specific steps of step 2.4 are as follows:

[0074] Step 2.4.1: In the Transformer decoder, a multi-head self-attention mechanism is introduced to positionally encode the elements in the character sequence of the Lao license plate to obtain the character position information. ,

[0075] Step 2.4.2: Flatten the Laos license plate image feature representation With character position information After fusion, character feature information is obtained ,Then a Lao license plate transcription module consisting of four Tranformer decoders is used to obtain the final character recognition result.

[0076] The Tranformer decoder contains 4 multi-head attention mechanism layers and a feed-forward neural network; by introducing the multi-head attention mechanism, it can more effectively generate outputs with the flattened Laos license plate image feature representation Perform association to improve the recognition performance of the Lao license plate recognition model;

[0077] During the transcription process of the Lao license plate, the Lao license plate transcription module adds the encoding vector and visual feature vector of the starting character "BOS" to the character feature information As the overall input; then, through the decoding unit, the first character is obtained, and its encoding vector and visual feature vector are used as the input of the decoding unit for the next character prediction; this process will continue until the predicted character is the end symbol "EOS" or the prediction is stopped when the maximum prediction length is reached.

[0078] Step 3: Train the constructed end-to-end Laos license plate recognition model that integrates image skeleton features. Specifically, the four modules in step 2 are jointly trained in an end-to-end manner, the skeleton map of the license plate is extracted through the improved residual network, and the skeleton map is integrated into the features of the original image using the cross attention network;

[0079] In some feasible implementations, the specific steps of step 3 can be performed using the method in the patent application number CN202310880842.X, and the patent name is an end-to-end Laos license plate recognition method and device integrating a double-layer attention network. Specifically:

[0080] Step 3.1: Input the Laos license plate image of size N into the sample As a dataset, given N Laos license plate image input samples The predicted label of the Lao license plate corresponding to the image , recognize the Laos license plate image content as a sequence The probability is defined as ,in , , D is a Lao dictionary, and the dataset is used to train the feature extraction module based on the improved Resnet-34, the skeleton feature fusion module, the Lao license plate province information classification module, and the Lao license plate character information recognition module;

[0081] Step 3.2: Joint training is performed in an end-to-end manner, considering the loss of the Laos license plate provincial information classification module in two aspects of supervision signals Loss of Laotian license plate character information recognition module , the total loss function is a linear combination of these two losses.

[0082] It should be noted that the Lao license plate is a double-row license plate, which consists of the full name of the province in the upper row, consonant characters and Arabic numerals in the lower row. Since the upper vowel characters, tone characters and consonant characters in the upper provincial information are separated, they belong to the edge features of the character area, which are easily disturbed by the image background noise during the recognition process. In addition, the characters in the upper provincial information are arranged relatively compactly, which easily leads to character adhesion problems, resulting in the difficulty of accurately segmenting characters using the segmentation method and the poor effect of character feature extraction using the sequence labeling method. The lower character information has a fixed arrangement rule and a large character spacing, but there are multiple groups of easily confused similar pairs, which have a great impact on the recognition accuracy of the license plate. Therefore, according to the different layout styles of the upper and lower rows of the Lao license plate, the Lao license plate province information classification module and the Lao license plate character information recognition module are used to obtain different information of the Lao license plate respectively, and provide the accuracy of Lao license plate recognition.

[0083] In some feasible implementations, the specific steps of step 3.2 are:

[0084] Step 3.2.1. In the Laos license plate provincial information classification module, a cross entropy loss is performed between the 18-dimensional vector representing the provincial classification probability of the 18 provinces in Laos obtained after two layers of fully connected networks and the provincial label information corresponding to the license plate to obtain the loss of the Laos license plate provincial information classification module. , the specific formula is as follows:

[0085]

[0086] in, Indicates province label information, which can be 0 or 1. The new flattened provincial feature map;

[0087] Step 3.2.2: The essence of Laos license plate image character recognition is to perform multi-classification of feature sequences. In order to ensure that the predicted distribution of the recognition network is close to the real label distribution during network training, the recognition part of the Laos license plate character information recognition module adopts cross entropy loss as the target optimization function. In the Laos license plate character information recognition module, the Softmax-based classifier is obtained. Character prediction for the moment decoder Calculate the cross entropy loss with the label information to get the loss of the Laos license plate character information recognition module as follows:

[0088]

[0089] in, Represented as the input Laos license plate image, Represents the model parameters of the current Laos license plate character information recognition module, Represented as the license plate image The true label corresponding to the feature sequence;

[0090] Step 3.2.3: Loss of the classification module based on the province information of Laos license plate Loss of Laotian license plate character information recognition module Calculate the total loss of the end-to-end Laotian license plate recognition method that integrates image skeleton features :

[0091] To balance the control parameters of the two loss items of the Laos license plate province information classification module and the Laos license plate character information recognition module;

[0092] By calculating the final minimum loss value, the parameters and weights obtained in the training network are fed back to the neural network model to obtain the final model.

[0093] Step 4: Save the trained end-to-end Laos license plate recognition model that integrates image skeleton features and deploy it on the server to realize information recognition in Laos license plate images.

[0094] In some feasible implementations, the specific steps of step 4 can be performed using the method in the patent application number CN202310880842.X, and the patent name is an end-to-end Laos license plate recognition method and device integrating a double-layer attention network. Specifically:

[0095] Step 4.1: First, define the height of the input image of the trained end-to-end Laos license plate recognition model with fused image skeleton features as 32, and the maximum width as , for input width less than The image is filled with white pixels, and the size of the model batch processing is the number of input images, so as to achieve parallel calculation of the model on the input images and improve the recognition speed;

[0096] Step 4.2, save the model as a ".pth" file, and load the model into memory through the Flask framework to avoid frequent model loading caused by request results and improve the running speed of the recognition model;

[0097] Step 4.3: Use the Flask framework to deploy the model as an API interface, thereby realizing the function of multiple concurrent requests on the Web port;

[0098] Step 4.4: On the Web side, call the Laos license plate recognition model deployed on the server side to test the input Laos license plate image, and then obtain the recognition result of the Laos license plate image and display it on the front-end interface.

[0099] The following is an embodiment of the device of the present invention. The embodiment of the device of the present invention is used to execute the method implemented by the first embodiment of the method of the present invention. For ease of explanation, only the relevant parts of the embodiment of the present invention are shown. For the specific parts not disclosed, please refer to the first embodiment of the present invention.

[0100] Expansion and preprocessing module: used to expand and preprocess the Lao license plate dataset and divide it into evaluation set, training set and test set;

[0101] Construction module: used to construct an end-to-end Laos license plate recognition model that integrates image skeleton features; the end-to-end Laos license plate recognition model that integrates image skeleton features includes constructing a feature extraction module based on an improved residual network, a skeleton feature fusion module, a Laos license plate province information classification module, and a Laos license plate character information recognition module that are sequentially connected;

[0102] Training module: used to train the constructed end-to-end Laos license plate recognition model that integrates image skeleton features. Specifically, the four modules in step 2 are jointly trained in an end-to-end manner, the skeleton map of the license plate is extracted through the improved residual network, and the skeleton map is integrated into the features of the original image using the cross attention network;

[0103] Recognition module: It is used to save the trained end-to-end Lao license plate recognition model that integrates image skeleton features and deploy it on the server to realize information recognition in Lao license plate images.

[0104] In a feasible implementation manner, the expansion and preprocessing module is specifically used to implement the following functions:

[0105] Collect Lao vehicle images and randomly select images to be used as evaluation and test sets;

[0106] After obtaining Lao vehicle images, the Lao license plate dataset is expanded by increasing the proportion of license plate images with similar character groups;

[0107] The license plate area of ​​Lao vehicles is obtained through the license plate area detection tool; and through image correction technology and scaling technology, the pixel height of the image is scaled to 32, and the pixel width is distributed in the range of 59-65.

[0108] In a feasible implementation, the construction module is specifically used to construct a feature extraction module based on an improved residual network, a skeleton feature fusion module, a Laos license plate province information classification module, and a Laos license plate character information recognition module that are connected in sequence;

[0109] It is used to construct a feature extraction module based on an improved residual network. The network architecture of the feature extraction module based on the improved residual network is to improve the model structure on the basis of the ResNet-34 feature extraction network. The activation function ReLU is introduced after the original convolution layer to increase the nonlinearity of the network. In the original Resnet-34 architecture, the convolution layer with an initial convolution kernel size of 7×7 is replaced by a convolution layer with a convolution kernel size of 3×3, and the first pooling layer is removed. It is used to extract the global feature map of the license plate image.

[0110] Used to build a skeleton feature fusion module, which is used to generate Laos license plate image feature representation and reconstruct the global feature map;

[0111] It is used to perform position encoding on the Laos license plate image feature representation, obtain provincial feature information, build a Laos license plate provincial information classification module, and obtain provincial information by overall classification of provincial features;

[0112] It is used to perform position encoding on the feature representation of Laos license plate images, obtain character feature information, build a Laos license plate character information recognition module, and use sequence labeling methods to identify the characters on the lower line of the license plate.

[0113] In order to illustrate the effect of the present invention, the present invention conducted the following experiment, and the selected Laos license plate image dataset is shown below.

[0114] Training dataset: This dataset contains 100,000 images of Lao license plates in real scenes;

[0115] Test dataset: This dataset contains 2,000 images of Lao license plates in real scenes;

[0116] Evaluation dataset: This dataset contains 2,000 real-life Lao license plate images;

[0117] The neural network architecture of the present invention is developed and designed based on the Pytorch framework, and the configuration of the experimental server is Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz and NVIDIA Corporation GP100GL GPU.

[0118] The experiment uses Lao license plate recognition accuracy (RA) as the evaluation indicator, as shown in the formula:

[0119]

[0120] in, The accuracy of Lao license plate image recognition, the number of Lao license plate information correctly identified, and the total number of Lao license plates.

[0121] In order to verify the effectiveness of the end-to-end Laos license plate recognition method that integrates image skeleton features, the present invention conducts experimental analysis on the Laos license plate image dataset. To ensure fairness, uniform experimental conditions are set for all comparative experiments. The selected optimizer is Adam, the initial learning rate is 1, and the CosineAnnealing strategy is used during training. The learning rate is dynamically transformed based on the cosine function to ensure that the network's objective function has a smaller learning rate when it is close to the optimal solution; the batch size of the model training is set to 120, the training step is set to 300000, and the training epoch is 10. The experimental results select the highest accuracy in the evaluation.

[0122] Experiment 1: Main experimental results

[0123] ShallowCNN+CrossAttn+Transformer: Due to the particularity of the size of Laotian license plates, which is different from the current mainstream image size used for image recognition, this paper attempts to use a latent image feature extraction network to obtain Laotian image features. ShallowCNN consists of a two-layer convolutional layer network, and the skeleton feature extractor also makes corresponding changes to the original image. Finally, the final output result is obtained through Transformer decoding.

[0124] VGG-16+CrossAttn+Transformer: The VGG-16 network is used as the Laotian license plate image feature and skeleton image feature extractor, and then the two features are fused through the cross attention mechanism and sent to the Transformer decoding to obtain the final output result.

[0125] ResNet-34+CrossAttn+Attn: ResNet-34 is used to extract the Lao license plate image features and skeleton map features, and then fused through the cross-attention mechanism. Finally, the feature decoding is performed through the Attention mechanism to obtain the results.

[0126] ResNet-18+CrossAttn+Transformer: Based on the method of the present invention, the image feature extraction module of the network is replaced with an improved ResNet-18 feature extraction network adapted to the Laotian license plate image size input.

[0127] Table 1 Comparison of the results of the method of the present invention with other methods

[0128]

[0129] As shown in Table 1, the method of the present invention has the best recognition effect on the Lao license plate dataset, with an accuracy rate of 94.7%. From the experimental results, the method of the present invention has shown obvious advantages over the ShallowCNN+CrossAttn+Transformer method in the Lao license plate image recognition task, improving the accuracy rate by nearly 3 percentage points. This shows that in this specific task, the use of a deep convolutional feature extraction network can more effectively capture the key features of Lao license plate images. The deep convolutional network can gradually extract the abstract features of the image through multi-level convolution operations, allowing the network to better understand the complex nature of the Lao license plate image.

[0130] Complex structure and character features. This layer-by-layer extraction method has advantages in processing the diversity and similarity of Lao characters, which helps to distinguish different characters more accurately and improve the overall recognition accuracy. In addition, we conducted experiments on convolutional extraction networks of different depths. Among them, compared with the VGG-16 convolutional feature extraction network, the method of the present invention improved by 1.6 percentage points. The network structure of ResNet-34 is deeper than that of VGG-16, and the residual learning mechanism is introduced, which helps to better capture the abstract features of the image. Compared with the ResNet-18 convolutional feature extraction network, the method of the present invention improved by 0.5 percentage points. For the complex image structure of Lao license plates, the ResNet-34 convolutional feature extraction network has a deeper network structure than the ResNet-18 network, and can more effectively extract features, which improves the model's discrimination of similar characters and overall recognition performance. Compared with traditional image sequence recognition methods, the method of the present invention effectively compensates for some limitations of traditional image sequence recognition methods by fusing image skeleton information, and improves the accuracy by 2 percentage points. By integrating the image skeleton information, we can better differentiate the similarity between similar characters and differentiate the feature representations of similar characters. At the same time, the image skeleton information more comprehensively contains the overall structure of the license plate image and the character arrangement rules, thereby improving the accuracy and robustness of recognition. Compared with the method using the Attention mechanism decoding, the Transformer decoding architecture adopted by the present invention achieved higher performance, an increase of 0.8 percentage points. This result shows that in the Laos license plate image recognition task, the use of the Transformer decoding architecture can more effectively improve the performance of the model compared to the traditional Attention mechanism decoding. The introduction of the Transformer decoding architecture brings more flexible and global feature modeling, enabling the model to better capture the long-distance dependencies between characters in the Laos license plate image. Compared with the traditional Attention mechanism, the Transformer's self-attention mechanism and position encoding design make the model more advantageous in character recognition tasks, thereby achieving higher accuracy.

[0131] Experiment 2: Ablation experiment

[0132] In order to verify the effectiveness of the skeleton feature fusion module, the province and character position information, and the self-attention mechanism module, the present invention conducts ablation experiments on the above three modules respectively, and the experimental results are shown in Table 2. ("×" indicates not applied, and "√" indicates applied)

[0133] Table 2 Ablation experiments of skeleton feature fusion module, province and character position information, and self-attention mechanism module

[0134]

[0135] From the experimental results, it can be seen that the overall recognition accuracy of the model increased by 2.1 percentage points after the skeleton feature fusion module was applied, indicating that the integration of image skeleton information has a good effect on improving the performance of the Lao license plate recognition model. The introduction of the skeleton feature fusion module can better handle similar Lao characters, enrich the vector representation of similar Lao characters by differentiating the vector representation of similar characters, and improve the overall performance of the Lao license plate recognition model. By incorporating the province and character position information into the model, the overall recognition accuracy of the model increased by 0.7 percentage points, indicating that the use of province and character position information enables the model to more fully distinguish the uplink and downlink information of the Lao license plate, reduce the influence of noise features, and thus more accurately identify the Lao license plate information. The introduction of the self-attention mechanism after the original image feature extraction increased the accuracy of the model by 0.3 percentage points. This result shows that the self-attention mechanism has a certain positive impact on the performance of the Lao license plate recognition model. The self-attention mechanism can more flexibly capture the association between different positions in the image, which helps the model better understand the relative position relationship of characters in the license plate image. It can more effectively enrich the image feature representation when processing license plate images with complex character arrangements and a large number of similar characters, thereby improving the model recognition accuracy.

[0136] Experiment 3: Ablation experiment comparison of the number of multi-head cross attention heads

[0137] In order to verify the influence of the number of multi-head cross attention heads on the Lao license plate recognition model, the present invention designs a related ablation experiment, and the experimental results are shown in Table 3. Among them, when the number of attention heads is 6, the recognition model has the best effect, and the recognition accuracy reaches 94.7%. When the number of attention heads is 2 and 4 respectively, the accuracy of the recognition model decreases. This may be because the lower number of attention heads limits the model's ability to model complex features in Lao license plate images, resulting in the loss of some key information when processing license plate images.

[0138] Table 3 The impact of the number of multi-head cross attention heads on recognition

[0139]

[0140] Experiment 4: Ablation experiment comparison of the number of Tranformer decoding units

[0141] In order to verify the influence of the number of Transformer decoding units on the Laos license plate image recognition model, the present invention conducted an ablation experiment and set the number of attention heads in the Transformer decoder to 6. The experimental results are shown in Table 4, where the performance of the recognition model is optimal when the number of Transformer decoder units is 4. This result shows that when the number of decoding units is 4, the model learns and captures the key features in the image more effectively, thereby achieving more accurate character recognition.

[0142] Table 4 shows the effect of the number of Tranformer decoding units on recognition

[0143]

[0144] The present invention attempts to extract the image skeleton of Laotian consonant characters and finds that the skeletons of similar characters have obvious differences. Compared with the original image, there are more different edge features from the visual aspect, which is conducive to improving the accuracy of character recognition. The specific skeleton of the Laotian license plate similar character group is as follows: Figure 3 shown.

[0145] Figure 6 The example of Laos license plate image recognition of the present invention is given. The second column in the table is the Laos license plate image, the third column is the skeleton diagram generated by the image in the model, and the fourth column is the recognition result output by the model. It can be seen from Examples 1 and 2, 3 and 4 that the two groups of similar characters in Laos The difference in the skeleton image is greater than that in the original image, and the arrangement and connection between pixels are also different. At the same time, the skeleton images of the same characters are similar and there will not be too much noise. As a supplement to the original image information of the Laotian license plate, the skeleton image can not only provide more information about the shape and structure of the characters, but also help distinguish similar characters. By extracting the skeleton image of the license plate characters, we can more clearly capture the main features of the characters, such as the thickness of the strokes, the location and shape of the connection points, etc. This rich feature representation enables the recognition system to distinguish different characters more accurately, thereby improving the accuracy and stability of the overall recognition.

[0146] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An end-to-end Laotian license plate recognition method integrating image skeleton features is characterized by: include: Step 1: Expand and preprocess the Laos license plate dataset and divide it into evaluation set, training set and test set; Step 2, construct an end-to-end Laos license plate recognition model that integrates image skeleton features; the end-to-end Laos license plate recognition model that integrates image skeleton features includes constructing a feature extraction module based on an improved residual network, a skeleton feature fusion module, a Laos license plate province information classification module, and a Laos license plate character information recognition module that are sequentially connected; Step 3: Train the constructed end-to-end Laos license plate recognition model that integrates image skeleton features. Specifically, the four modules in step 2 are jointly trained in an end-to-end manner, the skeleton map of the license plate is extracted through the improved residual network, and the skeleton map is integrated into the features of the original image using the cross attention network; Step 4: Save the trained end-to-end Laos license plate recognition model that integrates image skeleton features and deploy it on the server to realize information recognition in Laos license plate images; The specific steps of step 2 are: Step 2.1, construct a feature extraction module based on the improved residual network. The network architecture of the feature extraction module based on the improved residual network is to improve the model structure on the basis of the ResNet-34 feature extraction network, introduce the activation function ReLU after the original convolution layer, and increase the nonlinearity of the network; in the original Resnet-34 architecture, replace the convolution layer with an initial convolution kernel size of 7×7 with a convolution kernel size of 3×3, and remove the first pooling layer; it is used to extract the global feature map of the license plate image; Step 2.2, construct a skeleton feature fusion module to generate Laos license plate image feature representation and reconstruct the global feature map; Step 2.3, positionally encode the Laos license plate image feature representation to obtain provincial feature information, build a Laos license plate provincial information classification module, and obtain provincial information by overall classification of provincial features; Step 2.4, position encoding the Laos license plate image feature representation to obtain character feature information, construct a Laos license plate character information recognition module, and use a sequence labeling method to identify the license plate downlink characters; The step 2.2 includes: Step 2.2.1: Given an original Laos license plate image First, use the image skeleton generator to generate the license plate image skeleton ; Then X and M are sent to the two improved ResNet-34 feature extraction networks in step 2.1 respectively to generate and The local relationship between the original image and the skeleton image is modeled through the improved ResNet-34 feature extraction network, and two-dimensional position encoding is added to record the position information of the uplink and downlink to enhance the characteristics of the Laos license plate image. Step 2.2.2, after obtaining the skeleton feature map of the Lao license plate image, the skeleton feature information of the original image is fused through the skeleton guided encoder; first, the original image features of the improved ResNet-34 feature extraction network are extracted Obtaining global features of license plate images through multi-head self-attention networks ; A multi-head cross attention layer is added after the self-attention layer to convert the global features As key K and value V, the skeleton feature As the query Q is connected with Combined to generate Laos license plate image feature representation ; The skeleton-guided encoder consists of N blocks, each of which consists of a self-attention layer, a cross-attention layer, and a feed-forward layer; the skeleton query cross-attention mechanism is expressed as: ； Where CA represents cross attention and σ is the scaling factor; The step 2.3 includes: Step 2.3.1: Represent the Laotian license plate image features fused with skeleton features Flatten in the channel direction to obtain the flattened Laos license plate image feature representation , Respectively The height, width, and number of channels of and width The product of ; Then, the position coding method based on sine and cosine functions is used for weighting to obtain the provincial location information , Step 2.3.2: With provincial location information After fusion, we get the provincial characteristic information ; Then the vector matrix is ​​flattened and sent to the province classification module consisting of a two-layer fully connected network to identify the province information; In step 2.3.1, the province location information The specific formula for encoding is as follows: ； ； in, , , f represents the dimension of encoding, It represents the value of position p and dimension 2i in the position encoding matrix, Represents the value of position p and dimension 2i+1 in the position encoding matrix; The specific steps of step 2.4 are as follows: Step 2.4.1: In the Transformer decoder, a multi-head self-attention mechanism is introduced to positionally encode the elements in the character sequence of the Lao license plate to obtain the character position information. ; Step 2.4.2: Flatten the Laos license plate image feature representation With character position information After fusion, character feature information is obtained ,Then a Lao license plate transcription module consisting of four Tranformer decoders is used to obtain the final character recognition result; The Tranformer decoder contains 4 multi-head attention mechanism layers and a feed-forward neural network; by introducing the multi-head attention mechanism, it can more effectively generate outputs with the flattened Laos license plate image feature representation Perform association to improve the recognition performance of the Lao license plate recognition model; During the transcription process of the Lao license plate, the Lao license plate transcription module adds the encoding vector and visual feature vector of the starting character "BOS" to the character feature information As the overall input; then, through the decoding unit, the first character is obtained, and its encoding vector and visual feature vector are used as the input of the decoding unit for the next character prediction; this process will continue until the predicted character is the end symbol "EOS" or the prediction is stopped when the maximum prediction length is reached.

2. The end-to-end Laos license plate recognition method integrating image skeleton features according to claim 1 is characterized in that: In step 1, the expansion and preprocessing include: Step 1.1, collect Lao vehicle images and randomly select images to be used as evaluation sets and test sets; Step 1.2: After obtaining the Laotian vehicle images, the Laotian license plate dataset is expanded by increasing the proportion of license plate images with similar character groups; Step 1.3: Obtain the license plate area of ​​Lao vehicles through the license plate area detection tool; and after image correction technology and scaling technology, scale the pixel height of the image to 32, and the pixel width is distributed in the range of 59-65.

3. The end-to-end Laos license plate recognition device integrating image skeleton features is characterized by: The method comprises a module for executing the method according to any one of claims 1 to 2.

Images