A low-resource license plate recognition method and device based on feature shuffling

Abstract

The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a method and device for low-resource license plate recognition based on feature shuffling, the method comprising: obtaining a sample data set of license plate images; constructing an initial license plate recognition model; the license plate recognition model comprising: a global perception encoder, a feature shuffling module and a decoder module; inputting the sample data set into the initial license plate recognition model, extracting features of the sample data through the global perception encoder to obtain global visual features, shuffling the initial reading order through the feature shuffling module to obtain shuffled spatial attention features; inputting the spatial attention features into the decoder to obtain a prediction result; according to the prediction result, using a minimization loss function to train a model to obtain a trained license plate recognition model; obtaining license plate image data to be recognized and inputting it into the trained license plate recognition model to obtain a predicted license plate number. The present invention can improve the performance of low-resource license plate recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and computer vision technology, and in particular to a method and device for recognizing low-resource license plates based on feature shuffling. Background Art

[0002] License plate recognition technology uses computer vision and image processing technology to automatically identify and extract vehicle license plate information. It is a technology widely used in the field of modern transportation. License plate recognition methods in the field of computer vision take cropped license plate images as input and license plate numbers as output. In the real world, it is very difficult to obtain large-scale annotated datasets due to the high cost of manual annotation. Small-scale datasets have the characteristics of small sample size and limited variation in character arrangement, and the probability of occurrence of character categories in certain character positions in the training set often presents a long-tail distribution. Many existing license plate recognition methods based on sequence recognition perform well on large-scale datasets, but perform poorly when applied to small-scale datasets. In essence, they face the challenge of low-resource license plate recognition. Given a sufficient number of training samples, the character categories at each character position tend to be uniformly distributed; the probability of occurrence of all character categories at each character position is roughly equal. However, observations show that training samples in small-scale datasets are constrained and characterized by limited variation in character arrangement. Usually, the probability of occurrence of character categories in certain character positions presents a long-tail distribution, where only the head category has a higher probability of presentation in certain character positions, and the tail category has a lower probability of appearance, resulting in a higher number of errors in the recognition results.

[0003] At present, the methods for low-resource license plate recognition are mainly character detection methods and image generation methods. The character detection method can directly detect and recognize each character independently without considering its position in the license plate string. However, character detection basically bypasses the problems caused by the long-tail distribution of character categories, and the character detection method requires character-level annotations, which wastes manpower and resources. Although the image generation method can enhance low-resource license plate recognition by generating a large number of different synthetic license plates, it requires a large number of samples for training, which increases computing resources and training time. Summary of the invention

[0004] In order to solve the technical problems in the prior art that model training consumes a lot of computing power and human resources, and gives priority to the head class of the long-tail distribution of character categories while ignoring the tail class, the embodiment of the present invention provides a low-resource license plate recognition method and device based on feature shuffling. The technical solution is as follows:

[0005] On the one hand, a low-resource license plate recognition method based on feature shuffling is provided, the method is implemented by a low-resource license plate recognition device based on feature shuffling, and the method comprises:

[0006] S1. Obtain a sample data set of license plate images; the sample data set includes: a plurality of license plate images and license plate annotation information;

[0007] S2. constructing an initial license plate recognition model; the license plate recognition model comprises: a global perception encoder, a feature shuffling module and a decoder module;

[0008] S3, input the multiple license plate images into the initial license plate recognition model, extract features from the multiple license plate images through the global perception encoder to obtain global visual features; shuffle the preset initial reading order through the feature shuffling module to obtain the shuffled reading order; process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; shuffle the license plate annotation information according to the shuffled reading order to obtain the shuffled annotation information; input the shuffled spatial attention features into the decoder module for prediction to obtain the prediction results; according to the prediction results and the shuffled annotation information, train the initial license plate recognition model by minimizing the cross entropy loss function to obtain the trained license plate recognition model;

[0009] S4. Obtain the license plate image data to be identified; input the license plate image data to be identified into the trained license plate recognition model to obtain the predicted license plate number.

[0010] Optionally, the global perception encoder is used to extract global visual features of input data;

[0011] The global perception encoder includes: a global perception network and a local perception network; the local perception network is used to extract local visual features; the global perception network perceives the overall character layout in the license plate based on the local visual features, and is used to extract deep features;

[0012] The feature shuffling module is used to shuffle the preset initial reading order to obtain the shuffled reading order, and process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features;

[0013] The feature shuffling module includes: a reading order shuffling module, an embedding layer and a linear layer;

[0014] The decoder module is used to process the shuffled spatial attention features.

[0015] Optionally, before the step of shuffling the preset initial reading order by the feature shuffling module in S3 to obtain the shuffled reading order, the step further includes:

[0016] Constructing a nested dictionary of probability distribution of character categories according to the sample data set, and formulating a shuffling strategy according to the nested dictionary;

[0017] The specific implementation process of the shuffling strategy includes:

[0018] Create a nested dictionary of the probability distribution of character categories based on the training set;

[0019] Among them, the nested dictionary is represented as:

[0020] Where i represents the i-th character position among the T character positions of the license plate number; cls j represents the jth category; represents the probability of the jth category appearing at the i-th position in the training set; CP represents the probability of the character category appearing;

[0021] The nested dictionary includes: an outer dictionary and an inner dictionary;

[0022] The keys of the outer dictionary correspond to T=8 character positions; the values ​​of the outer dictionary are the inner dictionary.

[0023] Each key of the internal dictionary corresponds to NC=37 character classes; the value of the internal dictionary represents the probability of each class appearing at each position in the training set;

[0024] Among them, the label information of the license plate of one of the samples in the known training set is GT = "c1c2...c j , c T ″, wherein the preset initial reading order RO = [O1, O2, ..., O j , O T ], where T represents the maximum length of the license plate annotation information in the sample data set;

[0025] Among them, the probability distribution of known character categories in is the jth category cls in the training set j The probability of occurrence at position i, where j represents the jth character category and i represents the ith character position; the number of character categories NC = 37;

[0026] For character position i, a candidate set of reading order is constructed based on the annotation information GT of the current training sample license plate; characters that meet the following conditions are selected from the annotation information GT of the current license plate, including: character c j The probability of the corresponding class at position i in the training set The corresponding reading order O j Put it into the reading order candidate set; calculate the possibility of selecting reading order O from the reading order candidate set and placing it at position i SelectProb: If the character c at the current position ii The probability of occurrence of the corresponding class at position i in the training set is CP[i][c i ] is greater than Then SelectProb is c i The probability of occurrence CP[i][c i ] is based on the ratio of the maximum occurrence probability of all classes at position i, adjusted upward, with a base of 0.65 as an empirical value and a maximum upper limit of 0.9; if c i The probability of occurrence CP[i][c i ] is less than or equal to Then SelectProb will be adjusted upward with 0.1 as the base, and the maximum upper limit is 0.35; 0.1 is the empirical value;

[0027] A random number between 0 and 1 is randomly selected and compared with the SelectProb value; if the random number is less than the SelectProb value, a reading order O is randomly selected from the reading order candidate set as the alternative reading order at the current position i; if the random number is equal to or greater than the SelectProb value, the reading order at the current position i remains the current character c. i The corresponding initial reading order O i ; So far, the replacement of the reading order at the current position i is completed; and so on, the replacement of the reading order of all positions from position 1 to position T is completed. The above process completes a reading order shuffling process.

[0028] Optionally, the step of inputting the multiple license plate images into the initial license plate recognition model, extracting features from the multiple license plate images through the global perception encoder, and obtaining global visual features includes:

[0029] S31, inputting the multiple license plate images into the initial license plate recognition model, extracting features from the multiple license plate images through a local perception network, and obtaining local visual features;

[0030] S32, inputting the local visual features into the global perception network, processing the local visual features through the global perception network, and obtaining the global visual features.

[0031] Optionally, the step S3 shuffles the preset initial reading order through a feature shuffling module to obtain a shuffled reading order; and processes the global visual features according to the shuffled reading order to obtain a shuffled spatial attention feature, including:

[0032] S33, using a reading order shuffling module, using the probability distribution of character categories in a sample data set as a basis, shuffling a preset initial reading order to obtain a shuffled reading order; using an attention mechanism, using the shuffled reading order as a query, and using global visual features as a key; mapping the key and the query to the same embedding space according to a linear layer and an embedding layer; performing operations on the query and the key according to the linear layer to obtain a spatial attention weight;

[0033] S34. Perform spatial attention on each character in the global visual feature according to the spatial attention weight to obtain a shuffled spatial attention feature.

[0034] Optionally, the step S4 of inputting the license plate image data to be recognized into a trained license plate recognition model to obtain a predicted license plate number includes:

[0035] S41, inputting the license plate image data to be recognized into the trained license plate recognition model, and obtaining global visual features through the global perception encoder;

[0036] S42, inputting the global visual features into the feature shuffling module, processing the global visual features according to a preset initial reading order to obtain spatial attention features; predicting the spatial attention features through the decoder module to obtain a predicted license plate number.

[0037] On the other hand, a low-resource license plate recognition device based on feature shuffling is provided, and the device is applied to a low-resource license plate recognition method based on feature shuffling, and the device comprises:

[0038] An acquisition unit is used to acquire a sample data set of license plate images; the sample data set includes: a plurality of license plate images and labeling information of the license plates;

[0039] A construction unit, used to construct an initial license plate recognition model; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module;

[0040] A training unit is used to input the multiple license plate images into an initial license plate recognition model, extract features from the multiple license plate images through the global perception encoder to obtain global visual features; shuffle the preset initial reading order through the feature shuffling module to obtain the shuffled reading order; process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; shuffle the license plate annotation information according to the shuffled reading order to obtain the shuffled annotation information; input the shuffled spatial attention features into the decoder module for prediction to obtain the prediction results; train the initial license plate recognition model by minimizing the cross entropy loss function according to the prediction results and the shuffled annotation information to obtain the trained license plate recognition model;

[0041] The prediction unit is used to obtain the license plate image data to be recognized; the license plate image data to be recognized is input into the trained license plate recognition model to obtain the predicted license plate number.

[0042] Optionally, the construction unit is used to:

[0043] The global perception encoder includes: a global perception network and a local perception network; the local perception network is used to extract local visual features; the global perception network perceives the overall character layout in the license plate based on the local visual features, and is used to extract deep features;

[0044] The feature shuffling module is used to shuffle the preset initial reading order to obtain the shuffled reading order, and process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features;

[0045] The feature shuffling module includes: a reading order shuffling module, an embedding layer and a linear layer;

[0046] The decoder module is used to process the shuffled spatial attention features.

[0047] Optionally, before the step of shuffling the preset initial reading order by the feature shuffling module to obtain the shuffled reading order, the method further includes:

[0048] Constructing a nested dictionary of probability distribution of character categories according to the sample data set, and formulating a shuffling strategy according to the nested dictionary;

[0049] The specific implementation process of the shuffling strategy includes:

[0050] Create a nested dictionary of the probability distribution of character categories based on the training set;

[0051] Among them, the nested dictionary is represented as:

[0052] Where i represents the i-th character position among the T character positions of the license plate number; cls j represents the jth category; represents the probability of the jth category appearing at the i-th position in the training set; CP represents the probability of the character category appearing;

[0053] The nested dictionary includes: an outer dictionary and an inner dictionary;

[0054] The keys of the outer dictionary correspond to T=8 character positions; the values ​​of the outer dictionary are the inner dictionary.

[0055] Each key of the internal dictionary corresponds to NC=37 character classes; the value of the internal dictionary represents the probability of each class appearing at each position in the training set;

[0056] Among them, the label information of the license plate of one of the samples in the known training set is GT = "c1c2...c j , c T ″, wherein the preset initial reading order RO = [O1, O2, ..., O j , O T ], where T represents the maximum length of the license plate annotation information in the sample data set;

[0057] Among them, the probability distribution of known character categories in is the jth category cls in the training set j The probability of occurrence at position i, where j represents the jth character category and i represents the ith character position; the number of character categories NC = 37;

[0058] For character position i, a candidate set of reading order is constructed based on the annotation information GT of the current training sample license plate; characters that meet the following conditions are selected from the annotation information GT of the current license plate, including: character c j The probability of the corresponding class at position i in the training set The corresponding reading order O j Put it into the reading order candidate set; calculate the possibility of selecting reading order O from the reading order candidate set and placing it at position i SelectProb: If the character c at the current position i i The probability of occurrence of the corresponding class at position i in the training set is CP[i][c i ] is greater than Then SelectProb is c i The probability of occurrence CP[i][c i] is based on the ratio of the maximum occurrence probability of all classes at position i, adjusted upward, with a base of 0.65 as an empirical value and a maximum upper limit of 0.9; if c i The probability of occurrence CP[i][c i ] is less than or equal to Then SelectProb will be adjusted upward with 0.1 as the base, and the maximum upper limit is 0.35; 0.1 is the empirical value;

[0059] A random number between 0 and 1 is randomly selected and compared with the SelectProb value; if the random number is less than the SelectProb value, a reading order O is randomly selected from the reading order candidate set as the alternative reading order at the current position i; if the random number is equal to or greater than the SelectProb value, the reading order at the current position i remains the current character c. i The corresponding initial reading order O i ; So far, the replacement of the reading order at the current position i is completed; and so on, the replacement of the reading order of all positions from position 1 to position T is completed. The above process completes a reading order shuffling process.

[0060] Optionally, the multiple license plate images are input into the initial license plate recognition model, and feature extraction is performed on the multiple license plate images by the global perception encoder to obtain global visual features, including:

[0061] Inputting the multiple license plate images into the initial license plate recognition model, extracting features from the multiple license plate images through a local perception network to obtain local visual features;

[0062] The local visual features are input into the global perception network, and the local visual features are processed by the global perception network to obtain the global visual features.

[0063] Optionally, the feature shuffling module shuffles the preset initial reading order to obtain a shuffled reading order; and processes the global visual features according to the shuffled reading order to obtain a shuffled spatial attention feature, including:

[0064] Through the reading order shuffling module, the character category occurrence probability distribution of the sample data set is used as the basis to shuffle the preset initial reading order to obtain the shuffled reading order; the attention mechanism is used to take the shuffled reading order as the query and the global visual feature as the key; according to the linear layer and the embedding layer, the key and the query are mapped to the same embedding space; according to the linear layer, the query and the key are operated to obtain the spatial attention weight;

[0065] According to the spatial attention weight, spatial attention is performed on each character in the global visual feature to obtain the shuffled spatial attention feature.

[0066] Optionally, the prediction unit is used to:

[0067] The license plate image data to be recognized is input into the trained license plate recognition model, and the global visual features are obtained through the global perception encoder;

[0068] The global visual features are input into the feature shuffling module, and the global visual features are processed according to a preset initial reading order to obtain spatial attention features; the spatial attention features are predicted by the decoder module to obtain a predicted license plate number.

[0069] On the other hand, a low-resource license plate recognition device based on feature shuffling is provided, and the low-resource license plate recognition device based on feature shuffling comprises: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned low-resource license plate recognition methods based on feature shuffling is implemented.

[0070] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned low-resource license plate recognition methods based on feature shuffling.

[0071] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0072] According to an embodiment of the present invention, a sample data set of license plate images is first obtained; the sample data set includes: license plate images and license plate annotation information; an initial license plate recognition model is constructed; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module; secondly, the sample data set is input into the initial license plate recognition model, and the features of the plurality of license plate images are extracted by the global perception encoder to obtain global visual features; the preset initial reading order is shuffled by the feature shuffling module to obtain the shuffled reading order; the global visual features are processed according to the shuffled reading order to obtain the shuffled spatial attention features; the annotation information of the license plate is shuffled according to the shuffled reading order to obtain the shuffled annotation information; the shuffled spatial attention features are input into the decoder module for prediction to obtain the prediction results; according to the prediction results and the shuffled annotation information, the initial license plate recognition model is trained by minimizing the cross entropy loss function to obtain the trained license plate recognition model; finally, the license plate image data to be recognized is obtained; the license plate image data to be recognized is input into the trained license plate recognition model to obtain the predicted license plate number.

[0073] In an embodiment of the present invention, a feature shuffling module is used to shuffle the reading order of a license plate and the annotation information of the license plate; wherein the feature shuffling module can shuffle the features of each character in the license plate during the attention calculation process, and the occurrence probability distribution between different character categories can be balanced by shuffling; the recognition of the tail class is enhanced by the feature shuffling module, which can solve the problem of low recognition rate of the tail class; the embodiment of the present invention adopts a global perception encoder, which can have an overall understanding of the overall character layout in the license plate to ensure effective feature shuffling, assist the feature shuffling module, and improve the performance of low-resource license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 It is an overall flow chart of the training phase of a low-resource license plate recognition method based on feature shuffling provided by an embodiment of the present invention;

[0076] Figure 2 It is a flow chart of a low-resource license plate recognition method based on feature shuffling provided by an embodiment of the present invention;

[0077] Figure 3 It is a block diagram of a low-resource license plate recognition device based on feature shuffling provided by an embodiment of the present invention;

[0078] Figure 4 It is a structural schematic diagram of a low-resource license plate recognition device based on feature shuffling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0080] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0081] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0082] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0083] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0084] The embodiment of the present invention provides a low-resource license plate recognition method based on feature shuffling, which can be implemented by a low-resource license plate recognition device based on feature shuffling, and the low-resource license plate recognition device based on feature shuffling can be a terminal or a server. Figure 1The figure shows an overall flow chart of the training phase of a low-resource license plate recognition method based on feature shuffling provided by an embodiment of the present invention; a feasible implementation method, the embodiment of the present invention takes a license plate image with the license plate annotation information GT of AZH5961 as an example; obtains license plate image data; inputs the license plate image data into a global perception encoder to obtain global visual features arranged in the character sequence 'AZH5961'; the preset initial reading order is '1234567'; the preset initial reading order is input into a feature shuffling module, through which The reading order shuffling module shuffles the license plates five times, and the reading order after shuffling includes: '3514162', '4153126', '3276154', '2154363' and '6313745'; according to the reading order after shuffling, the license plate annotation information GT is shuffled five times accordingly, and the license plate annotation information GT after shuffling includes: 'H9A5A6Z', '5A9HAZ6', 'HZ16A95', 'ZA95H6H' and '6HAH159'. The reading order after shuffling is calculated by the global visual features of the embedding layer, linear layer and input feature shuffling module to calculate the attention weights. The global visual features are shuffled by the attention weights to obtain the shuffled spatial attention features including: 'H9A5A6Z', '5A9HAZ6', 'HZ16A95', 'ZA95H6H' and '6HAH159'; the shuffled spatial attention features are input into the decoder module to obtain the prediction results including: 'H9A5A6Z', '5A9HAZ6', 'HZ16A95', 'ZA95H6H' and '6HAH159'. According to the prediction results and the shuffled annotation information, the initial license plate recognition model is trained by minimizing the cross loss function to obtain the trained license plate recognition model. The initial license plate annotation information GT is compared with the shuffled license plate annotation information GT, and the distribution probability of long-tail characters in the sample data set of license plate images is increased by shuffling.

[0085] like Figure 2 The flowchart of the low-resource license plate recognition method based on feature shuffling is shown in the figure. The processing flow of the method may include the following steps:

[0086] S1. Obtain a sample data set of license plate images; the sample data set includes: multiple license plate images and license plate annotation information.

[0087] S2. Build an initial license plate recognition model; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module.

[0088] Optionally, a global perceptual encoder for extracting global visual features of the input data;

[0089] The global perception encoder includes: a global perception network and a local perception network; the local perception network is used to extract local visual features; the global perception network perceives the overall character layout in the license plate based on the local visual features, and is used to extract deep features;

[0090] Among them, the embodiment of the present invention adopts the original Transformer encoder as the global perception network, wherein the Transformer encoder includes two 4-head self-attention layers, a feedforward layer with a dimension of 512, and an output layer with a dimension of 128.

[0091] Among them, the local perception network consists of 9 convolutional layers, each convolutional layer uses a 3×3 convolution kernel with a step size of 1; among them, the maximum pooling layer of the local perception network is 2×2 after the third and sixth convolutions, and the final downsampling rate is 4.

[0092] In a feasible implementation, the input image is initially converted into a grayscale image, where the size of the image is defined as H×W×C, where H represents the height of the input image, W represents the width of the input image, and C represents the number of channels of the image. The input image size is 32×96×1.

[0093] Among them, the local perception network can generate a feature map of size 8×24×128 to represent local visual features.

[0094] A feature shuffling module is used to shuffle the preset initial reading order to obtain the shuffled reading order, process the global visual features through the shuffled reading order to obtain the shuffled spatial attention features;

[0095] The feature shuffling module includes: a reading order shuffling module, an embedding layer, and a linear layer;

[0096] Decoder module for processing the shuffled spatial attention features.

[0097] S3. Input multiple license plate images into the initial license plate recognition model, extract features from the multiple license plate images through a global perception encoder to obtain global visual features; shuffle the preset initial reading order through a feature shuffling module to obtain a shuffled reading order; process the global visual features according to the shuffled reading order to obtain shuffled spatial attention features; shuffle the labeling information of the license plate according to the shuffled reading order to obtain shuffled labeling information; input the shuffled spatial attention features into the decoder module for prediction to obtain prediction results; based on the prediction results and the shuffled labeling information, train the initial license plate recognition model by minimizing the cross entropy loss function to obtain a trained license plate recognition model.

[0098] Among them, the fully connected layer is used as the decoder to predict all characters of the license plate in parallel. The following formula (1) represents the cross entropy loss function:

[0099]

[0100] Among them, p t represents the predicted probability of decoding step t; y t Indicates the labeling information of the license plate in decoding step t; L ce represents the cross entropy loss function; the value range of t is 1 to T, t represents the decoding step; T represents the maximum length of the license plate annotation information in the sample data set.

[0101] Optionally, before the step of shuffling the preset initial reading order by the feature shuffling module to obtain the shuffled reading order in S3, the step further includes:

[0102] Construct a nested dictionary of the probability distribution of character categories based on the sample data set, and formulate a shuffling strategy based on the nested dictionary;

[0103] The specific implementation process of the shuffling strategy includes:

[0104] Create a nested dictionary of the probability distribution of character categories based on the training set;

[0105] Among them, the nested dictionary is represented as:

[0106] Where i represents the i-th character position among the T character positions of the license plate number; cls j represents the jth category; represents the probability of the jth category appearing at the i-th position in the training set; CP represents the probability of the character category appearing;

[0107] The nested dictionary includes: an outer dictionary and an inner dictionary;

[0108] The keys of the outer dictionary correspond to T=8 character positions; the values ​​of the outer dictionary are the inner dictionary.

[0109] Each key of the internal dictionary corresponds to NC=37 character classes; the value of the internal dictionary represents the probability of each class appearing at each position in the training set;

[0110] Among them, the label information of the license plate of one of the samples in the known training set is GT = "c1c2...c j , c T ″, wherein the preset initial reading order RO = [O1, O2, ..., O j , O T], where T represents the maximum length of the license plate annotation information in the sample data set;

[0111] Among them, the probability distribution of known character categories in is the jth category cls in the training set j The probability of occurrence at position i, where j represents the jth character category and i represents the ith character position; the number of character categories NC = 37;

[0112] For character position i, a candidate set of reading order is constructed based on the annotation information GT of the current training sample license plate; characters that meet the following conditions are selected from the annotation information GT of the current license plate, including: character c j The probability of the corresponding class at position i in the training set The corresponding reading order O j Put it into the reading order candidate set; calculate the possibility of selecting reading order O from the reading order candidate set and placing it at position i SelectProb: If the character c at the current position i i The probability of occurrence of the corresponding class at position i in the training set is CP[i][c i ] is greater than Then SelectProb is c i The probability of occurrence CP[i][c i ] is based on the ratio of the maximum occurrence probability of all classes at position i, adjusted upward, with a base of 0.65 as an empirical value and a maximum upper limit of 0.9; if c i The probability of occurrence CP[i][c i ] is less than or equal to Then SelectProb will be adjusted upward with 0.1 as the base, and the maximum upper limit is 0.35; 0.1 is the empirical value;

[0113] A random number between 0 and 1 is randomly selected and compared with the SelectProb value; if the random number is less than the SelectProb value, a reading order O is randomly selected from the reading order candidate set as the alternative reading order at the current position i; if the random number is equal to or greater than the SelectProb value, the reading order at the current position i remains the current character c. i The corresponding initial reading order O i ; So far, the replacement of the reading order at the current position i is completed; and so on, the replacement of the reading order of all positions from position 1 to position T is completed. The above process completes a reading order shuffling process.

[0114] In this embodiment of the present invention, each sample is shuffled 5 times in each training round.

[0115] The embodiment of the present invention shuffles each license plate in the same batch independently to enhance the diversity of character arrangement. The shuffling strategy can expand the occurrence probability of the tail class and establish a more balanced occurrence probability for all character categories.

[0116] Optionally, S3 inputs multiple license plate images into an initial license plate recognition model, extracts features from the multiple license plate images through a global perception encoder, and obtains global visual features, including:

[0117] S31, inputting a plurality of license plate images into an initial license plate recognition model, extracting features of the plurality of license plate images through a local perception network, and obtaining local visual features;

[0118] S32, inputting the local visual features into the global perception network, processing the local visual features through the global perception network, and obtaining the global visual features.

[0119] Optionally, the step S3 shuffles the preset initial reading order through a feature shuffling module to obtain a shuffled reading order; and processes the global visual features according to the shuffled reading order to obtain a shuffled spatial attention feature, including:

[0120] S33, using a reading order shuffling module, using the probability distribution of character categories in a sample data set as a basis, shuffling a preset initial reading order to obtain a shuffled reading order; using an attention mechanism, using the shuffled reading order as a query, and using global visual features as a key; mapping the key and the query to the same embedding space according to a linear layer and an embedding layer; performing operations on the query and the key according to the linear layer to obtain a spatial attention weight;

[0121] S34. Perform spatial attention on each character in the global visual feature according to the spatial attention weight to obtain a shuffled spatial attention feature.

[0122] In a feasible implementation method, the specific process of obtaining the spatial attention feature after shuffling includes:

[0123] (1) Take the shuffled reading order as the query and the global visual features as the key;

[0124] (2) The shuffled reading order is passed through the embedding layer W E Perform embedding encoding to obtain embedding representation Among them, d h =128 indicates hidden dimension; F E Represents a size of T×d h A two-dimensional tensor of , the values ​​of which are from the set of real numbers; R indicates that the numbers in the set are real numbers; the reading order after shuffling is RO∈RT , where T = 8, represents the maximum length of the license plate annotation information in the sample data set; where, in order to ensure the uniformity of the length of the annotation information of all license plates, multiple blank characters are added to the annotation information GT of the license plate whose annotation information length is less than T;

[0125] (3) Through the linear layer W Q , the embedding representation Project to the query embedding space to get the query embedding representation Among them, H represents the original height of the license plate image data, W represents the original width of the license plate image data; r represents the downsampling rate of the global perception encoder, and its value is 4;

[0126] (4) Through the linear projection layer W K Project the global visual features into the key embedding space to obtain the key embedding representation

[0127] (5) Calculate the spatial attention weight based on the query embedding representation Q and the key embedding representation K; use the spatial attention weight and global visual features to generate spatial attention features.

[0128] The process of obtaining spatial attention features can be expressed by the following formulas (2)-(4):

[0129] Q=(W E (RO) T ×W Q ) T , K=F G ×W K , V=F G (2)

[0130] W A =softmax((Tanh(Q+K)×W I ) T ) (3)

[0131] F SA =W A ×V (4)

[0132] in, as well as are all trainable parameters; the superscript T represents the transpose of the matrix, Q represents the query embedding representation, and W E represents the embedding layer, RO represents the reading order after shuffling, and W Q represents the linear projection layer used to map the embedding representation obtained from the reading order to the query embedding space, K represents the key embedding representation, and F G is the global visual feature, W Krepresents the linear projection layer used to map the global visual information to the key embedding space, V is the global visual feature; W A represents the spatial attention weight; softmax is a mathematical function used to convert a set of arbitrary real numbers into real numbers representing probability distribution; Tanh represents an activation function; W I represents the attention weight transformation layer used to map the attention weights to the new representation space; F SA Represents the spatial attention feature.

[0133] S4. Obtain the license plate image data to be identified; input the license plate image data to be identified into the trained license plate recognition model to obtain the predicted license plate number.

[0134] Optionally, the specific implementation process of S4 includes S41-S42:

[0135] S41, inputting the license plate image data to be recognized into the trained license plate recognition model, and obtaining global visual features through a global perception encoder;

[0136] S42, inputting the global visual features into the feature shuffling module, processing the global visual features according to a preset initial reading order to obtain spatial attention features; predicting the spatial attention features through the decoder module to obtain a predicted license plate number.

[0137] The data sets used in the embodiments of the present invention include: UFPR data set and SSIG-SegPlate data set.

[0138] The UFPR dataset includes 4,500 fully annotated images from 150 moving vehicles, which are taken by mobile cameras. The dataset is divided into 40% for training, 40% for testing, and 20% for validation. The 1,800 license plate images in the training set only contain 60 different license plate character arrangements. The character arrangements in the test set have no intersection with the training set.

[0139] Among them, the SSIG-SegPlate dataset consists of 2,000 Brazilian license plate images captured by static cameras from 101 on-rail vehicles during the day, with a size of 1920×1080 pixels; in the SSIG-SegPlate dataset, there are an average of 19.8 slightly changing license plate images per vehicle; the SSIG-SegPlate dataset is divided into: 40% for training, 40% for testing, and 20% for validation.

[0140] A feasible implementation method, the global perception network used in the embodiment of the present invention improves 3.61% on the UFPR dataset and 15.05% on the SSIG-SegPlate dataset; the reading order shuffling process performed in the feature shuffling module provides an additional performance improvement of 34.78% on the UFPR dataset without increasing parameters and frames per second, exceeding the baseline by 38.39%; and provides an additional performance improvement of 33.96% on the SSIG-SegPlate dataset, exceeding the baseline by 49.01%. Compared with the baseline, using only reading order shuffling without using the global perception network will result in a decrease in performance on the UFPR dataset; combining reading order shuffling with the global perception network can enhance effectiveness. Table 1 shows the ablation study results of the global perception network and reading order shuffling.

[0141] Table 1 Ablation study results of global perception and reading order shuffling

[0142]

[0143] As a feasible implementation method, the embodiment of the present invention conducts an ablation experiment on the reading order shuffling strategy. The feature shuffling module in the embodiment of the present invention highlights the effectiveness of the reading order shuffling in more balancing the probability distribution of character categories at each position. Considering the probability of characters appearing in the training set, the recognition performance can be improved by increasing the probability of the tail class; independently shuffling each sample in the same training batch improves the recognition performance by introducing more diverse character arrangements, thereby balancing the distribution of occurrences between different character classes; wherein, Table 2 shows the ablation study results of the reading order shuffling strategy.

[0144] Table 2 Ablation study results of reading order shuffling strategy

[0145]

[0146] A feasible implementation method, the embodiment of the present invention and the prior art are tested on sequence recognition and character detection on the UFPR dataset and the SSIG-SegPlate dataset. The embodiment of the present invention obtains the best recognition performance with the least parameters. The embodiment of the present invention improves the recognition performance when using synthetic license plates by introducing more diverse character arrangements and ensuring a more even distribution of character categories at each position. Table 3 shows the comparative experimental results obtained by testing the embodiment of the present invention and the prior art on the UFPR dataset and the SSIG-SegPlate dataset.

[0147] Table 3 Comparative experimental results of UFPR dataset and SSIG-SegPlate dataset

[0148]

[0149]

[0150] A feasible implementation method uses recognition accuracy as an indicator of performance evaluation. The recognition accuracy refers to the ratio of the number of correctly recognized license plates to the total number of license plates. When all characters on the license plate are recognized correctly, the entire license plate is considered to be correctly recognized.

[0151] According to an embodiment of the present invention, a sample data set of license plate images is first obtained; the sample data set includes: license plate images and license plate annotation information; an initial license plate recognition model is constructed; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module; secondly, the sample data set is input into the initial license plate recognition model, and the features of the plurality of license plate images are extracted by the global perception encoder to obtain global visual features; the preset initial reading order is shuffled by the feature shuffling module to obtain the shuffled reading order; the global visual features are processed according to the shuffled reading order to obtain the shuffled spatial attention features; the annotation information of the license plate is shuffled according to the shuffled reading order to obtain the shuffled annotation information; the shuffled spatial attention features are input into the decoder module for prediction to obtain the prediction results; according to the prediction results and the shuffled annotation information, the initial license plate recognition model is trained by minimizing the cross entropy loss function to obtain the trained license plate recognition model; finally, the license plate image data to be recognized is obtained; the license plate image data to be recognized is input into the trained license plate recognition model to obtain the predicted license plate number.

[0152] In an embodiment of the present invention, a feature shuffling module is used to shuffle the reading order of a license plate and the annotation information of the license plate; wherein the feature shuffling module can shuffle the features of each character in the license plate during the attention calculation process, and the occurrence probability distribution between different character categories can be balanced by shuffling; the recognition of the tail class is enhanced by the feature shuffling module, which can solve the problem of low recognition rate of the tail class; the embodiment of the present invention adopts a global perception encoder, which can have an overall understanding of the overall character layout in the license plate to ensure effective feature shuffling, assist the feature shuffling module, and improve the performance of low-resource license plate recognition.

[0153] Figure 3 1 is a block diagram of a low-resource license plate recognition device based on feature shuffling according to an exemplary embodiment, wherein the device is used in a low-resource license plate recognition method based on feature shuffling. Figure 3 The device includes an acquisition unit 310, a construction unit 320, a training unit 330 and a prediction unit 340. Among them:

[0154] The acquisition unit 310 is used to acquire a sample data set of license plate images; the sample data set includes: a plurality of license plate images and labeling information of the license plates;

[0155] A construction unit 320 is used to construct an initial license plate recognition model; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module;

[0156] The training unit 330 is used to input the multiple license plate images into the initial license plate recognition model, extract features from the multiple license plate images through the global perception encoder to obtain global visual features; shuffle the preset initial reading order through the feature shuffling module to obtain the shuffled reading order; process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; shuffle the license plate annotation information according to the shuffled reading order to obtain the shuffled annotation information; input the shuffled spatial attention features into the decoder module for prediction to obtain the prediction results; train the initial license plate recognition model by minimizing the cross entropy loss function according to the prediction results and the shuffled annotation information to obtain the trained license plate recognition model;

[0157] The prediction unit 340 is used to obtain the image data of the license plate to be recognized; input the image data of the license plate to be recognized into the trained license plate recognition model to obtain the predicted license plate number.

[0158] Optionally, the construction unit 320 is used to:

[0159] The global perception encoder includes: a global perception network and a local perception network; the local perception network is used to extract local visual features; the global perception network perceives the overall character layout in the license plate based on the local visual features, and is used to extract deep features;

[0160] The feature shuffling module is used to shuffle the preset initial reading order to obtain the shuffled reading order, and process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features;

[0161] The feature shuffling module includes: a reading order shuffling module, an embedding layer and a linear layer;

[0162] The decoder module is used to process the shuffled spatial attention features.

[0163] Optionally, before the step of shuffling the preset initial reading order by the feature shuffling module to obtain the shuffled reading order, the method further includes:

[0164] Constructing a nested dictionary of probability distribution of character categories according to the sample data set, and formulating a shuffling strategy according to the nested dictionary;

[0165] The specific implementation process of the shuffling strategy includes:

[0166] Create a nested dictionary of the probability distribution of character categories based on the training set;

[0167] Among them, the nested dictionary is represented as:

[0168] Where i represents the i-th character position among the T character positions of the license plate number; cls j represents the jth category; represents the probability of the jth category appearing at the i-th position in the training set; CP represents the probability of the character category appearing;

[0169] The nested dictionary includes: an outer dictionary and an inner dictionary;

[0170] The keys of the outer dictionary correspond to T=8 character positions; the values ​​of the outer dictionary are the inner dictionary.

[0171] Each key of the internal dictionary corresponds to NC=37 character classes; the value of the internal dictionary represents the probability of each class appearing at each position in the training set;

[0172] Among them, the label information of the license plate of one of the samples in the known training set is GT = "c1c2...c j , c T ″, wherein the preset initial reading order RO = [O1, O2, ..., O j , O T ], where T represents the maximum length of the license plate annotation information in the sample data set;

[0173] Among them, the probability distribution of known character categories in is the jth category cls in the training set j The probability of occurrence at position i, where j represents the jth character category and i represents the ith character position; the number of character categories NC = 37;

[0174] For character position i, a candidate set of reading order is constructed based on the annotation information GT of the current training sample license plate; characters that meet the following conditions are selected from the annotation information GT of the current license plate, including: character c j The probability of the corresponding class at position i in the training set The corresponding reading order O j Put it into the reading order candidate set; calculate the possibility of selecting reading order O from the reading order candidate set and placing it at position i SelectProb: If the character c at the current position i i The probability of occurrence of the corresponding class at position i in the training set is CP[i][ci ] is greater than Then SelectProb is c i The probability of occurrence CP[i][c i ] is based on the ratio of the maximum occurrence probability of all classes at position i, adjusted upward, with a base of 0.65 as an empirical value and a maximum upper limit of 0.9; if c i The probability of occurrence CP[i][c i ] is less than or equal to Then SelectProb will be adjusted upward with 0.1 as the base, and the maximum upper limit is 0.35; 0.1 is the empirical value;

[0175] A random number between 0 and 1 is randomly selected and compared with the SelectProb value; if the random number is less than the SelectProb value, a reading order O is randomly selected from the reading order candidate set as the alternative reading order at the current position i; if the random number is equal to or greater than the SelectProb value, the reading order at the current position i remains the current character c. i The corresponding initial reading order O i ; So far, the replacement of the reading order at the current position i is completed; and so on, the replacement of the reading order of all positions from position 1 to position T is completed. The above process completes a reading order shuffling process.

[0176] Optionally, the multiple license plate images are input into the initial license plate recognition model, and feature extraction is performed on the multiple license plate images by the global perception encoder to obtain global visual features, including:

[0177] Inputting the multiple license plate images into the initial license plate recognition model, extracting features from the multiple license plate images through a local perception network to obtain local visual features;

[0178] The local visual features are input into the global perception network, and the local visual features are processed by the global perception network to obtain the global visual features.

[0179] Optionally, the feature shuffling module shuffles the preset initial reading order to obtain a shuffled reading order; and processes the global visual features according to the shuffled reading order to obtain a shuffled spatial attention feature, including:

[0180] Through the reading order shuffling module, the character category occurrence probability distribution of the sample data set is used as the basis to shuffle the preset initial reading order to obtain the shuffled reading order; the attention mechanism is used to take the shuffled reading order as the query and the global visual feature as the key; according to the linear layer and the embedding layer, the key and the query are mapped to the same embedding space; according to the linear layer, the query and the key are operated to obtain the spatial attention weight;

[0181] According to the spatial attention weight, spatial attention is performed on each character in the global visual feature to obtain the shuffled spatial attention feature.

[0182] Optionally, the prediction unit 340 is configured to:

[0183] The license plate image data to be recognized is input into the trained license plate recognition model, and the global visual features are obtained through the global perception encoder;

[0184] The global visual features are input into the feature shuffling module, and the global visual features are processed according to a preset initial reading order to obtain spatial attention features; the spatial attention features are predicted by the decoder module to obtain a predicted license plate number.

[0185] According to an embodiment of the present invention, a sample data set of license plate images is first obtained; the sample data set includes: license plate images and license plate annotation information; an initial license plate recognition model is constructed; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module; secondly, multiple license plate images are input into the initial license plate recognition model, and feature extraction is performed on the multiple license plate images by the global perception encoder to obtain global visual features; a preset initial reading order is shuffled by the feature shuffling module to obtain a shuffled reading order; the global visual features are processed according to the shuffled reading order to obtain a shuffled spatial attention feature; the annotation information of the license plate is shuffled according to the shuffled reading order to obtain the shuffled annotation information; the shuffled spatial attention feature is input into the decoder module for prediction to obtain a prediction result; according to the prediction result and the shuffled annotation information, the initial license plate recognition model is trained by minimizing the cross entropy loss function to obtain a trained license plate recognition model; finally, the license plate image data to be recognized is obtained; the license plate image data to be recognized is input into the trained license plate recognition model to obtain a predicted license plate number.

[0186] In an embodiment of the present invention, a feature shuffling module is used to shuffle the reading order of a license plate and the annotation information of the license plate; wherein the feature shuffling module can shuffle the features of each character in the license plate during the attention calculation process, and the occurrence probability distribution between different character categories can be balanced by shuffling; the recognition of the tail class is enhanced by the feature shuffling module, which can solve the problem of low recognition rate of the tail class; the embodiment of the present invention adopts a global perception encoder, which can have an overall understanding of the overall character layout in the license plate to ensure effective feature shuffling, assist the feature shuffling module, and improve the performance of low-resource license plate recognition.

[0187] Figure 4 is a structural diagram of a low-resource license plate recognition device based on feature shuffling provided by an embodiment of the present invention, such as Figure 4 As shown, the low-resource license plate recognition device based on feature shuffling may include the above Figure 3 The low-resource license plate recognition device based on feature shuffling is shown. Optionally, the low-resource license plate recognition device based on feature shuffling 410 may include a first processor 2001 .

[0188] Optionally, the low-resource license plate recognition device 410 based on feature shuffling may further include a memory 2002 and a transceiver 2003 .

[0189] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0190] Combine the following Figure 4 The components of the low-resource license plate recognition device 410 based on feature shuffling are specifically introduced:

[0191] The first processor 2001 is the control center of the low-resource license plate recognition device 410 based on feature shuffling, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0192] Optionally, the first processor 2001 can perform various functions of the low-resource license plate recognition device 410 based on feature shuffling by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0193] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0194] In a specific implementation, as an embodiment, the low-resource license plate recognition device 410 based on feature shuffling may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0195] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0196] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0197] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0198] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0199] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0200] It should be noted that Figure 4 The structure of the low-resource license plate recognition device 410 based on feature shuffling shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0201] In addition, the technical effects of the low-resource license plate recognition device 410 based on feature shuffling can refer to the technical effects of the low-resource license plate recognition method based on feature shuffling described in the above method embodiment, which will not be repeated here.

[0202] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0203] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0204] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0205] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0206] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0207] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0208] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0209] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0210] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0213] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0214] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A low-resource license plate recognition method based on feature shuffling, characterized in that: The method comprises: S1. Obtain a sample data set of license plate images; the sample data set includes: a plurality of license plate images and license plate annotation information; S2. constructing an initial license plate recognition model; the license plate recognition model comprises: a global perception encoder, a feature shuffling module and a decoder module; S3, input the multiple license plate images into the initial license plate recognition model, extract features from the multiple license plate images through the global perception encoder to obtain global visual features; shuffle the preset initial reading order through the feature shuffling module to obtain the shuffled reading order; process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; shuffle the license plate annotation information according to the shuffled reading order to obtain the shuffled annotation information; input the shuffled spatial attention features into the decoder module for prediction to obtain the prediction results; according to the prediction results and the shuffled annotation information, train the initial license plate recognition model by minimizing the cross entropy loss function to obtain the trained license plate recognition model; Wherein, before the step of shuffling the preset initial reading order by the feature shuffling module to obtain the shuffled reading order in S3, the step further includes: A nested dictionary of the probability distribution of character categories is constructed according to the sample data set, and a shuffling strategy is formulated according to the nested dictionary; wherein the specific implementation process of the shuffling strategy includes: Create a nested dictionary of the probability distribution of character categories based on the training set; Among them, the nested dictionary is represented as: Where i represents the i-th character position among the T character positions of the license plate number; cls j represents the jth category; represents the probability of the jth category appearing at the i-th position in the training set; CP represents the probability of the character category appearing; The nested dictionary includes: an outer dictionary and an inner dictionary; Wherein, the keys of the outer dictionary correspond to T=8 character positions; the values ​​of the outer dictionary are the inner dictionary; Each key of the internal dictionary corresponds to NC=37 character classes; the value of the internal dictionary represents the probability of each class appearing at each position in the training set; Among them, the label information of the license plate of one of the samples in the known training set is GT = "c1c2...c j , c T ″, wherein the preset initial reading order RO = [O1, O2, ..., O j , O T ], where T represents the maximum length of the license plate annotation information in the sample data set; Among them, the probability distribution of known character categories in is the jth category cls in the training set j The probability of occurrence at position i, where j represents the jth character category and i represents the ith character position; the number of character categories NC = 37; For character position i, a candidate set of reading order is constructed based on the annotation information GT of the current training sample license plate; characters that meet the following conditions are selected from the annotation information GT of the current license plate, including: character c j The probability of the corresponding class at position i in the training set The corresponding reading order O j Put it into the reading order candidate set; calculate the possibility of selecting reading order O from the reading order candidate set and placing it at position i SelectProb: If the character c at the current position i i The probability of occurrence of the corresponding class at position i in the training set is CP[i][c i ] is greater than Then SelectProb is c i The probability of occurrence CP[i][c i ] is based on the ratio of the maximum occurrence probability of all classes at position i, adjusted upward, with a base of 0.65 as an empirical value and a maximum upper limit of 0.9; if c i The probability of occurrence CP[i][c i ] is less than or equal to Then SelectProb will be adjusted upward with 0.1 as the base, and the maximum upper limit is 0.35; 0.1 is the empirical value; A random number between 0 and 1 is randomly selected and compared with the SelectProb value; if the random number is less than the SelectProb value, a reading order O is randomly selected from the reading order candidate set as the alternative reading order at the current position i; if the random number is equal to or greater than the SelectProb value, the reading order at the current position i remains the current character c. i The corresponding initial reading order O i ; So far, the replacement of the reading order at the current position i is completed; and so on, the replacement of the reading order of all positions from position 1 to position T is completed. The above process completes a reading order shuffling process; S4. Obtain the license plate image data to be identified; input the license plate image data to be identified into the trained license plate recognition model to obtain the predicted license plate number.

2. The low-resource license plate recognition method based on feature shuffling according to claim 1 is characterized in that: The global perception encoder is used to extract global visual features of input data; The global perception encoder includes: a global perception network and a local perception network; the local perception network is used to extract local visual features; the global perception network perceives the overall character layout in the license plate based on the local visual features, and is used to extract deep features; The feature shuffling module is used to shuffle the preset initial reading order to obtain the shuffled reading order, and process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; The feature shuffling module includes: a reading order shuffling module, an embedding layer and a linear layer; The decoder module is used to process the shuffled spatial attention features.

3. The low-resource license plate recognition method based on feature shuffling according to claim 2 is characterized in that: The step S3 of inputting the plurality of license plate images into the initial license plate recognition model and extracting features from the plurality of license plate images by the global perception encoder to obtain global visual features includes: S31, inputting the multiple license plate images into the initial license plate recognition model, extracting features from the multiple license plate images through a local perception network, and obtaining local visual features; S32, inputting the local visual features into the global perception network, processing the local visual features through the global perception network, and obtaining the global visual features.

4. The low-resource license plate recognition method based on feature shuffling according to claim 2 is characterized in that: The S3 shuffles the preset initial reading order through a feature shuffling module to obtain a shuffled reading order; processes the global visual features according to the shuffled reading order to obtain a shuffled spatial attention feature, including: S33, using a reading order shuffling module, using the probability distribution of character categories in a sample data set as a basis, shuffling a preset initial reading order to obtain a shuffled reading order; using an attention mechanism, using the shuffled reading order as a query, and using global visual features as a key; mapping the key and the query to the same embedding space according to a linear layer and an embedding layer; performing operations on the query and the key according to the linear layer to obtain a spatial attention weight; S34. Perform spatial attention on each character in the global visual feature according to the spatial attention weight to obtain a shuffled spatial attention feature.

5. The low-resource license plate recognition method based on feature shuffling according to claim 1 is characterized in that: The step S4 inputs the license plate image data to be recognized into the trained license plate recognition model to obtain the predicted license plate number, including: S41, inputting the license plate image data to be recognized into the trained license plate recognition model, and obtaining global visual features through the global perception encoder; S42, inputting the global visual features into the feature shuffling module, processing the global visual features according to a preset initial reading order to obtain spatial attention features; predicting the spatial attention features through the decoder module to obtain a predicted license plate number.

6. A low-resource license plate recognition device based on feature shuffling, the low-resource license plate recognition device based on feature shuffling is used to implement the low-resource license plate recognition method based on feature shuffling as claimed in any one of claims 1 to 5, characterized in that: The device comprises: An acquisition unit is used to acquire a sample data set of a license plate image; the sample data set includes: a license plate image and license plate annotation information; A construction unit, used to construct an initial license plate recognition model; the license plate recognition model includes: a global perception encoder, a feature shuffling module and a decoder module; A training unit is used to input the multiple license plate images into an initial license plate recognition model, extract features from the multiple license plate images through the global perception encoder to obtain global visual features; shuffle the preset initial reading order through the feature shuffling module to obtain the shuffled reading order; process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; shuffle the license plate annotation information according to the shuffled reading order to obtain the shuffled annotation information; input the shuffled spatial attention features into the decoder module for prediction to obtain the prediction results; train the initial license plate recognition model by minimizing the cross entropy loss function according to the prediction results and the shuffled annotation information to obtain the trained license plate recognition model; The prediction unit is used to obtain the license plate image data to be recognized; the license plate image data to be recognized is input into the trained license plate recognition model to obtain the predicted license plate number.

7. The low-resource license plate recognition device based on feature shuffling according to claim 6 is characterized in that: The building block is used for: The global perception encoder includes: a global perception network and a local perception network; the local perception network is used to extract local visual features; the global perception network perceives the overall character layout in the license plate based on the local visual features, and is used to extract deep features; The feature shuffling module is used to shuffle the preset initial reading order to obtain the shuffled reading order, and process the global visual features according to the shuffled reading order to obtain the shuffled spatial attention features; Wherein, before the step of shuffling the preset initial reading order by the feature shuffling module to obtain the shuffled reading order, the step further includes: A nested dictionary of the probability distribution of character categories is constructed according to the sample data set, and a shuffling strategy is formulated according to the nested dictionary; wherein the specific implementation process of the shuffling strategy includes: Create a nested dictionary of the probability distribution of character categories based on the training set; Among them, the nested dictionary is represented as: Where i represents the i-th character position among the T character positions of the license plate number; cls j represents the jth category; represents the probability of the jth category appearing at the i-th position in the training set; CP represents the probability of the character category appearing; The nested dictionary includes: an outer dictionary and an inner dictionary; Wherein, the keys of the outer dictionary correspond to T=8 character positions; the values ​​of the outer dictionary are the inner dictionary; Each key of the internal dictionary corresponds to NC=37 character classes; the value of the internal dictionary represents the probability of each class appearing at each position in the training set; Among them, the label information of the license plate of one of the samples in the known training set is GT = "c1c2...c j , c T ″, wherein the preset initial reading order RO = [O1, O2, ..., O j , O T ], where T represents the maximum length of the license plate annotation information in the sample data set; Among them, the probability distribution of known character categories in is the jth category cls in the training set j The probability of occurrence at position i, where j represents the jth character category and i represents the ith character position; the number of character categories NC = 37; For character position i, a candidate set of reading order is constructed based on the annotation information GT of the current training sample license plate; characters that meet the following conditions are selected from the annotation information GT of the current license plate, including: character c j The probability of the corresponding class at position i in the training set The corresponding reading order O j Put it into the reading order candidate set; calculate the possibility of selecting reading order O from the reading order candidate set and placing it at position i SelectProb: If the character c at the current position i i The probability of occurrence of the corresponding class at position i in the training set is CP[i][c i ] is greater than Then SelectProb is c i The probability of occurrence CP[i][c i ] is based on the ratio of the maximum occurrence probability of all classes at position i, adjusted upward, with a base of 0.65 as an empirical value and a maximum upper limit of 0.9; if c i The probability of occurrence CP[i][c i ] is less than or equal to Then SelectProb will be adjusted upward with 0.1 as the base, and the maximum upper limit is 0.35; 0.1 is the empirical value; A random number between 0 and 1 is randomly selected and compared with the SelectProb value; if the random number is less than the SelectProb value, a reading order O is randomly selected from the reading order candidate set as the alternative reading order at the current position i; if the random number is equal to or greater than the SelectProb value, the reading order at the current position i remains the current character c. i The corresponding initial reading order O i ; So far, the replacement of the reading order at the current position i is completed; and so on, the replacement of the reading order of all positions from position 1 to position T is completed. The above process completes a reading order shuffling process; The feature shuffling module includes: a reading order shuffling module, an embedding layer and a linear layer; The decoder module is used to process the shuffled spatial attention features.

8. A low-resource license plate recognition device based on feature shuffling, characterized in that: The low-resource license plate recognition device based on feature shuffling includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images