A license plate recognition method and system based on attention distillation and a storage medium

Abstract

The application discloses a license plate recognition method and system based on attention distillation and a storage medium, relates to the technical field of license plate recognition, and inputs a preprocessed license plate image into a trained deep neural network model to output a license plate recognition result; the deep neural network model comprises a feature extraction module, a main branch bottleneck module, an auxiliary branch bottleneck module and a prediction module, the input end of the main branch bottleneck module and the auxiliary branch bottleneck module in parallel is connected with the output end of the feature extraction module, and the output end is connected with the input end of the prediction module; the feature extraction module inputs the preprocessed license plate image, and the prediction module outputs the license plate recognition result; the license plate recognition method and system and the storage medium distill the global attention knowledge of a core of a teacher model based on a transformer decoder, and transfer the global attention knowledge to a student model based on CNN, so that the recognition accuracy and robustness are improved.

Description

Technical Field

[0001] This invention relates to the field of license plate recognition technology, and in particular to a license plate recognition method, system, and storage medium based on attention distillation. Background Technology

[0002] License plate recognition is a core technology of intelligent transportation, comprising two main parts: license plate location detection and license plate recognition. License plate recognition is the most crucial component, and the quality of the license plate recognition engine directly determines the overall performance of the technology.

[0003] License plate recognition refers to the accurate and complete identification of multiple information about a license plate from an image with a known license plate location, such as license plate color, license plate type, license plate characters, and the position of the license plate characters.

[0004] License plate color recognition mainly involves converting the image from the RGB color space to the HSV color space, and then identifying the license plate color based on the color histogram distribution or a color-based classifier.

[0005] License plate type recognition is mainly achieved by classifying the appearance features of the license plate (such as color, size, and character layout), usually using machine learning or deep learning methods.

[0006] License plate character location recognition refers to accurately locating each individual character in an image with known license plate locations. The main methods include:

[0007] (1) The method based on vertical projection obtains the vertical projection curve of the license plate characters and obtains the edge position of each character according to the peak and trough positions of the curve.

[0008] (2) The method based on connected component analysis first binarizes the license plate image, then analyzes it using the feature that each character is a single connected component, and finally obtains the position of the character.

[0009] (3) Machine learning-based methods, such as “Chinese Patent Publication No. CN103207998A (A method for segmenting license plate characters based on support vector machine)”, which obtains the layout pattern features of license plates, trains and learns with the help of a classifier, and finally completes the segmentation of license plate characters.

[0010] License plate character recognition refers to identifying the true meaning of a single, precisely segmented character. Commonly used methods include the following:

[0011] (1) Global features: This type of feature uses global transformation to obtain the overall features of the character. It uses ordered global features or subset features to construct the feature vector. Common features include GABOR transform features, moment features, projection features, stroke density features, HARR features, HOG features, etc. The advantage of these features is that they are not sensitive to local changes and have strong anti-interference ability; the disadvantage is that they are easy to ignore some important local features and cannot distinguish similar characters.

[0012] (2) Local features: This type of feature calculates corresponding features in multiple local regions of a character and uses concatenated ordered local features to form the final feature vector. The main features include local grayscale histogram features, LBP features, line-through features, SIFT features, etc. The advantage of this type of feature is its strong ability to distinguish characters; its disadvantage is that it focuses too much on the local features of the character and often misdistinguishes characters with noise interference.

[0013] The technologies mentioned above can achieve good results for clear license plate images. However, license plate images captured in real-world environments often suffer from low resolution, color cast, faded or missing characters, blurred edges, and tilted characters, making accurate license plate recognition difficult and even causing failures, severely impacting the overall performance of license plate recognition. Therefore, accurate and robust license plate recognition remains a challenge for domestic license plate recognition systems.

[0014] In recent years, the transformer technology, with its global attention mechanism, has been able to make accurate global receptive field nonlinear predictions and capture global long-distance dependencies, and has received widespread attention and application in various fields. However, the disadvantage of this technology is that the model consumes a lot of memory and computation, thus requiring a large amount of memory and computing power. Summary of the Invention

[0015] Based on the technical problems existing in the background technology, the present invention proposes a license plate recognition method, system and storage medium based on attention distillation, which improves the recognition accuracy and robustness.

[0016] The present invention proposes a license plate recognition method based on attention distillation, which preprocesses the license plate image and inputs it into a trained deep neural network model to output the license plate recognition result;

[0017] The deep neural network model includes a feature extraction module, a main branch bottleneck module, an auxiliary branch bottleneck module, and a prediction module. The input of the main branch bottleneck module and the auxiliary branch bottleneck module connected in parallel is connected to the output of the feature extraction module, and the output is connected to the input of the prediction module. The feature extraction module inputs the preprocessed license plate image, and the prediction module outputs the license plate recognition result.

[0018] The main branch bottleneck module abstracts license plate image features based on the CNN network structure to obtain the main feature map; the auxiliary branch bottleneck module abstracts the global and local features of the license plate image through self-attention and global attention mechanisms to obtain the auxiliary feature map.

[0019] Furthermore, the feature extraction module is used to obtain a high-level feature map of the license plate image with high abstraction and rich expressive power; the feature extraction module includes a convolutional layer conv0, a max pooling layer maxpool0, a first residual module, a second residual module, a merging layer eltsum0, a merging layer eltsum1, and a convolutional layer conv1;

[0020] The input license plate image is the convolutional layer conv0. The convolutional layer conv0, the max pooling layer maxpool0, and the first residual module are connected in sequence and then connected to the input of the merging layer eltsum0.

[0021] The output of the merging layer eltsum0 is connected to the input of the merging layer eltsum1 and the input of the second residual module, respectively. The output of the second residual module is connected to the input of the merging layer eltsum1. The output of the merging layer eltsum1 is connected to the input of the convolutional layer conv1. The convolutional layer conv1 outputs a high-level feature map.

[0022] Furthermore, the first residual module includes convolutional layers rconv0, rconv1, and rconv2, and the second residual module includes convolutional layers rconv3 and rconv4.

[0023] Convolutional layers rconv0 and rconv1 are concatenated and then set in parallel with convolutional layer rconv2. The input of the parallel layer is connected to the output of the max pooling layer maxpool0, and the output of the parallel layer is connected to the input of the merging layer eltsum0.

[0024] The inputs of concatenated convolutional layers rconv3 and rconv4 are connected to the output of merged layer eltsum0, and the concatenated outputs are connected to the input of merged layer eltsum1.

[0025] Furthermore, the main branch bottleneck module includes a convolutional layer main_conv, a feature map reshaping layer main_reshape, and a fully connected layer main_fc connected in sequence, with the high-level feature map being input to the convolutional layer main_conv;

[0026] The feature map reshaping layer `main_reshape` is used to adjust the size of the input feature map to meet the needs of subsequent feature layer operations; the fully connected layer `main_fc` is used to adjust the number of channels of the output feature map to output the main feature map.

[0027] Furthermore, the auxiliary branch bottleneck module includes an M-layer decoder structure, where the license plate feature query vector output by the previous layer decoder structure is used as the input of the current layer decoder structure, and the M-layer decoder structure outputs an auxiliary feature map.

[0028] Each layer of the decoder structure is based on self-attention mechanism and global attention mechanism to abstract the global and local features of the license plate image. The current layer decoder structure includes a multi-head self-attention layer, a first additive normalization layer, a multi-head global attention layer, a fully connected layer, a second additive normalization layer and a third additive normalization layer.

[0029] The license plate feature query vector output by the previous layer decoder structure is input into the multi-head self-attention layer of the current layer. After processing by the self-attention mechanism, the intermediate layer features are obtained through the first summing and normalization layer.

[0030] The intermediate layer features are pixel-wise added to the license plate feature query vector output by the previous layer decoder structure and used as the query Q of the multi-head global attention layer. The high-level feature map and the corresponding position-encoded feature map are pixel-wise added and used as the key K of the multi-head global attention layer. The high-level feature map is used as the value V of the multi-head global attention layer. The attention features output by the multi-head global attention layer and the intermediate layer features are combined through the second addition normalization layer to obtain the added features. After passing through the fully connected layer, the added features enter the third addition normalization layer for further addition and normalization. The current layer outputs the license plate feature query vector through the third addition normalization layer.

[0031] Furthermore, the prediction module includes a feature channel splitting layer (split) and a fully connected layer (fc);

[0032] The feature channel splitting layer splits the input main feature map or auxiliary feature map into multiple sub-feature maps according to the channel dimension. The sub-feature maps are then passed through the fully connected layer fc to obtain the license plate recognition result. The number of fully connected layers fc corresponds to the number of sub-feature maps.

[0033] Furthermore, a supervision function is set for the deep neural network model. The supervision function includes a main supervision function main_loss and an auxiliary supervision function aux_loss. The auxiliary branch bottleneck module is only used to assist training and is not used in the execution process after the deep neural network model is trained.

[0034] The main supervision function is used to supervise the training of license plate recognition-related tasks. The license plate character recognition task uses the CTC loss function, the license plate color recognition task and the license plate type recognition task both use the cross-entropy loss function, and the license plate character location task uses the mean square error loss function.

[0035] The auxiliary supervision function is used to distill the knowledge of the bottleneck module of the auxiliary branch, and then to teach the training of the bottleneck module of the main branch. The auxiliary supervision function adopts the KD divergence loss function.

[0036] Furthermore, the training process of the deep neural network model is as follows:

[0037] Collect license plate images under various scenes, lighting conditions, and angles. Use license plate character segmentation methods to obtain local region images of license plate characters. Label the license plate information in each local region image of a license plate character. The license plate information includes license plate color, license plate type, license plate characters, and the center position of the license plate characters, in order to construct training sample images.

[0038] The training sample images are input into the constructed deep neural network model to train the deep neural network model;

[0039] The teacher model is trained by ignoring the main branch bottleneck module and training the feature extraction module, auxiliary branch bottleneck module, and prediction module based on the main supervision function main_loss.

[0040] Train the student model, ignoring the auxiliary branch bottleneck module, freezing the feature extraction module and the prediction module, and training the main branch bottleneck module based on the main supervision function main_loss;

[0041] Comprehensive supervised training is employed, with the auxiliary branch bottleneck module frozen. The feature extraction module, main branch bottleneck module, and prediction module are trained based on the main supervision function main_loss. Simultaneously, the feature extraction module and main branch bottleneck module are trained using the auxiliary supervision function aux_loss.

[0042] A license plate recognition system based on attention distillation preprocesses the license plate image and then inputs it into a trained deep neural network model to output the license plate recognition result.

[0043] The deep neural network model includes a feature extraction module, a main branch bottleneck module, an auxiliary branch bottleneck module, and a prediction module. The input of the main branch bottleneck module and the auxiliary branch bottleneck module connected in parallel is connected to the output of the feature extraction module, and the output is connected to the input of the prediction module. The feature extraction module inputs the preprocessed license plate image, and the prediction module outputs the license plate recognition result.

[0044] The main branch bottleneck module abstracts license plate image features based on the CNN network structure to obtain the main feature map; the auxiliary branch bottleneck module abstracts the global and local features of the license plate image through self-attention and global attention mechanisms to obtain the auxiliary feature map.

[0045] A computer-readable storage medium storing a plurality of programs for being invoked by a processor and executed as described above for license plate recognition.

[0046] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0047] The advantages of the license plate recognition method, system, and storage medium based on attention distillation provided by this invention are as follows: By utilizing knowledge distillation technology, a heavyweight teacher model based on a transformer decoder has its core global attention knowledge distilled and then taught to a CNN-based student model. Ultimately, a deep neural network model with both local and global attention mechanisms is obtained, directly completing multiple license plate recognition tasks and outputting the overall optimal license plate recognition result. Multiple license plate recognition tasks can be completed with only one deep neural network model, resulting in faster recognition speed, fewer error sources, and more accurate recognition results. It also demonstrates stronger robustness to low-quality license plate images with color casts, dirt, adhesion, missing characters, or insufficient localization. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of the present invention;

[0049] Figure 2 Here is a structural diagram of the feature extraction module;

[0050] Figure 3 This is the main branch bottleneck module structure diagram;

[0051] Figure 4 This is a structural diagram of the bottleneck module for auxiliary branches;

[0052] Figure 5 This is a diagram of the prediction module structure;

[0053] The alphanumeric characters next to each module graphic represent the name of the current feature layer and the size of the feature map of the current feature layer, i.e., feature map height × feature map width × number of feature map channels. Detailed Implementation

[0054] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0055] like Figures 1 to 5 As shown, the present invention proposes a license plate recognition method based on attention distillation, which preprocesses the license plate image and inputs it into a trained deep neural network model to output the license plate recognition result.

[0056] The deep neural network model includes a feature extraction module, a main branch bottleneck module, an auxiliary branch bottleneck module, and a prediction module. The input terminals of the main branch bottleneck module and the auxiliary branch bottleneck module are both connected to the output terminal of the feature extraction module, and the output terminals of the main branch bottleneck module and the auxiliary branch bottleneck module are both connected to the input terminal of the prediction module. The feature extraction module takes the preprocessed license plate image as input, and the prediction module outputs the license plate recognition result.

[0057] The main branch bottleneck module abstracts license plate image features based on the CNN network structure to obtain the main feature map; the auxiliary branch bottleneck module abstracts the global and local features of the license plate image through self-attention and global attention mechanisms to obtain the auxiliary feature map.

[0058] This embodiment utilizes knowledge distillation technology to distill the core global attention knowledge of a heavyweight teacher model based on a transformer decoder and then pass it on to a CNN-based student model. Ultimately, this results in a deep neural network model with both local and global attention mechanisms, directly performing multiple license plate recognition tasks and outputting the overall optimal license plate recognition result. Multiple license plate recognition tasks can be completed with just one deep neural network model, resulting in faster recognition speed, fewer error sources, and more accurate results. It also demonstrates stronger robustness to low-quality license plate images with color casts, dirt, adhesion, missing characters, or insufficient localization. The following is a detailed explanation.

[0059] The method proposed in this invention is applicable to license plate recognition of any type, requiring only the acquisition of a partial image of the entire license plate. For ease of explanation, this embodiment uses Chinese license plates (including: single-layer blue plates, single-layer yellow plates, double-layer yellow plates, military and police plates, school plates, and new energy vehicle plates) as examples.

[0060] A license plate recognition method based on attention distillation includes the following steps:

[0061] S1. Designing a Deep Neural Network Model: The deep neural network model designed in this embodiment mainly utilizes a deep neural network teacher model based on a transformer decoder. Through knowledge distillation, the advantages of the transformer decoder's global attention and global perspective are passed to a CNN-based deep neural network student model, directly assisting in the recognition of various license plate features, including license plate color, license plate type, license plate character position, and license plate character recognition. It outputs the overall optimal license plate recognition result. Since the steps of license plate position correction, license plate character segmentation, and license plate character recognition are no longer explicitly distinguished, the entire license plate character recognition speed is faster, there are fewer sources of error, and the recognition accuracy is higher. This embodiment uses a convolutional neural network (CNN) and transformer structure for deep neural network model design. For ease of description, some terms are defined: feature map resolution refers to feature map height × feature map width; feature map size refers to feature map height × feature map width × number of feature map channels; kernel size refers to kernel width × kernel height; span refers to span in the width direction × span in the height direction. Additionally, each convolutional layer is followed by a batch normalization layer and a non-linear activation layer. Figure 1 As shown, the specific design steps are as follows:

[0062] S11. Design the feature extraction module backbone. This module is primarily used to obtain high-level feature maps with high abstraction and rich expressive power from the input license plate image. The quality of the extracted high-level feature maps directly affects the performance of subsequent license plate recognition. This embodiment uses an improved ResNet classic network as the feature extraction network, such as... Figure 2 As shown, the feature extraction module includes a convolutional layer conv0, a max pooling layer maxpool0, a first residual module, a second residual module, a merging layer eltsum0, a merging layer eltsum1, and a convolutional layer conv1. The convolutional layer conv0 takes the license plate image as input. The convolutional layer conv0, the max pooling layer maxpool0, and the first residual module are sequentially connected and then connected to the input of the merging layer eltsum0. The output of the merging layer eltsum0 is connected to the input of the merging layer eltsum1 and the input of the second residual module. The output of the second residual module is connected to the input of the merging layer eltsum1. The output of the merging layer eltsum1 is connected to the input of the convolutional layer conv1. The convolutional layer conv1 outputs a high-level feature map.

[0063] In addition, the first residual module includes convolutional layers rconv0, rconv1, and rconv2, and the second residual module includes convolutional layers rconv3 and rconv4. Convolutional layers rconv0 and rconv1 are concatenated and then set in parallel with convolutional layer rconv2. The input of the parallel concatenation is connected to the output of the max pooling layer maxpool0, and the output of the parallel concatenation is connected to the input of the merging layer eltsum0. The input of concatenated concatenation of convolutional layers rconv3 and rconv4 is connected to the output of the merging layer eltsum0, and the output of the concatenation is connected to the input of the merging layer eltsum1.

[0064] The input image for this feature extraction module is a 3-channel RGB image with a resolution of 224×72. conv0 is a convolutional layer with a kernel size of 3x3 and a span of 2x2, maxpool0 is a max pooling layer with a kernel size of 2x2 and a span of 2x2, rconv2 is a convolutional layer with a kernel size of 1x1 and a span of 2x2, rconv0 and rconv3 are convolutional layers with a kernel size of 1x1 and a span of 1x1, rconv1 is a convolutional layer with a kernel size of 3x3 and a span of 2x2, rconv4 and conv1 are both convolutional layers with a kernel size of 3x3 and a span of 1x1, and eltsum0 and eltsum1 are merging layers, which combine multiple input feature maps element-wise into a single output feature map. The output of this feature extraction network module is the high-level feature map.

[0065] S12. Design the main branch bottleneck module, main_neck. The main branch bottleneck module further abstracts the license plate image features based on the high-level feature map output by the feature extraction module and the CNN network structure. The specific network structure is shown in Figure 3. The main branch bottleneck module, main_neck, includes a convolutional layer, main_conv, a feature map reshaping layer, main_reshape, and a fully connected layer, main_fc, connected in sequence. The convolutional layer, main_conv, takes the high-level feature map as input; the feature map reshaping layer, main_reshape, is used to adjust the size of the input feature map to meet the needs of subsequent feature layer operations; the fully connected layer, main_fc, is used to adjust the number of channels of the output feature to output the main feature map.

[0066] The feature map represents the high-level feature map output by the feature extraction module. main_conv is a convolutional layer with a kernel size of 3x3 and a span of 2x2. main_reshape is a feature map size transformation layer, whose main function is to adjust the size of the input feature map to meet the needs of subsequent feature layer operations. The main_fc layer is a fully connected layer used to adjust the number of channels of the output feature.

[0067] S13. Design the auxiliary branch bottleneck module aux_neck. The auxiliary branch bottleneck module is based on the high-level feature map output by the feature extraction module. Based on the transformer decoder network structure, it further abstracts the global and local features of the license plate image through self-attention and global attention mechanisms. The specific network structure is shown in Figure 4. The auxiliary branch bottleneck module includes an M-layer decoder structure. The license plate feature query vector output by the previous layer decoder structure is used as the input of the current layer decoder structure. The M-th layer decoder structure outputs an auxiliary feature map.

[0068] Each decoder layer is based on self-attention and global attention mechanisms to abstract the global and local features of the license plate image. The current decoder layer consists of a multi-head self-attention layer, a first additive normalization layer, a multi-head global attention layer, a fully connected layer, a second additive normalization layer, and a third additive normalization layer. The license plate feature query vector output from the previous decoder layer is input into the current multi-head self-attention layer. After processing by the self-attention mechanism, it passes through the first additive normalization layer to obtain the intermediate layer features. The intermediate layer features are then compared with the license plate features output from the previous decoder layer. The query vector is summed pixel by pixel and used as the query Q of the multi-head global attention layer. The high-level feature map and the corresponding positional encoding feature map are summed pixel by pixel and used as the key K of the multi-head global attention layer. The high-level feature map is used as the value V of the multi-head global attention layer. The attention feature output by the multi-head global attention layer and the intermediate layer feature are combined through the second summing and normalization layer to obtain the summed feature. The summed feature is then passed through the fully connected layer and enters the third summing and normalization layer for further summing and normalization. The current layer outputs the license plate feature query vector through the third summing and normalization layer.

[0069] The feature map represents the high-level feature map output by the feature extraction module; feature pos represents the position-encoded feature map corresponding to the feature map, used to represent the spatial position of different pixels in the feature map. The resolution of the position-encoded feature map is the same as the resolution of the high-level feature map output by the feature extraction module. The specific feature value of each position in the feature pos position-encoded feature map is obtained through network training; multi-head attention, multi-head self-attention, FFN, add&norm represent multi-head global attention layer, multi-head self-attention layer, fully connected layer, and additive normalization layer, respectively. Their specific meanings are consistent with the definitions in the ViT (AN IMAGE IS WORTH 16X16WORDS: TRANSFORMERS FORIMAGE RECOGNITION AT SCALE) module; V, K, and Q represent the input components of the attention layer, respectively; + indicates a pixel-wise addition layer between two input feature layers; aux feature queries are the license plate feature query vectors, used to define the encoded feature vectors that the decoding module needs to output. The specific feature values ​​in the queries are obtained through network training; the decoder structure will be executed multiple times (M times). During the first execution, the input auxfeature queries of the decoding module come from the initial target query identifier vector. During subsequent executions, the input auxfeature queries of the decoding module all come from the aux feature queries output in the previous execution. M represents the number of times the execution is repeated. In this embodiment, M = 4.

[0070] S14. Design the prediction module predict_head. The prediction module mainly predicts the output results of multiple tasks based on the license plate feature representation vector obtained in step S12 or S13 (S12 obtains the main feature map, S13 obtains the auxiliary feature map). The specific network structure is shown in Figure 5. The main / aux feature map represents the main feature map output by the bottleneck module of the main branch or the auxiliary feature map output by the bottleneck module of the auxiliary branch. The split layer is a feature channel splitting layer, which is used to split the input feature map into multiple sub-feature maps according to the channel dimension. feature0, feature1, feature2, and feature3 all represent the sub-feature maps after splitting by the feature channel splitting layer. The fc0, fc1, fc2, and fc3 layers are all fully connected layers. Feature0 and fc0 are mainly used for the license plate character recognition task, feature1 and fc1 are mainly used for the license plate color recognition task, feature2 and fc2 are mainly used for the license plate type recognition task, and feature3 and fc3 are mainly used for the license plate character center position localization task.

[0071] S15. Design the supervision function. In this embodiment, the supervision function consists of two parts: the main supervision function `main_loss` and the auxiliary supervision function `aux_loss`. The main supervision function is used to supervise the training of license plate recognition-related tasks. For the license plate character recognition task, the CTC loss function is used, which is the most effective training loss function for recognizing language sentences with contextual information. For the license plate color recognition task and the license plate type recognition task, the cross-entropy loss function is used. For the license plate character location task, the mean squared error (MSE) loss function is used. The auxiliary supervision function is mainly used to distill the knowledge of the bottleneck module in the auxiliary branch, thereby teaching and supervising the training of the bottleneck module in the main branch. The auxiliary supervision function uses the KD divergence loss function, which is the most commonly used supervision function for knowledge distillation.

[0072] S2. Training the deep neural network model mainly involves optimizing the model parameters using a large amount of labeled training sample data to achieve the best recognition performance. The specific steps are as follows:

[0073] S21. Obtain training sample images, mainly by collecting license plate images under various scenes, lighting conditions, and angles. Use existing license plate character segmentation methods to obtain local region images of license plate characters, and then label the information of each license plate, including license plate color, license plate type, license plate characters, and the center position of the license plate characters.

[0074] S22. Training the deep neural network model mainly involves feeding the labeled set of license plate sample images into a predefined deep neural network model to learn the relevant model parameters. This embodiment includes two supervision functions. To accelerate the model's training speed and improve the training effect, the following steps are used for training:

[0075] S221. The training of the teacher model mainly ignores the main branch bottleneck module and trains the feature extraction module, auxiliary branch bottleneck module, and prediction module based on the main supervision function main_loss.

[0076] S222. Training the student model mainly involves ignoring the auxiliary branch bottleneck module, freezing the feature extraction module and the prediction module, and training the main branch bottleneck module based on the main supervision function main_loss.

[0077] S223. Comprehensive supervised training mainly involves freezing the auxiliary branch bottleneck module, training the feature extraction module, the main branch bottleneck module, and the prediction module based on the main supervision function main_loss, and simultaneously training the feature extraction module and the main branch bottleneck module based on the auxiliary supervision function aux_loss.

[0078] S3. Using a deep neural network model: After training the deep neural network model, the next step is to use it in a real-world environment. It's important to note that the auxiliary branch bottleneck module `aux_neck` is only used to assist training. When using the deep neural network model directly, ignore the auxiliary branch bottleneck module `aux_nec`. For any given partial image of a license plate, feed it into the trained deep neural network model and output the license plate recognition result. The specific steps are as follows:

[0079] S31. Detecting local license plate images mainly involves detecting the location of the license plate in the entire image, extracting the local license plate image from the entire image through cropping operations, and normalizing the image size. The main method is to use existing mature methods, which will not be elaborated here.

[0080] S32. License plate recognition: The main method is to feed the normalized local license plate image into a trained deep neural network model, and the final output is the optimal license plate recognition result.

[0081] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An attention distillation-based license plate recognition method, characterized in that, The license plate image is input into a trained deep neural network model after preprocessing, to output a license plate recognition result; The deep neural network model comprises a feature extraction module, a main branch bottleneck module, an auxiliary branch bottleneck module, and a prediction module, the input end of the main branch bottleneck module and the auxiliary branch bottleneck module connected in parallel is connected to the output end of the feature extraction module, and the output end is connected to the input end of the prediction module, the feature extraction module inputs the preprocessed license plate image, and the prediction module outputs the license plate recognition result; The main branch bottleneck module abstracts license plate image features based on a CNN network structure to obtain a main feature map; the auxiliary branch bottleneck module abstracts global features and local features of the license plate image based on a transformer decoder network structure through a self-attention mechanism and a global attention mechanism to obtain an auxiliary feature map; A supervision function is set for the deep neural network model, the supervision function comprising a main supervision function main_loss and an auxiliary supervision function aux_loss, wherein the auxiliary branch bottleneck module is only used for auxiliary training and is not used in the execution process after the deep neural network model is trained; The main supervision function is used to supervise the training of the license plate recognition related task; The auxiliary supervision function is used to distill the knowledge of the auxiliary branch bottleneck module, and then teach the training of the main branch bottleneck module; The training process of the deep neural network model comprises: inputting a training sample image into the deep neural network model constructed, and training the deep neural network model; training a teacher model, ignoring the main branch bottleneck module, training the feature extraction module, the auxiliary branch bottleneck module, and the prediction module based on the main supervision function; training a student model, ignoring the auxiliary branch bottleneck module, freezing the feature extraction module and the prediction module, and training the main branch bottleneck module based on the main supervision function; comprehensive supervision training, freezing the auxiliary branch bottleneck module, training the feature extraction module, the main branch bottleneck module, and the prediction module based on the main supervision function, and simultaneously, training the feature extraction module and the main branch bottleneck module based on the auxiliary supervision function.

2. The attention distillation based license plate recognition method of claim 1, wherein, The feature extraction module is used to obtain a high-level feature map of the license plate image with high abstraction and rich expression ability; the feature extraction module comprises a convolution layer conv0, a maximum value pooling layer maxpool0, a first residual module, a second residual module, a merging layer eltsum0, a merging layer eltsum1, and a convolution layer conv1; The convolution layer conv0 inputs the license plate image, and the convolution layer conv0, the maximum value pooling layer maxpool0, and the first residual module are connected in sequence and connected to the input of the merging layer eltsum0; The output of the merging layer eltsum0 is connected to the input of the merging layer eltsum1 and the input of the second residual module respectively, the output of the second residual module is connected to the input of the merging layer eltsum1, the output of the merging layer eltsum1 is connected to the input of the convolution layer conv1, and the convolution layer conv1 outputs the high-level feature map. 3.The attention distillation based license plate recognition method of claim 2, wherein, The first residual module comprises a convolutional layer rconv0, a convolutional layer rconv1 and a convolutional layer rconv2, and the second residual module comprises a convolutional layer rconv3 and a convolutional layer rconv4; The convolutional layer rconv0 and the convolutional layer rconv1 are connected in series and connected in parallel with the convolutional layer rconv2, the input after parallel connection is connected to the output of the maximum pooling layer maxpool0, and the output after parallel connection is connected to the input of the merging layer eltsum0; The input after series connection of the convolutional layer rconv3 and the convolutional layer rconv4 is connected to the output of the merging layer eltsum0, and the output after series connection is connected to the input of the merging layer eltsum1.

4. The attention distillation based license plate recognition method of claim 2, wherein, The main branch bottleneck module comprises a convolutional layer main_conv, a feature map size transformation layer main_reshape and a fully connected layer main_fc connected in sequence, and the convolutional layer main_conv inputs a high-level feature map; The feature map size transformation layer main_reshape is used to adjust the size of the input feature map to meet the needs of subsequent feature layer operations, and the fully connected layer main_fc is used to adjust the number of channels of the output feature to output a main feature map.

5. The attention distillation based license plate recognition method of claim 2, wherein, The auxiliary branch bottleneck module comprises an M-layer decoder structure, and a license plate feature query vector output by a previous layer of the decoder structure is input into a current layer of the decoder structure, and an auxiliary feature map is output by an Mth layer of the decoder structure; Each layer of the decoder structure is based on a self-attention mechanism and a global attention mechanism to abstract global features and local features of a license plate image, and the current layer of the decoder structure comprises a multi-head self-attention layer, a first addition normalization layer, a multi-head global attention layer, a fully connected layer, a second addition normalization layer and a third addition normalization layer; The license plate feature query vector output by the previous layer of the decoder structure is input into the multi-head self-attention layer of the current layer, and an intermediate layer feature is obtained through the first addition normalization layer after self-attention mechanism processing; The intermediate layer feature is pixel by pixel added to the license plate feature query vector output by the previous layer of the decoder structure to serve as a query Q of the multi-head global attention layer, a position encoding feature map corresponding to a high-level feature map is pixel by pixel added to the high-level feature map to serve as a key K of the multi-head global attention layer, and the high-level feature map serves as a value V of the multi-head global attention layer, an attention feature output by the multi-head global attention layer is added to the intermediate layer feature through the second addition normalization layer to obtain an added feature, and the added feature is input into the third addition normalization layer through a de-fully connected layer to be added and normalized again, and the current layer outputs a license plate feature query vector through the third addition normalization layer.

6. The attention distillation based license plate recognition method of claim 1, wherein, The prediction module comprises a feature channel splitting layer split and a fully connected layer fc; The feature channel splitting layer split splits the input main feature map or auxiliary feature map into a plurality of sub-feature maps according to the channel dimension, the sub-feature maps are input into the fully connected layer fc to obtain a license plate recognition result, and the number of the fully connected layers fc corresponds to the number of the sub-feature maps.

7. The attention distillation based license plate recognition method of claim 1, wherein, The license plate character recognition task adopts a CTC loss function, the license plate color recognition task and the license plate type recognition task adopt a cross-entropy loss function, and the license plate character position positioning task adopts a mean square error loss function; The auxiliary supervision function adopts a KD divergence loss function.

8. An attention distillation based license plate recognition system, characterized in that, The license plate image is input into a trained deep neural network model after preprocessing to output a license plate recognition result. The deep neural network model comprises a feature extraction module, a main branch bottleneck module, an auxiliary branch bottleneck module, and a prediction module. The input end of the main branch bottleneck module and the auxiliary branch bottleneck module connected in parallel is connected to the output end of the feature extraction module, and the output end is connected to the input end of the prediction module. The feature extraction module inputs the preprocessed license plate image, and the prediction module outputs the license plate recognition result. The main branch bottleneck module abstracts license plate image features based on a CNN network structure to obtain a main feature map. The auxiliary branch bottleneck module abstracts global features and local features of the license plate image based on a transformer decoder network structure through a self-attention mechanism and a global attention mechanism to obtain an auxiliary feature map. A supervision function is set for the deep neural network model, which comprises a main supervision function main_loss and an auxiliary supervision function aux_loss. The auxiliary branch bottleneck module is only used for auxiliary training and is not used in the execution process after the deep neural network model is trained. The main supervision function is used to supervise the training of license plate recognition related tasks. The auxiliary supervision function is used to distill the knowledge of the auxiliary branch bottleneck module to supervise the training of the main branch bottleneck module. The training process of the deep neural network model comprises: inputting a training sample image into a constructed deep neural network model to train the deep neural network model; training a teacher model, ignoring the main branch bottleneck module, training the feature extraction module, the auxiliary branch bottleneck module, and the prediction module based on the main supervision function; training a student model, ignoring the auxiliary branch bottleneck module, freezing the feature extraction module and the prediction module, and training the main branch bottleneck module based on the main supervision function; comprehensive supervision training, freezing the auxiliary branch bottleneck module, training the feature extraction module, the main branch bottleneck module, and the prediction module based on the main supervision function, and simultaneously training the feature extraction module and the main branch bottleneck module based on the auxiliary supervision function.

9. A computer readable storage medium, characterized in that, The computer readable storage medium stores a plurality of programs for being called and executed by the processor to implement the license plate recognition method based on attention distillation according to any one of claims 1 to 7.

Images