License plate character recognition method and system based on multi-modal information and storage medium
Through the deep neural network model of multimodal information, combined with the knowledge distillation technology of the teacher model and the student model, the problem of low-quality license plate image recognition is solved, and more accurate and robust license plate character recognition is achieved.
Patent Information
- Application Number
- CN202410964633.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-18
AI Technical Summary
Existing license plate recognition technology is difficult to accurately identify license plate characters under low-quality license plate images (such as low resolution, lighter or missing parts of characters, blurred edges, tilted characters, etc.), affecting the overall performance.
Using a deep neural network model based on multimodal information, the knowledge distillation technology of the teacher model and the student model is combined with visual encoder and text encoder to perform cross-modal attention fusion, extract multimodal features of license plate images, and realize license plate Character recognition.
It improves the accuracy and robustness of license plate character recognition, and can recognize faster, have fewer errors and more accurate recognition results in low-quality license plate images with color cast, defilement, sticking, character loss, and inaccurate positioning.
Smart Images

Figure CN118941292A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a license plate character recognition method, system and storage medium based on multimodal information. Background Art
[0002] License plate character recognition refers to accurately identifying all Chinese characters, letters and numbers on a license plate in an image with a known license plate position. The quality of license plate character recognition directly determines the overall performance of license plate recognition technology. It specifically includes the following technical steps: license plate position correction, license plate character segmentation, license plate character recognition, etc.
[0003] License plate position correction refers to the spatial transformation of the initially detected license plate in an undesirable position, so that it becomes a license plate in an ideal position, which facilitates the subsequent accurate character segmentation. Spatial transformation includes translation transformation, rotation transformation, scaling transformation, shear transformation, perspective transformation, etc. Commonly used correction methods are as follows:
[0004] (1) Methods based on line detection. Representative methods include "correction method based on Hough line detection" and "correction method based on Radon line detection". The principle is to directly detect the straight line on the license plate and correct the license plate position according to the inclination angle of the straight line.
[0005] (2) Methods based on traversal search, the representative method is "Correction method based on rotation projection". Its principle is to first rotate the license plate to each allowed angle position, then perform projection to obtain the corresponding eigenvalues, and obtain the best eigenvalue by comparison. At this time, the corresponding angle is the best license plate tilt angle.
[0006] (3) Methods based on feature analysis. Representative methods include "correction method based on principal component analysis" and "correction method based on least squares method". This type of method directly analyzes the grayscale image or binary image as a whole to obtain the optimal correction parameters in the overall sense.
[0007] Character segmentation refers to accurately segmenting each single character in an image with a known license plate position. There are mainly the following methods:
[0008] (1) A method based on vertical projection: This method obtains the vertical projection curve of the license plate characters and obtains the edge position of each character according to the peak and trough positions of the curve.
[0009] (2) A method based on connected region analysis: This method first binarizes the license plate image, uses the feature that individual characters are simply connected regions for analysis, and finally obtains the position of the characters.
[0010] (3) Methods based on machine learning, such as “A license plate character segmentation method based on support vector machine”. This type of method obtains the layout characteristics of the license plate, uses a classifier for training and learning, and finally completes the segmentation of the license plate characters.
[0011] License plate character recognition refers to identifying the true meaning of individual characters that have been accurately segmented. Commonly used methods include the following:
[0012] (1) Global features: This type of features uses global transformation to obtain the overall features of the characters, and uses ordered overall features or subset features to form feature vectors. Common features include GABOR transformation features, moment features, projection features, stroke density features, HARR features, HOG features, etc. The advantages of these features are that they are insensitive to local changes and have strong anti-interference capabilities; their disadvantages are that they are easy to ignore some important local features and cannot distinguish similar characters.
[0013] (2) Local features: This type of feature calculates the corresponding features in multiple local areas of a character and uses the serialized ordered local features to form the final feature vector. The main features include local grayscale histogram features, LBP features, threading features, SIFT features, etc. The advantage of this type of feature is its strong ability to distinguish characters; its disadvantage is that it pays too much attention to the local features of the characters and often misclassifies characters with noise interference.
[0014] The above technologies can achieve very good results for clear license plate images. However, the license plate images collected in the actual environment often have low resolution, some characters are faded or missing, edges are blurred, characters are tilted, etc., which makes it difficult to accurately correct the license plate position, segment the license plate characters, and recognize the license plate characters. Some even fail, which seriously affects the overall performance of license plate recognition. Therefore, how to accurately and robustly recognize license plate characters is still a difficulty for domestic license plate recognition systems.
[0015] In recent years, multimodal large model technology has received widespread attention and application in various fields due to its ability to focus on both text features and image features at the same time and provide richer and more comprehensive feature information. However, the disadvantage of this technology is that the model consumes a lot of memory and has a large amount of computation, so it requires a lot of memory and computing power. Summary of the invention
[0016] Based on the technical problems existing in the background technology, the present invention proposes a license plate character recognition method, system and storage medium based on multimodal information. Only one deep neural network model is needed to complete the recognition of license plate characters, and the recognition result is more accurate and more robust.
[0017] The license plate character recognition method based on multimodal information proposed in the present invention pre-processes the license plate image and inputs it into a trained deep neural network model to output the license plate recognition result;
[0018] The deep neural network model includes a prediction module, a student model and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are transferred to the student model through knowledge distillation technology to complete the recognition of license plate characters.
[0019] The input of the teacher model is the license plate image and the license plate description text. The license plate image is fine-tuned through the visual encoder and then cross-modal attention fused with the license plate description text to obtain the visual features of the license plate image with text feature information; the input of the student model is the license plate image, and the license plate image is feature extracted to obtain a high-level feature map; the visual features of the license plate image and the high-level feature map are used as the input of the prediction module to output the recognition results of the license plate characters.
[0020] Furthermore, the teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn;
[0021] The input of the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image;
[0022] The visual encoder CLIP_V inputs the license plate image, the input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V, and the output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn. The cross-modal attention fusion module Crossattn outputs the visual features of the license plate image.
[0023] Furthermore, the cross-modal attention fusion module Crossattn includes a feature extraction layer select, an image text attention module and a text image attention module, the image features of the license plate image are input into the feature extraction layer select as an input feature sequence, and the feature extraction layer select extracts the position features of the last feature sequence as the global information features of the license plate image;
[0024] The global information features of the license plate image and the text features of the license plate image are input into the image-text attention module to integrate the image feature information into the text features, and the image-text features are output;
[0025] The image text features and the image features of the license plate image are input into the text-image attention module to integrate the text feature information into the image features, and the visual features of the license plate image are output.
[0026] Furthermore, the image-text attention module includes an element-by-element multiplication operation layer attn_it, an activation function layer softmax_it and a weighted operation layer attn_mul_it connected in sequence, the input of the element-by-element multiplication operation layer attn_it is connected to the output of the feature extraction layer select, the element-by-element multiplication operation layer attn_it and the weighted operation layer attn_mul_it both input the text features of the license plate image, and the weighted operation layer attn_mul_it outputs the image text features.
[0027] Furthermore, the text image attention module includes a matrix multiplication operation layer attn_ti, an activation function layer softmax_ti and a weighted operation layer attn_mul_ti connected in sequence, the matrix multiplication operation layer attn_ti inputs the image features and image text features of the license plate image, the weighted operation layer attn_mul_ti inputs the image features of the license plate image and the output features of the activation function layer softmax_ti, and the weighted operation layer attn_mul_ti outputs.
[0028] Furthermore, the prediction module includes a fully connected layer Fuse and a fully connected layer Fc connected in sequence; the input of the fully connected layer Fuse is respectively connected to the output of the student model and the output of the cross-modal attention fusion module Crossattn, and the fully connected layer Fc outputs the recognition result of the card characters.
[0029] Furthermore, the student model Backbone includes a convolution layer conv0, a maximum pooling layer maxpool, a convolution layer rconv0, a convolution layer rconv1, a convolution layer rconv2, a convolution layer rconv3, a convolution layer rconv4, a merging layer eltsum0, a merging layer eltsum1, a convolution layer conv1, and a feature size adjustment layer reshape;
[0030] The convolution layer conv0 inputs the license plate image, the input of the maximum pooling layer maxpool is connected to the output of the convolution layer conv0, the output of the maximum pooling layer maxpool is connected to the input of the convolution layer rconv0 and the input of the convolution layer rconv2 respectively, the output of the convolution layer rconv0 and the convolution layer rconv1 are connected in series and the output of the convolution layer rconv2 are connected to the input of the merging layer eltsum0, the output of the merging layer eltsum0 is connected to the input of the merging layer eltsum1 and the input of the convolution layer rconv3 respectively, the input of the convolution layer rconv4 is connected to the output of the convolution layer rconv3, and the output is connected to the input of the merging layer eltsum1; the merging layer eltsum1, the convolution layer conv1, and the feature size adjustment layer reshape are connected in series, and the feature size adjustment layer reshape outputs a high-level feature map.
[0031] Furthermore, the training process of the deep neural network model is as follows:
[0032] Collect license plate images in various scenes, various lights and various angles, use the license plate character segmentation method to obtain the license plate character local area image, and mark the license plate information in each license plate character local area image, the license plate information includes license plate color, license plate type, license plate character, and license plate character center position, so as to construct a training sample image;
[0033] Input the training sample images into the constructed deep neural network model to train the deep neural network model;
[0034] Train the teacher model, ignore the student model, freeze the visual encoder CLIP_V and the text encoder CLIP_T, train the linear projection layer Finetune, the cross-modal attention fusion module Crossattn and the prediction module based on the main supervision function main_loss, and after the teacher model training converges, unfreeze the visual encoder CLIP_V, and train the visual encoder CLIP_V, the linear projection layer Finetune, the cross-modal attention fusion module Crossattn and the prediction module based on the main supervision function main_loss; the main supervision function main_loss is used to supervise the training of the license plate character recognition task, using the ctc loss function;
[0035] Train the student model, freeze the teacher model and prediction module, and train the student model Backbone based on the main supervision function main_loss;
[0036] Comprehensive supervised training, freezing the teacher model, training the student model and the prediction module based on the main supervision function main_loss, and assisting the training of the student model Backbone based on the auxiliary supervision function aux_loss. The auxiliary supervision function aux_loss is used to distill the knowledge of the auxiliary teacher model, and then teach the training of the supervised student model. The auxiliary supervision function aux_loss uses the KD divergence loss function.
[0037] Furthermore, the license plate image is preprocessed and then input into the trained deep neural network model to output the license plate recognition result;
[0038] The deep neural network model includes a prediction module, a student model and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are transferred to the student model through knowledge distillation technology to complete the recognition of license plate characters.
[0039] The input of the teacher model is the license plate image and the license plate description text. The license plate image is fine-tuned through the visual encoder and then cross-modal attention fused with the license plate description text to obtain the visual features of the license plate image with text feature information; the input of the student model is the license plate image, and the license plate image is feature extracted to obtain a high-level feature map; the visual features of the license plate image and the high-level feature map are used as the input of the prediction module to output the recognition results of the license plate characters.
[0040] A computer-readable storage medium stores a plurality of programs, which are used to be called by a processor and execute the license plate character recognition method as described above.
[0041] The advantages of the license plate character recognition method, system and storage medium based on multimodal information provided by the present invention are: a heavyweight teacher model based on CLIP multimodal encoder is distilled to extract its core multimodal information knowledge, and the knowledge is passed to the student model based on CNN, and finally, a deep neural network model with the ability to obtain multimodal information is obtained. Only one deep neural network model is needed to complete the recognition of license plate characters, with faster recognition speed, fewer sources of error, more accurate recognition results, and stronger robustness for low-quality license plate images with color cast, stains, adhesion, missing characters, and inaccurate positioning. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A structural diagram of the deep neural network model designed for the present invention;
[0043] Figure 2 A structural diagram of the student model;
[0044] Figure 3 Figure 2 is the structural diagram of the cross-modal attention fusion module. DETAILED DESCRIPTION
[0045] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.
[0046] like Figures 1 to 3 As shown, the license plate character recognition method based on multimodal information proposed in the present invention preprocesses the license plate image and inputs it into the trained deep neural network model to output the license plate recognition result; the license plate image preprocessing method is detailed in step S31.
[0047] The deep neural network model includes a prediction module, a student model and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are passed to the student model through knowledge distillation technology to complete the recognition of license plate characters; the input of the teacher model is the license plate image and the license plate description text. The license plate image is fine-tuned by the visual encoder and then cross-modal attention fused with the license plate description text to obtain the visual features of the license plate image with text feature information; the input of the student model is the license plate image, and the license plate image is subjected to feature extraction to obtain a high-level feature map; the license plate image visual features and the high-level feature map are used as the input of the prediction module to output the recognition result of the license plate character.
[0048] This embodiment distills the core multimodal information knowledge from a heavyweight teacher model based on CLIP multimodal encoder, and teaches it to the student model based on CNN, and finally obtains a deep neural network model with the ability to obtain multimodal information. Only one deep neural network model is needed to complete the recognition of license plate characters, with faster recognition speed, fewer error sources, more accurate recognition results, and stronger robustness for low-quality license plate images with color cast, stains, adhesions, missing characters, and inaccurate positioning.
[0049] The method proposed in this embodiment is applicable to the character recognition of any standard license plate, and only the partial image of the entire license plate needs to be obtained. In order to facilitate the description of the present invention, this embodiment takes Chinese license plates (including: single-layer blue plates, single-layer yellow plates, double-layer yellow plates, military and police plates, school plates, and new energy license plates) as an example for description, and the input image is a 3-channel RGB image with an image resolution of 224×64.
[0050] The license plate character recognition method based on multimodal information includes the following steps:
[0051] S1. Design a deep neural network model. The deep neural network model designed in this embodiment mainly relies on a teacher model based on a multimodal encoder. Through the knowledge distillation technology, the multimodal feature advantages of the multimodal encoder are passed to the deep neural network student model based on CNN to directly complete the recognition of license plate characters. Since the steps of license plate position correction, license plate character segmentation, and license plate character recognition are no longer deliberately distinguished, the entire license plate character recognition speed is faster, the error sources are fewer, and the recognition accuracy is higher. This embodiment uses a convolutional neural network (CNN) structure and a transformer structure to design a deep neural network model. In order to facilitate the description of the present invention, some terms are defined: feature map resolution refers to feature map height × feature map width, feature map size refers to feature map height × feature map width × number of feature map channels, kernel size refers to kernel width × kernel height, span refers to width direction span × height direction span, and each convolution layer is followed by a batch normalization layer and a nonlinear activation layer. Figure 1As shown, the specific design steps are as follows:
[0052] S11. Design a teacher model. The teacher model designed in this embodiment is mainly based on the existing multimodal encoder CLIP (Learning Transferable Visual Models From Natural Language Supervision) pre-training model. First, the visual encoding features are fine-tuned, and then cross-modal attention is fused with the text encoding features to finally obtain the visual features of the license plate image with text feature information.
[0053] The teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn; the input of the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image; the visual encoder CLIP_V inputs the license plate image, the input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V, and the output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn, and the cross-modal attention fusion module Crossattn outputs the visual features of the license plate image.
[0054] The text encoder CLIP_T only outputs the global information features of the last sequence position, and the visual encoder CLIP_V outputs the global information features and all local patch (image slice) information features. Finetune is the fine-tuning layer of the visual features, which is a linear projection layer. Its main functions are: first, to adjust the number of channels of the input features to keep the feature dimensions of the visual features and text features consistent; second, to fine-tune the output features of the visual encoder CLIP_V to obtain visual features that are more in line with the current license plate character recognition task. Crossattn is a cross-modal attention fusion module, which mainly completes the cross-modal fusion of visual features and text features to obtain the visual features of license plate images with text feature information. The specific network results are as follows: Figure 3 As shown:
[0055] The cross-modal attention fusion module Crossattn includes a feature extraction layer select, an image text attention module and a text image attention module. The image features of the license plate image are input into the feature extraction layer select as an input feature sequence. The feature extraction layer select extracts the position features of the last feature sequence as the global information features of the license plate image; the global information features of the license plate image and the text features of the license plate image are input into the image text attention module to integrate the image feature information into the text features, and output the image text features; the image text features and the image features of the license plate image are input into the text image attention module to integrate the text feature information into the image features, and output the visual features of the license plate image. Among them, the feature extraction layer select is mainly used to extract the last sequence position feature from the input feature sequence as the global information feature of the input image; attn_mul_it is a weighted operation layer for sequence features one by one according to the last dimension.
[0056] The image-text attention module is used to integrate image feature information into text features. It specifically includes the element-wise multiplication operation layer attn_it, the activation function layer softmax_it and the weighted operation layer attn_mul_it connected in sequence. The input of the element-wise multiplication operation layer attn_it is connected to the output of the feature extraction layer select. The element-wise multiplication operation layer attn_it and the weighted operation layer attn_mul_it both input the text features of the license plate image, and the weighted operation layer attn_mul_it outputs the image text features.
[0057] The text image attention module is used to integrate text feature information into image features, specifically including the matrix multiplication operation layer attn_ti, the activation function layer softmax_ti and the weighted operation layer attn_mul_ti connected in sequence. The matrix multiplication operation layer attn_ti inputs the image features and image text features of the license plate image, the weighted operation layer attn_mul_ti inputs the image features of the license plate image and the output features of the activation function layer softmax_ti, and the weighted operation layer attn_mul_ti outputs the visual features of the license plate image.
[0058] S12, design the student model Backbone, which is also the feature extraction module that this embodiment ultimately wants. It is mainly used to obtain high-level features with rich expressive capabilities of the input license plate image. The quality of high-level feature extraction directly affects the performance of subsequent license plate recognition. The specific network structure is as follows Figure 2As shown in Figure 1, the student model Backbone includes convolution layer conv0, maximum pooling layer maxpool, convolution layer rconv0, convolution layer rconv1, convolution layer rconv2, convolution layer rconv3, convolution layer rconv4, merging layer eltsum0, merging layer eltsum1, convolution layer conv1, and feature size adjustment layer reshape; the convolution layer conv0 inputs the license plate image, the input of the maximum pooling layer maxpool is connected to the output of the convolution layer conv0, and the output of the maximum pooling layer maxpool is connected to the input of the convolution layer rconv0 and the convolution layer rconv2 respectively. The input of conv2, the output of the convolution layer rconv0 and the convolution layer rconv1 set in series and the output of the convolution layer rconv2 are all connected to the input of the merging layer eltsum0, the output of the merging layer eltsum0 is respectively connected to the input of the merging layer eltsum1 and the input of the convolution layer rconv3, the input of the convolution layer rconv4 is connected to the output of the convolution layer rconv3, and the output is connected to the input of the merging layer eltsum1; the merging layer eltsum1, the convolution layer conv1, and the feature size adjustment layer reshape are set in series, and the feature size adjustment layer reshape outputs a high-level feature map.
[0059] conv0 is a convolutional layer with a kernel size of 3x3 and a stride of 2x2, maxpool is a maximum pooling layer with a kernel size of 2x2 and a stride of 2x2; rconv2 is a convolutional layer with a kernel size of 1x1 and a stride of 2x2, rconv0 and rconv3 are convolutional layers with a kernel size of 1x1 and a stride of 1x1, rconv1 is a convolutional layer with a kernel size of 3x3 and a stride of 2x2, rconv4 is a convolutional layer with a kernel size of 3x3 and a stride of 1x1, and conv1 is a convolutional layer with a kernel size of 3x3 and a stride of 2x1; eltsum0 and eltsum1 are both merging layers, which add and merge multiple input feature maps into an output feature map according to corresponding elements; reshape layer is a feature size adjustment layer, which is used to adjust the input feature size; the output of the student model is the high-level feature map image feature.
[0060] S13. Design a prediction module. The prediction module mainly predicts the output result of the license plate characters based on the license plate visual feature vector (high-level feature map or high-level feature map) obtained in step S11 or step S12. The module is mainly composed of a Fuse layer and an Fc layer. Both the Fuse layer and the Fc layer are fully connected layers. The fully connected layer Fuse is mainly used to fuse the input multimodal features, and the fully connected layer Fc is mainly used for recognition and prediction of license plate characters.
[0061] S14. Design a supervision function. The supervision function in this embodiment consists of two parts, namely the main supervision function main_loss and the auxiliary supervision function aux_loss. The main supervision function is used to supervise the training of the license plate character recognition task, and the ctc loss function (ctc_loss) is adopted. The ctc_loss is the most effective training loss function for recognizing language sentences with context information relationships. The auxiliary supervision function is mainly used to distill the knowledge of the teacher module and then teach and supervise the training of the student model. The KD divergence loss function is adopted for the auxiliary supervision function, and this loss function is the most commonly used supervision function for knowledge distillation.
[0062] S2. Train a deep neural network model. Specifically, by using a large number of labeled training sample data, optimize the parameters of the deep neural network model to make the recognition performance of the deep neural network model optimal. The specific steps are as follows:
[0063] S21. Obtain training sample data. Mainly collect license plate images in various scenarios, under various lighting conditions and at various angles, use existing license plate detection methods to obtain license plate local area images, and then label the information of each license plate, including license plate characters and license plate comprehensive description text. Among them, the template adopted for the license plate comprehensive description text is "a xxx license plate with chars of **********", where xxx represents any adjective, and Chinese characters on the license plate are represented by pinyin. For example, "a blurry double yellow chinese license plate with chars of jingA123456" represents "a blurred Chinese double yellow license plate with the license number 'jingA123456'".
[0064] S22. Train the deep neural network model. Mainly send the labeled license plate sample image set into the defined deep neural network model to learn the relevant model parameters. This embodiment includes two supervision functions. To accelerate the training speed of the model and improve the training effect, the following steps are specifically adopted for training:
[0065] S221. Train the teacher model. Mainly ignore the student model. First, freeze the visual encoder CLIP_V and the text encoder CLIP_T, and train the linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module based on the main supervision function main_loss. After the teacher model training converges, unfreeze the visual encoder CLIP_V and train the visual encoder CLIP_V, the linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module based on the main supervision function main_loss.
[0066] S222, train the student model, freeze the teacher model and the prediction module, and train the student model Backbone based on the main supervision function main_loss.
[0067] S223, comprehensive supervision training, freeze the teacher model, train the student model and prediction module based on the main supervision function main_loss, and assist in training the student model Backbone based on the auxiliary supervision function aux_loss.
[0068] S3. Use the deep neural network model. After training the deep neural network model, the next step is to use the model in a real environment. It should be noted that the teacher module is only used to assist training. When the deep neural network model is needed, the teacher module can be ignored. For any given partial image of a license plate, send it to the trained deep neural network model and output the license plate recognition result. The specific steps are as follows:
[0069] S31. Detect the partial license plate image, mainly detecting the position of the license plate in the whole image, extracting the partial license plate image from the whole image through cropping operation, and normalizing the image size, the main method is to use the existing mature method.
[0070] S32. To recognize license plates, the main method is to send the normalized local license plate image into the trained deep neural network model, and the final output result is the optimal license plate recognition result.
[0071] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A license plate character recognition method based on multimodal information, characterized in that: After preprocessing the license plate image, it is input into the trained deep neural network model to output the license plate recognition result; The deep neural network model includes a prediction module, a student model and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are transferred to the student model through knowledge distillation technology to complete the recognition of license plate characters. The input of the teacher model is the license plate image and the license plate description text. The license plate image is fine-tuned through the visual encoder and then cross-modal attention fused with the license plate description text to obtain the visual features of the license plate image with text feature information; the input of the student model is the license plate image, and the license plate image is feature extracted to obtain a high-level feature map; the visual features of the license plate image and the high-level feature map are used as the input of the prediction module to output the recognition results of the license plate characters.
2. The license plate character recognition method based on multimodal information according to claim 1 is characterized in that: The teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn; The input of the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image; The visual encoder CLIP_V inputs the license plate image, the input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V, and the output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn. The cross-modal attention fusion module Crossattn outputs the visual features of the license plate image.
3. The license plate character recognition method based on multimodal information according to claim 2 is characterized in that: The cross-modal attention fusion module Crossattn includes a feature extraction layer select, an image text attention module and a text image attention module. The image features of the license plate image are input into the feature extraction layer select as an input feature sequence. The feature extraction layer select extracts the position features of the last feature sequence as the global information features of the license plate image. The global information features of the license plate image and the text features of the license plate image are input into the image-text attention module to integrate the image feature information into the text features, and the image-text features are output; The image text features and the image features of the license plate image are input into the text-image attention module to integrate the text feature information into the image features, and the visual features of the license plate image are output.
4. The license plate character recognition method based on multimodal information according to claim 3 is characterized in that: The image text attention module includes an element-by-element multiplication operation layer attn_it, an activation function layer softmax_it and a weighted operation layer attn_mul_it which are connected in sequence. The input of the element-by-element multiplication operation layer attn_it is connected to the output of the feature extraction layer select. The element-by-element multiplication operation layer attn_it and the weighted operation layer attn_mul_it both input the text features of the license plate image, and the weighted operation layer attn_mul_it outputs the image text features.
5. The license plate character recognition method based on multimodal information according to claim 4 is characterized in that: The text image attention module includes a matrix multiplication operation layer attn_ti, an activation function layer softmax_ti and a weighted operation layer attn_mul_ti connected in sequence. The matrix multiplication operation layer attn_ti inputs the image features and image text features of the license plate image, the weighted operation layer attn_mul_ti inputs the image features of the license plate image and the output features of the activation function layer softmax_ti, and the weighted operation layer attn_mul_ti outputs.
6. The license plate character recognition method based on multimodal information according to claim 2 is characterized in that: The prediction module includes a fully connected layer Fuse and a fully connected layer Fc connected in sequence; the input of the fully connected layer Fuse is respectively connected to the output of the student model and the output of the cross-modal attention fusion module Crossattn, and the fully connected layer Fc outputs the recognition result of the card characters.
7. The license plate character recognition method based on multimodal information according to claim 1 is characterized in that: The student model Backbone includes a convolution layer conv0, a maximum pooling layer maxpool, a convolution layer rconv0, a convolution layer rconv1, a convolution layer rconv2, a convolution layer rconv3, a convolution layer rconv4, a merging layer eltsum0, a merging layer eltsum1, a convolution layer conv1, and a feature size adjustment layer reshape; The convolution layer conv0 inputs the license plate image, the input of the maximum pooling layer maxpool is connected to the output of the convolution layer conv0, the output of the maximum pooling layer maxpool is connected to the input of the convolution layer rconv0 and the input of the convolution layer rconv2 respectively, the output of the convolution layer rconv0 and the convolution layer rconv1 are connected in series and the output of the convolution layer rconv2 are connected to the input of the merging layer eltsum0, the output of the merging layer eltsum0 is connected to the input of the merging layer eltsum1 and the input of the convolution layer rconv3 respectively, the input of the convolution layer rconv4 is connected to the output of the convolution layer rconv3, and the output is connected to the input of the merging layer eltsum1; the merging layer eltsum1, the convolution layer conv1, and the feature size adjustment layer reshape are connected in series, and the feature size adjustment layer reshape outputs a high-level feature map.
8. The license plate character recognition method based on multimodal information according to claim 2 is characterized in that: The training process of the deep neural network model is as follows: Collect license plate images in various scenes, various lights and various angles, use the license plate character segmentation method to obtain the license plate character local area image, and mark the license plate information in each license plate character local area image, the license plate information includes license plate color, license plate type, license plate character, and license plate character center position, so as to construct a training sample image; Input the training sample images into the constructed deep neural network model to train the deep neural network model; Train the teacher model, ignore the student model, freeze the visual encoder CLIP_V and the text encoder CLIP_T, train the linear projection layer Finetune, the cross-modal attention fusion module Crossattn and the prediction module based on the main supervision function main_loss, and after the teacher model training converges, unfreeze the visual encoder CLIP_V, and train the visual encoder CLIP_V, the linear projection layer Finetune, the cross-modal attention fusion module Crossattn and the prediction module based on the main supervision function main_loss; the main supervision function main_loss is used to supervise the training of the license plate character recognition task, using the ctc loss function; Train the student model, freeze the teacher model and prediction module, and train the student model Backbone based on the main supervision function main_loss; Comprehensive supervised training, freezing the teacher model, training the student model and the prediction module based on the main supervision function main_loss, and assisting the training of the student model Backbone based on the auxiliary supervision function aux_loss. The auxiliary supervision function aux_loss is used to distill the knowledge of the auxiliary teacher model, and then teach the training of the supervised student model. The auxiliary supervision function aux_loss uses the KD divergence loss function.
9. A license plate character recognition system based on multimodal information, characterized in that: After preprocessing the license plate image, it is input into the trained deep neural network model to output the license plate recognition result; The deep neural network model includes a prediction module, a student model and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are transferred to the student model through knowledge distillation technology to complete the recognition of license plate characters. The input of the teacher model is the license plate image and the license plate description text. The license plate image is fine-tuned through the visual encoder and then cross-modal attention fused with the license plate description text to obtain the visual features of the license plate image with text feature information; the input of the student model is the license plate image, and the license plate image is feature extracted to obtain a high-level feature map; the visual features of the license plate image and the high-level feature map are used as the input of the prediction module to output the recognition results of the license plate characters.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of programs, which are used to be called by a processor and execute the license plate character recognition method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
License plate recognition system
CN112308066A
License plate recognition method, device, electronic equipment and storage medium
CN112364864A
License plate detection and identification method based on knowledge distillation guided by key foreground features
CN115861997A
License plate detection and recognition method and system, storage medium and equipment
CN116259049A
Scene text recognition method and system, storage medium and computing equipment
CN116486391A
Cited By
Improved SwinTextSpotter v2-based robust generalization detection and identification method for plant station wiring image text
CN119992563A
Platform license plate recognition method and system
CN120932221A
