License plate character recognition method and system based on multi-modal information and storage medium
By using a deep neural network model based on multimodal information and knowledge distillation technology to extract features from a heavyweight encoder, the accuracy and robustness issues of low-quality license plate image recognition are solved, and fast and accurate license plate character recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI TSINGLINK INFORMATION TECH
- Filing Date
- 2024-07-18
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies lack accuracy and robustness when recognizing low-quality license plate images, especially those with color cast, dirt, adhesion, or missing characters, thus affecting the overall performance of the license plate recognition system.
A deep neural network model based on multimodal information is adopted. Multimodal features are extracted from the heavyweight CLIP multimodal encoder through knowledge distillation technology and passed to the student model to realize the recognition of license plate characters. The model includes a prediction module, a student model and a teacher model. Feature fusion is performed using cross-modal attention fusion and knowledge distillation technology.
It achieves fast and accurate recognition of low-quality license plate images, improves robustness, reduces error sources, and is suitable for license plate recognition under various lighting and angle conditions.
Smart Images
Figure CN118941292B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition technology, and in particular to a method, system, and storage medium for license plate character recognition based on multimodal information. Background Technology
[0002] License plate character recognition refers to the accurate and complete identification of all Chinese characters, symbols, and numbers on a license plate in an image with a known license plate location. The quality of license plate character recognition directly determines the overall performance of the license plate recognition technology. Specific technical steps include: license plate location correction, license plate character segmentation, and license plate character recognition.
[0003] License plate position correction refers to spatial transformation of initially detected license plates that are not in an ideal position, so as to make them ideal for subsequent accurate character segmentation. Spatial transformations include translation, rotation, scaling, shearing, perspective, etc. Commonly used correction methods include the following:
[0004] (1) Methods based on line detection, such as "correction method based on Hough line detection" and "correction method based on Radon line detection", are based on the principle of directly detecting the straight line on the license plate and correcting the license plate position according to the tilt angle of the line.
[0005] (2) The method based on traversal search is represented by the "correction method based on rotation projection". The principle is to first rotate the license plate to each allowed angle position, then project it to obtain the corresponding feature value, and obtain the best feature value by comparison. The corresponding angle is the best license plate tilt angle.
[0006] (3) Feature-based methods, such as “principal component analysis-based correction method” and “least squares-based correction method”, directly analyze the grayscale image or binarized image as a whole to obtain the optimal correction parameters in an overall sense.
[0007] Character segmentation refers to accurately segmenting each individual character in an image where the location of a license plate is known. The main methods include:
[0008] (1) The method based on vertical projection obtains the vertical projection curve of the license plate characters and obtains the edge position of each character according to the peak and trough positions of the curve.
[0009] (2) The method based on connected component analysis first binarizes the license plate image, then analyzes it using the feature that each character is a single connected component, and finally obtains the position of the character.
[0010] (3) Machine learning-based methods, such as “a method for segmenting license plate characters based on support vector machine”, which obtains the layout pattern features of license plates, trains and learns with the help of classifiers, and finally completes the segmentation of license plate characters.
[0011] License plate character recognition refers to identifying the true meaning of a single, precisely segmented character. Commonly used methods include the following:
[0012] (1) Global features: This type of feature uses global transformation to obtain the overall features of the character. It uses ordered global features or subset features to construct the feature vector. Common features include GABOR transform features, moment features, projection features, stroke density features, HARR features, HOG features, etc. The advantage of these features is that they are not sensitive to local changes and have strong anti-interference ability; the disadvantage is that they are easy to ignore some important local features and cannot distinguish similar characters.
[0013] (2) Local features: This type of feature calculates corresponding features in multiple local regions of a character and uses concatenated ordered local features to form the final feature vector. The main features include local grayscale histogram features, LBP features, line-through features, SIFT features, etc. The advantage of this type of feature is its strong ability to distinguish characters; its disadvantage is that it focuses too much on the local features of the character and often misdistinguishes characters with noise interference.
[0014] The technologies described above all achieve good results for clear license plate images. However, license plate images captured in real-world environments often suffer from low resolution, faded or missing characters, blurred edges, and tilted characters. This makes accurate license plate position correction, character segmentation, and character recognition very difficult, and can even lead to failures, severely impacting the overall performance of license plate recognition. Therefore, accurately and robustly recognizing license plate characters remains a challenge for domestic license plate recognition systems.
[0015] In recent years, multimodal large model technology has gained widespread attention and application in various fields because it can simultaneously focus on text features and image features, providing richer and more comprehensive feature information. However, the disadvantage of this technology is that the model consumes a lot of memory and computation, thus requiring a large amount of memory and computing power. Summary of the Invention
[0016] Based on the technical problems existing in the background technology, this invention proposes a license plate character recognition method, system and storage medium based on multimodal information. Only one deep neural network model is needed to complete the recognition of license plate characters, and the recognition results are more accurate and more robust.
[0017] The license plate character recognition method proposed in this invention preprocesses the license plate image and inputs it into a trained deep neural network model to output the license plate recognition result.
[0018] The deep neural network model includes a prediction module, a student model, and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are passed to the student model through knowledge distillation technology to complete the recognition of license plate characters.
[0019] The input to the teacher model is a license plate image and a license plate description text. The license plate image is fine-tuned by a visual encoder and then fused with the license plate description text through cross-modal attention to obtain the visual features of the license plate image with textual feature information. The input to the student model is a license plate image, which is processed by feature extraction to obtain a high-level feature map. The visual features of the license plate image and the high-level feature map are used as input to the prediction module to output the recognition result of the license plate characters.
[0020] Furthermore, the teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn;
[0021] The input to the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image;
[0022] The visual encoder CLIP_V takes the license plate image as input. The input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V. The output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn. The cross-modal attention fusion module Crossattn outputs the visual features of the license plate image.
[0023] Furthermore, the cross-modal attention fusion module Crossattn includes a feature extraction layer select, an image-text attention module, and a text-image attention module. The image features of the license plate image are input as an input feature sequence to the feature extraction layer select. The feature extraction layer select extracts the positional features of the last feature sequence as the global information features of the license plate image.
[0024] The global information features and text features of the license plate image are input into the image text attention module to integrate the image feature information into the text features and output the image text features.
[0025] The text features of the image and the image features of the license plate image are input into the text image attention module to integrate the text feature information into the image features and output the visual features of the license plate image.
[0026] Furthermore, the image text attention module includes an element-wise multiplication layer attn_it, an activation function layer softmax_it, and a weighted operation layer attn_mul_it connected in sequence. The input of the element-wise multiplication layer attn_it is connected to the output of the feature extraction layer select. Both the element-wise multiplication layer attn_it and the weighted operation layer attn_mul_it take the text features of the license plate image as input, and the weighted operation layer attn_mul_it outputs the image text features.
[0027] Furthermore, the text image attention module includes a matrix multiplication layer attn_ti, an activation function layer softmax_ti, and a weighted operation layer attn_mul_ti connected in sequence. The matrix multiplication layer attn_ti takes the image features and text features of the license plate image as input, the weighted operation layer attn_mul_ti takes the image features of the license plate image as input and the output features of the activation function layer softmax_ti as input, and the weighted operation layer attn_mul_ti outputs the text features.
[0028] Furthermore, the prediction module includes a fully connected layer Fuse and a fully connected layer Fc connected in sequence; the input of the fully connected layer Fuse is connected to the output of the student model and the output of the cross-modal attention fusion module Crossattn, respectively, and the fully connected layer Fc outputs the recognition result of the card character.
[0029] Furthermore, the student model Backbone includes a convolutional layer conv0, a max pooling layer maxpool, a convolutional layer rconv0, a convolutional layer rconv1, a convolutional layer rconv2, a convolutional layer rconv3, a convolutional layer rconv4, a merging layer eltsum0, a merging layer eltsum1, a convolutional layer conv1, and a feature size adjustment layer reshape;
[0030] The input to the convolutional layer conv0 is the license plate image. The input of the max pooling layer is connected to the output of conv0. The output of max pooling is connected to the inputs of convolutional layers rconv0 and rconv2. The outputs of concatenated convolutional layers rconv0 and rconv1, and rconv2, are both connected to the input of the merged layer eltsum0. The output of eltsum0 is connected to the inputs of eltsum1 and rconv3. The input of convolutional layer rconv4 is connected to the output of rconv3, and its output is connected to the input of eltsum1. The merged layer eltsum1, convolutional layer conv1, and feature size adjustment layer reshape are concatenated. The feature size adjustment layer reshape outputs a high-level feature map.
[0031] Furthermore, the training process of the deep neural network model is as follows:
[0032] Collect license plate images under various scenes, lighting conditions, and angles. Use license plate character segmentation methods to obtain local region images of license plate characters. Label the license plate information in each local region image of a license plate character. The license plate information includes license plate color, license plate type, license plate characters, and the center position of the license plate characters, in order to construct training sample images.
[0033] The training sample images are input into the constructed deep neural network model to train the deep neural network model;
[0034] The teacher model is trained while the student model is ignored. The visual encoder CLIP_V and text encoder CLIP_T are frozen. The linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module are trained based on the main supervision function main_loss. After the teacher model training converges, the visual encoder CLIP_V is unfrozen, and the visual encoder CLIP_V, the linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module are trained based on the main supervision function main_loss. The main supervision function main_loss is used to supervise the training of the license plate character recognition task, and the CTC loss function is used.
[0035] Train the student model, freeze the teacher model and prediction module, and train the student model Backbone based on the main supervision function main_loss;
[0036] The training process involves comprehensive supervision, freezing the teacher model, training the student model and prediction module based on the main supervision function main_loss, and simultaneously training the student model Backbone based on the auxiliary supervision function aux_loss. The auxiliary supervision function aux_loss is used to distill the knowledge of the auxiliary teacher model, thereby teaching and supervising the training of the student model. The auxiliary supervision function aux_loss adopts the KD divergence loss function.
[0037] Furthermore, the license plate image is preprocessed and then input into a trained deep neural network model to output the license plate recognition result;
[0038] The deep neural network model includes a prediction module, a student model, and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are passed to the student model through knowledge distillation technology to complete the recognition of license plate characters.
[0039] The input to the teacher model is a license plate image and a license plate description text. The license plate image is fine-tuned by a visual encoder and then fused with the license plate description text through cross-modal attention to obtain the visual features of the license plate image with textual feature information. The input to the student model is a license plate image, which is processed by feature extraction to obtain a high-level feature map. The visual features of the license plate image and the high-level feature map are used as input to the prediction module to output the recognition result of the license plate characters.
[0040] A computer-readable storage medium storing a plurality of programs for being invoked by a processor and executed as described above for license plate character recognition.
[0041] The advantages of the license plate character recognition method, system, and storage medium based on multimodal information provided by this invention are as follows: A heavyweight teacher model based on a CLIP multimodal encoder is distilled to extract its core multimodal information knowledge, which is then passed on to a CNN-based student model. Ultimately, a deep neural network model capable of acquiring multimodal information is obtained. Only one deep neural network model is needed to complete license plate character recognition, resulting in faster recognition speed, fewer error sources, and more accurate recognition results. It also demonstrates stronger robustness against low-quality license plate images with color casts, contamination, adhesion, missing characters, or insufficient localization. Attached Figure Description
[0042] Figure 1 The structural diagram of the deep neural network model designed for this invention;
[0043] Figure 2 A structural diagram of the student model;
[0044] Figure 3 This is a structural diagram of the cross-modal attention fusion module. Detailed Implementation
[0045] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0046] like Figures 1 to 3 As shown, the license plate character recognition method based on multimodal information proposed in this invention preprocesses the license plate image and inputs it into a trained deep neural network model to output the license plate recognition result; the license plate image preprocessing method is detailed in step S31.
[0047] The deep neural network model includes a prediction module, a student model, and a teacher model based on a multimodal encoder. Knowledge distillation is used to transfer the multimodal features of the teacher model to the student model to complete license plate character recognition. The teacher model takes a license plate image and descriptive text as input. The license plate image is fine-tuned by a visual encoder and then fused with the descriptive text through cross-modal attention to obtain visual features of the license plate image with textual information. The student model takes a license plate image as input, which is then processed through feature extraction to obtain a high-level feature map. The visual features of the license plate image and the high-level feature map serve as input to the prediction module to output the license plate character recognition result.
[0048] This embodiment extracts the core multimodal information from a heavyweight CLIP-based multimodal encoder teacher model and teaches it to a CNN-based student model, ultimately resulting in a deep neural network model capable of acquiring multimodal information. Only one deep neural network model is needed to recognize license plate characters, resulting in faster recognition speed, fewer error sources, and more accurate results. It also demonstrates stronger robustness to low-quality license plate images with color casts, contamination, character adhesion, missing characters, or inaccurate localization.
[0049] The method proposed in this embodiment is applicable to the recognition of characters on any type of license plate, requiring only the acquisition of a partial image of the entire license plate. For ease of explanation, this embodiment uses Chinese license plates (including: single-layer blue plates, single-layer yellow plates, double-layer yellow plates, military and police plates, school plates, and new energy vehicle plates) as an example. The input image is a 3-channel RGB image with a resolution of 224×64.
[0050] The license plate character recognition method based on multimodal information includes the following steps:
[0051] S1. Design of a Deep Neural Network Model. The deep neural network model designed in this embodiment primarily utilizes a teacher model based on a multimodal encoder. Through knowledge distillation, the multimodal features of the multimodal encoder are transferred to a CNN-based deep neural network student model, directly completing license plate character recognition. Since the steps of license plate position correction, character segmentation, and character recognition are no longer explicitly distinguished, the entire license plate character recognition process is faster, with fewer error sources and higher accuracy. This embodiment uses a convolutional neural network (CNN) structure and a transformer structure for the deep neural network model design. For ease of description, some terms are defined: feature map resolution refers to feature map height × feature map width; feature map size refers to feature map height × feature map width × number of feature map channels; kernel size refers to kernel width × kernel height; span refers to span in the width direction × span in the height direction. Additionally, each convolutional layer is followed by a batch normalization layer and a non-linear activation layer. Figure 1As shown, the specific design steps are as follows:
[0052] S11. Design the teacher model. The teacher model designed in this embodiment is mainly based on the existing multimodal encoder CLIP (Learning Transferable Visual Models From Natural Language Supervision) pre-trained model. First, it is fine-tuned by visual encoding features, and then cross-modal attention fusion is performed with text encoding features to finally obtain the visual features of the license plate image with text feature information.
[0053] The teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn. The input of the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image. The visual encoder CLIP_V takes the license plate image as input. The input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V, and the output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn. The cross-modal attention fusion module Crossattn outputs the visual features of the license plate image.
[0054] The text encoder CLIP_T outputs only the global information features of the last sequence position, while the visual encoder CLIP_V outputs both global information features and all local patch (image slice) information features. Finetune is a linear projection layer for fine-tuning visual features. Its main functions are: first, to adjust the number of channels in the input features to ensure consistency in feature dimensions between visual and text features; and second, to fine-tune the output features of the visual encoder CLIP_V to obtain visual features more suitable for the current license plate character recognition task. Crossattn is a cross-modal attention fusion module, primarily responsible for cross-modal fusion of visual and text features to obtain visual features of the license plate image with textual feature information. The specific network results are as follows: Figure 3 As shown:
[0055] The cross-modal attention fusion module Crossattn comprises a feature extraction layer (select), an image-text attention module, and a text-image attention module. The image features of the license plate image are input as a feature sequence to the feature extraction layer (select). Select extracts the positional features of the last feature sequence as the global information feature of the license plate image. The global information feature and the text features of the license plate image are input to the image-text attention module to integrate the image feature information into the text feature, outputting the image-text feature. The image-text feature and the image features of the license plate image are input to the text-image attention module to integrate the text feature information into the image feature, outputting the visual feature of the license plate image. Specifically, the feature extraction layer (select) is mainly used to extract the positional features of the last sequence from the input feature sequence as the global information feature of the input image; attn_mul_it is a layer that performs weighted operations on each sequence feature according to the last dimension.
[0056] The image-text attention module is used to integrate image feature information into text features. Specifically, it includes an element-wise multiplication layer attn_it, an activation function layer softmax_it, and a weighted operation layer attn_mul_it connected in sequence. The input of the element-wise multiplication layer attn_it is connected to the output of the feature extraction layer select. Both the element-wise multiplication layer attn_it and the weighted operation layer attn_mul_it take the text features of the license plate image as input, and the weighted operation layer attn_mul_it outputs the image-text features.
[0057] The text-image attention module is used to integrate textual feature information into image features. Specifically, it includes a matrix multiplication layer attn_ti, an activation function layer softmax_ti, and a weighted operation layer attn_mul_ti connected in sequence. The matrix multiplication layer attn_ti takes the image features and text features of the license plate image as input, the weighted operation layer attn_mul_ti takes the image features of the license plate image as input and the output features of the activation function layer softmax_ti as input, and the weighted operation layer attn_mul_ti outputs the visual features of the license plate image.
[0058] S12. Design the Backbone model, which is also the final feature extraction module desired in this embodiment. It is mainly used to obtain high-level features with rich expressive power from the input license plate image. The quality of high-level feature extraction directly affects the performance of subsequent license plate recognition. The specific network structure is as follows: Figure 2As shown, the student model Backbone includes a convolutional layer conv0, a max pooling layer, a convolutional layer rconv0, a convolutional layer rconv1, a convolutional layer rconv2, a convolutional layer rconv3, a convolutional layer rconv4, a merged layer eltsum0, a merged layer eltsum1, a convolutional layer conv1, and a feature size adjustment layer reshape. The input to the convolutional layer conv0 is the license plate image. The input of the max pooling layer is connected to the output of the convolutional layer conv0, and the output of the max pooling layer is connected to the input of the convolutional layer rconv0 and the convolutional layer rconv4. The input of conv2, the output of concatenated convolutional layers rconv0 and rconv1, and the output of convolutional layer rconv2 are all connected to the input of merging layer eltsum0. The output of merging layer eltsum0 is connected to the input of merging layer eltsum1 and the input of convolutional layer rconv3. The input of convolutional layer rconv4 is connected to the output of convolutional layer rconv3, and the output is connected to the input of merging layer eltsum1. Merging layer eltsum1, convolutional layer conv1, and feature size adjustment layer reshape are concatenated. Feature size adjustment layer reshape outputs a high-level feature map.
[0059] conv0 is a convolutional layer with a kernel size of 3x3 and a span of 2x2; maxpool is a max pooling layer with a kernel size of 2x2 and a span of 2x2; rconv2 is a convolutional layer with a kernel size of 1x1 and a span of 2x2; rconv0 and rconv3 are convolutional layers with a kernel size of 1x1 and a span of 1x1; rconv1 is a convolutional layer with a kernel size of 3x3 and a span of 2x2; rconv4 is a convolutional layer with a kernel size of 3x3 and a span of 1x1; conv1 is a convolutional layer with a kernel size of 3x3 and a span of 2x1; eltsum0 and eltsum1 are both merging layers, which combine multiple input feature maps element-wise into a single output feature map; the reshape layer is a feature size adjustment layer used to adjust the size of the input features; the output of the student model is the high-level feature map, image feature.
[0060] S13. Design the prediction module. The prediction module mainly predicts the output result of license plate characters based on the license plate visual feature vector (high-level feature map or high-level feature map) obtained in step S11 or step S12. It mainly consists of the Fuse layer and the Fc layer. Both the Fuse layer and the Fc layer are fully connected layers. The fully connected Fuse layer is mainly used to fuse the multimodal features of the input, and the fully connected Fc layer is mainly used for the recognition and prediction of license plate characters.
[0061] S14. Design the supervision function. The supervision function in this embodiment consists of two parts: the main supervision function main_loss and the auxiliary supervision function aux_loss. The main supervision function is used to supervise the training of the license plate character recognition task and adopts the CTC loss function (ctc_loss). CTC_loss is the most effective training loss function for recognizing language sentences with contextual information relationships. The auxiliary supervision function is mainly used to distill the knowledge of the teacher module and then teach and supervise the training of the student model. The auxiliary supervision function adopts the KD divergence loss function, which is the most commonly used supervision function for knowledge distillation.
[0062] S2. Training the deep neural network model mainly involves optimizing the model parameters using a large amount of labeled training sample data to achieve the best recognition performance. The specific steps are as follows:
[0063] S21. Obtain training sample data, mainly by collecting license plate images under various scenes, lighting conditions, and angles. Use existing license plate detection methods to obtain local license plate region images, and then label the information of each license plate, including license plate characters and comprehensive license plate description text. The template used for the comprehensive license plate description text is "a xxx license plate with chars of **********", where xxx represents any adjective, and the Chinese characters on the license plate are represented by pinyin, such as "ablurry double yellow chinese license plate with chars of jingA123456" which means "an unclear double yellow Chinese license plate with chars of jingA123456".
[0064] S22. Training the deep neural network model mainly involves feeding the labeled set of license plate sample images into a predefined deep neural network model to learn the relevant model parameters. This embodiment includes two supervision functions. To accelerate the model's training speed and improve the training effect, the following steps are used for training:
[0065] S221. Train the teacher model, mainly ignoring the student model. First, freeze the visual encoder CLIP_V and the text encoder CLIP_T. Train the linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module based on the main supervision function main_loss. After the teacher model training converges, unfreeze the visual encoder CLIP_V and train the visual encoder CLIP_V, the linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module based on the main supervision function main_loss.
[0066] S222. Train the student model, freeze the teacher model and prediction module, and train the student model Backbone based on the main supervision function main_loss.
[0067] S223. Comprehensive supervised training: Freeze the teacher model, train the student model and prediction module based on the main supervision function main_loss, and train the student model Backbone based on the auxiliary supervision function aux_loss.
[0068] S3. Using a deep neural network model: After training the deep neural network model, the next step is to use it in a real-world environment. It's important to note that the teacher module is only used to assist training; it can be ignored when you need to use the deep neural network model directly. For any given partial image of a license plate, feed it into the trained deep neural network model and output the license plate recognition result. The specific steps are as follows:
[0069] S31. Detecting local license plate images mainly involves detecting the location of the license plate in the entire image, extracting the local license plate image from the entire image through cropping operations, and normalizing the image size. The main method is to use existing mature methods.
[0070] S32. License plate recognition: The main method is to feed the normalized local license plate image into a trained deep neural network model, and the final output is the optimal license plate recognition result.
[0071] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A license plate character recognition method based on multimodal information, characterized in that, After preprocessing, the license plate image is input into a trained deep neural network model to output the license plate recognition result. The deep neural network model includes a prediction module, a student model, and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are passed to the student model through knowledge distillation technology to complete the recognition of license plate characters. The input to the teacher model is a license plate image and a license plate description text. The license plate image is fine-tuned by a visual encoder and then fused with the license plate description text through cross-modal attention to obtain the visual features of the license plate image with textual feature information. The input to the student model is a license plate image, which is processed by feature extraction to obtain a high-level feature map. The visual features of the license plate image and the high-level feature map are used as input to the prediction module to output the recognition result of the license plate characters. The teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn. The input to the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image; The visual encoder CLIP_V takes the license plate image as input. The input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V. The output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn. The cross-modal attention fusion module Crossattn outputs the visual features of the license plate image. The cross-modal attention fusion module Crossattn includes a feature extraction layer select, an image-text attention module, and a text-image attention module. The image features of the license plate image are used as input feature sequences and input into the feature extraction layer select. The feature extraction layer select extracts the position features of the last feature sequence as the global information features of the license plate image. The global information features and text features of the license plate image are input into the image text attention module to integrate the image feature information into the text features and output the image text features. The text features of the image and the image features of the license plate image are input into the text image attention module to integrate the text feature information into the image features and output the visual features of the license plate image. The prediction module includes a fully connected layer Fuse and a fully connected layer Fc connected in sequence; the input of the fully connected layer Fuse is connected to the output of the student model and the output of the cross-modal attention fusion module Crossattn, respectively, and the fully connected layer Fc outputs the recognition result of the card character.
2. The license plate character recognition method based on multimodal information according to claim 1, characterized in that, The image text attention module includes an element-wise multiplication layer attn_it, an activation function layer softmax_it, and a weighted operation layer attn_mul_it connected in sequence. The input of the element-wise multiplication layer attn_it is connected to the output of the feature extraction layer select. Both the element-wise multiplication layer attn_it and the weighted operation layer attn_mul_it take the text features of the license plate image as input, and the weighted operation layer attn_mul_it outputs the image text features.
3. The license plate character recognition method based on multimodal information according to claim 2, characterized in that, The text image attention module includes a matrix multiplication layer attn_ti, an activation function layer softmax_ti, and a weighted operation layer attn_mul_ti connected in sequence. The matrix multiplication layer attn_ti takes the image features and text features of the license plate image as input, the weighted operation layer attn_mul_ti takes the image features of the license plate image as input and the output features of the activation function layer softmax_ti as input, and the weighted operation layer attn_mul_ti outputs the text.
4. The license plate character recognition method based on multimodal information according to claim 1, characterized in that, The student model Backbone includes a convolutional layer conv0, a max pooling layer maxpool, a convolutional layer rconv0, a convolutional layer rconv1, a convolutional layer rconv2, a convolutional layer rconv3, a convolutional layer rconv4, a merged layer eltsum0, a merged layer eltsum1, a convolutional layer conv1, and a feature size adjustment layer reshape; The input to the convolutional layer conv0 is the license plate image. The input of the max pooling layer is connected to the output of conv0. The output of max pooling is connected to the inputs of convolutional layers rconv0 and rconv2. The outputs of concatenated convolutional layers rconv0 and rconv1, and rconv2, are all connected to the input of the merged layer eltsum0. The output of eltsum0 is connected to the inputs of eltsum1 and rconv3. The input of convolutional layer rconv4 is connected to the output of rconv3, and its output is connected to the input of eltsum1. The merged layer eltsum1, convolutional layer conv1, and feature size adjustment layer reshape are concatenated. The feature size adjustment layer reshape outputs a high-level feature map.
5. The license plate character recognition method based on multimodal information according to claim 1, characterized in that, The training process of the deep neural network model is as follows: Collect license plate images under various scenes, lighting conditions, and angles. Use license plate character segmentation methods to obtain local region images of license plate characters. Label the license plate information in each local region image of a license plate character. The license plate information includes license plate color, license plate type, license plate characters, and the center position of the license plate characters, in order to construct training sample images. The training sample images are input into the constructed deep neural network model to train the deep neural network model; The teacher model is trained while the student model is ignored. The visual encoder CLIP_V and text encoder CLIP_T are frozen. The linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module are trained based on the main supervision function main_loss. After the teacher model training converges, the visual encoder CLIP_V is unfrozen, and the visual encoder CLIP_V, the linear projection layer Finetune, the cross-modal attention fusion module Crossattn, and the prediction module are trained based on the main supervision function main_loss. The main supervision function main_loss is used to supervise the training of the license plate character recognition task, and the CTC loss function is used. Train the student model, freeze the teacher model and prediction module, and train the student model Backbone based on the main supervision function main_loss; The training process involves comprehensive supervision, freezing the teacher model, training the student model and prediction module based on the main supervision function main_loss, and simultaneously training the student model Backbone based on the auxiliary supervision function aux_loss. The auxiliary supervision function aux_loss is used to distill the knowledge of the auxiliary teacher model, thereby teaching and supervising the training of the student model. The auxiliary supervision function aux_loss adopts the KD divergence loss function.
6. A license plate character recognition system based on multimodal information, characterized in that, After preprocessing, the license plate image is input into a trained deep neural network model to output the license plate recognition result. The deep neural network model includes a prediction module, a student model, and a teacher model based on a multimodal encoder. The multimodal features of the teacher model are passed to the student model through knowledge distillation technology to complete the recognition of license plate characters. The input to the teacher model is a license plate image and a license plate description text. The license plate image is fine-tuned by a visual encoder and then fused with the license plate description text through cross-modal attention to obtain the visual features of the license plate image with textual feature information. The input to the student model is a license plate image, which is processed by feature extraction to obtain a high-level feature map. The visual features of the license plate image and the high-level feature map are used as input to the prediction module to output the recognition result of the license plate characters. The teacher model includes a visual encoder CLIP_V, a text encoder CLIP_T, a linear projection layer Finetune, and a cross-modal attention fusion module Crossattn. The input to the text encoder CLIP_T is the license plate description text, and the output is the text features of the license plate image; The visual encoder CLIP_V takes the license plate image as input. The input of the linear projection layer Finetune is connected to the output of the visual encoder CLIP_V. The output of the linear projection layer Finetune is connected to the input of the cross-modal attention fusion module Crossattn. The cross-modal attention fusion module Crossattn outputs the visual features of the license plate image. The cross-modal attention fusion module Crossattn includes a feature extraction layer select, an image-text attention module, and a text-image attention module. The image features of the license plate image are used as input feature sequences and input into the feature extraction layer select. The feature extraction layer select extracts the position features of the last feature sequence as the global information features of the license plate image. The global information features and text features of the license plate image are input into the image text attention module to integrate the image feature information into the text features and output the image text features. The text features of the image and the image features of the license plate image are input into the text image attention module to integrate the text feature information into the image features and output the visual features of the license plate image. The prediction module includes a fully connected layer Fuse and a fully connected layer Fc connected in sequence; the input of the fully connected layer Fuse is connected to the output of the student model and the output of the cross-modal attention fusion module Crossattn, respectively, and the fully connected layer Fc outputs the recognition result of the card character.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of programs, which are used to be called by a processor and executed as described in any one of claims 1 to 5.