License plate character recognition method and device based on deep learning, and electronic equipment

Abstract

The embodiment of the invention provides a license plate character recognition method and device based on deep learning, and electronic equipment. The method comprises the steps of obtaining a to-be-recognized license plate image; inputting the to-be-recognized license plate image into a backbone network, and calculating to obtain a to-be-coded feature; and inputting the to-be-encoded features into an encoder to obtain a license plate recognition character string. Through the scheme of the application, the to-be-identified license plate image can be expanded into one-dimensional sequence data through the patch embedding module in the backbone network, and then feature extraction and convolution calculation are performed through the first image feature extraction module, so that the calculation complexity is reduced; moreover, normalization and convolution calculation can be carried out through the second image feature extraction module, the calculation complexity is reduced, the problem of how to reduce the calculation complexity in the license plate recognition process is solved, and the response speed and efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of image recognition, and in particular to a method, apparatus, and electronic device for license plate character recognition based on deep learning. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, its applications in daily life have become increasingly widespread. This is especially true in the field of license plate recognition, where network models are commonly used to identify license plate images. However, the inventors have discovered that current license plate recognition processes often involve high computational complexity, resulting in long response times. This is particularly true for embedded devices with lower computing power, where response times are even longer, failing to meet real-time requirements. Summary of the Invention

[0003] The purpose of this application is to provide a method, apparatus, and electronic device for license plate character recognition based on deep learning, in order to solve the problem of how to reduce the computational complexity in the license plate recognition process. The specific technical solution is as follows: A first aspect of this application provides a deep learning-based license plate character recognition method, comprising: Acquire the image of the license plate to be identified; The license plate image to be identified is input into the backbone network to calculate the features to be encoded. The backbone network includes multiple sequentially connected network layers. Each network layer includes: a patch embedding module, a first image feature extraction module, a spatial location awareness module, and a second image feature extraction module. The processing procedure of each network layer includes: expanding the input data one-dimensionally through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information annotation and feature extraction on the calculation result of the first image feature extraction module through the spatial location awareness module to obtain the output data of the spatial location awareness module; and performing feature extraction and convolution calculation on the output data of the spatial location awareness module through the second image feature extraction module to obtain the calculation result of the second image feature extraction module. The features to be encoded are input into an encoder to obtain a license plate recognition string. The encoder includes a location embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module. The encoder's processing includes: performing one-dimensional location encoding on the features to be encoded through the location embedding module to obtain location-encoded features; sequentially inputting the location-encoded features into the third image feature extraction module and the fourth image feature extraction module to extract global features through an attention mechanism; calculating the global features through the batch normalization module to obtain the encoding result; and inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string.

[0004] In one possible implementation, the step of performing a one-dimensional expansion of the input data through the patch embedding module to obtain one-dimensional sequence data includes: The patch embedding module downsamples the license plate image to be identified to obtain downsampled image data; the downsampled image data is then expanded to obtain image data of a specified size; and the image data of the specified size is then expanded into the one-dimensional sequence data.

[0005] In one possible implementation, the step of performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module includes: The first image feature extraction module extracts the correlation information between characters in the license plate from the one-dimensional sequence data. When the license plate image to be identified is distorted and / or blurred, the first image feature extraction module performs global information extraction on the one-dimensional sequence data. The calculation results of the first image feature extraction module include: correlation information and global information between characters in the license plate.

[0006] In one possible implementation, the step of performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module includes: The first image feature extraction module downsamples the one-dimensional sequence data using a spatial reduction attention mechanism to obtain downsampled sequence data, wherein the length of the downsampled sequence data is less than the length of the one-dimensional sequence data. The downsampled sequence data is used to extract global features using an attention mechanism to obtain the calculation results of the first image feature extraction module.

[0007] In one possible implementation, inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string includes: The encoding result is input into the fully connected module, and a three-dimensional feature tensor is calculated by the fully connected module. The dimensions of the three-dimensional feature tensor correspond to the sequence length, the total number of character categories, and the batch size, respectively. The license plate recognition string is obtained by performing connection-time classification decoding on the three-dimensional feature tensor.

[0008] A second aspect of this application provides a license plate character recognition device based on deep learning, comprising: Image acquisition module, used to acquire images of license plates to be recognized; A backbone network module is used to input the license plate image to be identified into the backbone network and calculate the features to be encoded. The backbone network includes multiple network layers connected in sequence. Each network layer includes: a patch embedding module, a first image feature extraction module, a spatial location awareness module, and a second image feature extraction module. The processing procedure of each network layer includes: expanding the input data one-dimensionally through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information annotation and feature extraction on the calculation result of the first image feature extraction module through the spatial location awareness module to obtain the output data of the spatial location awareness module; and performing feature extraction and convolution calculation on the output data of the spatial location awareness module through the second image feature extraction module to obtain the calculation result of the second image feature extraction module. A feature encoding module is used to input the features to be encoded into an encoder to obtain a license plate recognition string. The encoder includes a location embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module. The encoder's processing includes: performing one-dimensional location encoding on the features to be encoded through the location embedding module to obtain location-encoded features; sequentially inputting the location-encoded features into the third image feature extraction module and the fourth image feature extraction module to extract global features through an attention mechanism; calculating the global features through the batch normalization module to obtain an encoding result; and inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string.

[0009] In one possible implementation, the backbone network module is specifically used to downsample the license plate image to be identified through the patch embedding module to obtain downsampled image data; expand the downsampled image data to obtain image data of a specified size; and expand the image data of the specified size into the one-dimensional sequence data.

[0010] In one possible implementation, the backbone network module is specifically used to extract correlation information between characters in the license plate from the one-dimensional sequence data through the first image feature extraction module; when the license plate image to be identified is distorted and / or blurred, global information is extracted from the one-dimensional sequence data through the first image feature extraction module, wherein the calculation result of the first image feature extraction module includes: correlation information between characters in the license plate and global information.

[0011] In one possible implementation, the backbone network module is specifically used to downsample the one-dimensional sequence data using a spatial reduction attention mechanism through the first image feature extraction module to obtain downsampled sequence data, wherein the length of the downsampled sequence data is less than the length of the one-dimensional sequence data; and to extract global features from the downsampled sequence data using an attention mechanism to obtain the calculation result of the first image feature extraction module.

[0012] In one possible implementation, the feature encoding module is specifically used to input the encoding result into the fully connected module, and calculate a three-dimensional feature tensor through the fully connected module, wherein the dimensions of the three-dimensional feature tensor correspond to the sequence length, the total number of character categories, and the batch size, respectively; and perform connection-time classification decoding on the three-dimensional feature tensor to obtain the license plate recognition string.

[0013] Another aspect of the application embodiments also provides an electronic device, including: Memory, used to store computer programs; When the processor executes the program stored in the memory, it implements any of the above-mentioned deep learning-based license plate character recognition methods.

[0014] In another aspect of the embodiments of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements any of the above-described deep learning-based license plate character recognition methods.

[0015] In another aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the above-described deep learning-based license plate character recognition methods.

[0016] Beneficial effects of the embodiments in this application: This application provides a deep learning-based method, apparatus, and electronic device for license plate character recognition, including: acquiring a license plate image to be recognized; inputting the license plate image to be recognized into a backbone network to calculate features to be encoded; the backbone network includes multiple sequentially connected network layers, each network layer including: a patch embedding module, a first image feature extraction module, a spatial location awareness module, and a second image feature extraction module; wherein, the processing of each network layer includes: performing one-dimensional expansion on the input data through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information annotation and feature extraction on the calculation result of the first image feature extraction module through the spatial location awareness module to obtain the output data of the spatial location awareness module; and so on. The second image feature extraction module performs feature extraction and convolution calculation on the output data of the spatial location perception module to obtain the calculation result of the second image feature extraction module; the feature to be encoded is input into the encoder to obtain the license plate recognition string; the encoder includes: a location embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module; wherein, the processing of the encoder includes: performing one-dimensional location encoding on the feature to be encoded through the location embedding module to obtain location encoded features; inputting the location encoded features sequentially into the third image feature extraction module and the fourth image feature extraction module to extract global features through an attention mechanism; calculating the global features through the batch normalization module to obtain the encoding result; inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string. The solution proposed in this application not only expands the license plate image to be recognized into one-dimensional sequence data through the patch embedding module in the backbone network, and then performs feature extraction and convolution calculation through the first image feature extraction module to reduce computational complexity, but also performs normalization and convolution calculation through the second image feature extraction module to reduce computational complexity. This solves the problem of high computational complexity in the license plate recognition process and improves response speed and efficiency.

[0017] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0019] Figure 1 A flowchart illustrating a deep learning-based license plate character recognition method provided in an embodiment of this application; Figure 2 A schematic diagram of a deep learning-based license plate character recognition device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0021] The first aspect of this application's embodiments provides a license plate character recognition method based on deep learning, see [link to relevant documentation]. Figure 1 , Figure 1 A flowchart illustrating a deep learning-based license plate character recognition method provided in this application includes: Step S11: Obtain the image of the license plate to be identified; Step S12: Input the license plate image to be identified into the backbone network and calculate the features to be encoded. The backbone network includes multiple sequentially connected network layers. Each network layer includes: a patch embedding module, a first image feature extraction module, a spatial location awareness module, and a second image feature extraction module. The processing procedure of each network layer includes: expanding the input data one-dimensionally through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information annotation and feature extraction on the calculation result of the first image feature extraction module through the spatial location awareness module to obtain the output data of the spatial location awareness module; and performing feature extraction and convolution calculation on the output data of the spatial location awareness module through the second image feature extraction module to obtain the calculation result of the second image feature extraction module. Step S13: Input the feature to be encoded into the encoder to obtain the license plate recognition string; the encoder includes: a position embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module; wherein, the processing of the encoder includes: performing one-dimensional position encoding on the feature to be encoded through the position embedding module to obtain position encoded features; sequentially inputting the position encoded features into the third image feature extraction module and the fourth image feature extraction module to extract global features through an attention mechanism; calculating the global features through the batch normalization module to obtain the encoding result; inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string.

[0022] Corresponding to step S11 above, when acquiring the image of the license plate to be recognized, an image of the vehicle can be captured, thereby acquiring an image containing the license plate. In one example, the solution of this application can be applied to a license plate recognition device, which can be installed at the entrance and exit of a parking lot for license plate recognition. The device can include a camera, through which license plate images can be captured.

[0023] Patch embedding module, first image feature extraction module, spatial location awareness module, and second image feature extraction module Corresponding to step S12 above, the backbone network in this application can be a multi-layer network, including multiple network layers. Each network layer may include: PatchEmbedding, a first VITBlock (image feature extraction module), PosCNN (spatial location awareness module), and a second VITBlock (image feature extraction module). In this application, the input data is one-dimensionally unfolded through the patch embedding module to obtain one-dimensional sequence data; the first image feature extraction module performs feature extraction and convolution calculation on the one-dimensional sequence data to obtain the calculation result of the first image feature extraction module; the spatial location awareness module performs information annotation and feature extraction on the calculation result of the first image feature extraction module to obtain the output data of the spatial location awareness module; and the second image feature extraction module performs feature extraction and convolution calculation on the output data of the spatial location awareness module to obtain the calculation result of the second image feature extraction module. Specifically, PatchEmbedding performs one-dimensional expansion on the input data to obtain one-dimensional sequence data. This includes: downsampling the license plate image to be identified using the patch embedding module to obtain downsampled image data; expanding the downsampled image data to obtain image data of a specified size; and expanding the image data of the specified size into the one-dimensional sequence data. In one example, a two-dimensional image can be segmented into fixed-size image patches, then mapped to fixed-dimensional vectors through linear projection, ultimately outputting one-dimensional sequence data. In another example, the main body of PatchEmbedding is a CNN with stride=2, responsible for downsampling the feature map resolution to 1 / 2 and expanding the channels by 2 times, ultimately expanding the H and W dimensions to the same dimension for easier subsequent processing. The first VITBlock is used for feature extraction and convolution calculation to obtain the calculation results. Specifically, it can capture global dependencies through a self-attention mechanism to learn the global representation of the image; it typically includes two sub-modules: multi-head attention and a feedforward network. Unlike the local modeling of traditional CNN (Convolutional Neural Network) networks, the second VITBlock achieves the global receptive field through a self-attention mechanism. PosCNN is used to annotate and extract features from the input computation results by incorporating position-sensitive convolutional neural network layers, resulting in output data for these layers. Specifically, by introducing positional information, it can enhance the model's focus on key regions in an image, such as detecting object features at different locations. It also partially incorporates POA (Pollution Optimization) algorithms (such as the Pelican Optimization algorithm) to adjust CNN parameters, thereby improving regression prediction accuracy.In PosCNN, since the Transformer requires positional information of sequence elements, a CNN is set up here to label the positions of the feature maps, which is consistent with the function of the PosEmbedding component in the original Transformer structure. The difference is that, because image information is two-dimensional, the information labeling in PosCNN is also two-dimensional, while the original PosEmbedding only needs to label one-dimensional sequence information.

[0024] Corresponding to step S13 above, the features to be encoded are input into the encoder, which sequentially includes: a position embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module, to obtain a license plate recognition string. Here, the third and fourth image feature extraction modules can have the same structure and computational logic as the first and second image feature extraction modules mentioned above. They are all used to extract global features from the input data through an attention mechanism. The extracted global features are then sequentially processed through a batch normalization layer, a first convolutional layer, an activation function layer, and a second convolutional layer to obtain a first calculation result. The encoder / decoder can sequentially include: PosEmbedding (position embedding module), a third VITBlock (image feature extraction module), a fourth VITBlock (image feature extraction module), and a BN Linear layer (normalization and fully connected layer). The location encoding features are sequentially input into the third image feature extraction module and the fourth image feature extraction module, and global features are extracted through an attention mechanism. The global features are then calculated using a batch normalization module to obtain the encoding result. The encoding result is then input into the fully connected module for temporal classification decoding to obtain the license plate recognition string. Specifically, PosEmbedding is a key component for encoding sequence location information. By adding the location information to the word embedding vector, the model can perceive the sequential relationship within the sequence. In one possible implementation, inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string includes: inputting the encoding result into the fully connected module, calculating a three-dimensional feature tensor through the fully connected module, where the dimensions of the three-dimensional feature tensor correspond to the sequence length, the total number of character categories, and the batch size, respectively; performing connection-based temporal classification decoding on the three-dimensional feature tensor to obtain the license plate recognition string. The BN Linear layer is used to normalize the input data, and then the fully connected layer performs classification based on the normalization result to obtain the classification result.

[0025] As can be seen, the scheme of this application can not only expand the license plate image to be identified into one-dimensional sequence data through the patch embedding module in the backbone network, and then perform feature extraction and convolution calculation through the first image feature extraction module to reduce the computational complexity, but also perform normalization and convolution calculation through the second image feature extraction module to reduce the computational complexity, thus solving the problem of how to reduce the high computational complexity in the license plate recognition process and improving the response speed and efficiency.

[0026] In one possible implementation, the step of performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module includes: extracting correlation information between characters in the license plate from the one-dimensional sequence data through the first image feature extraction module; when the license plate image to be identified is distorted and / or blurred, extracting global information from the one-dimensional sequence data through the first image feature extraction module, wherein the calculation result of the first image feature extraction module includes: correlation information between characters in the license plate and global information. Specifically, the core of the backbone network feature extraction is a Transformer. Transformers were originally only used for sequence data such as text and speech. When processing image data, the dimensions of the input image tensor need to be expanded to the same dimension, transformed into sequence data, and then subjected to attention operations, and then reshaped back into a tensor with the same shape as the input. Attention operations can mine relevant information of all elements in the sequence and have a strong ability to extract global information. In contrast, CNNs focus more on local details. For example, in license plate recognition tasks, the first character usually contains regional information, which affects the length and value range of subsequent characters. Therefore, statistical analysis of the correlation between these characters improves performance. In addition, in some distorted or blurry scenarios, CNNs that only focus on a square local area may not obtain complete information. Transformers, which have the ability to extract global information, can better handle these complex situations.

[0027] In one possible implementation, the step of performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module includes: downsampling the one-dimensional sequence data using a spatial reduction attention mechanism through the first image feature extraction module to obtain downsampled sequence data, wherein the length of the downsampled sequence data is less than the length of the one-dimensional sequence data; and extracting global features from the downsampled sequence data using an attention mechanism to obtain the calculation result of the first image feature extraction module. Specifically, in this embodiment, the SRA (Spatial Reduction Attention) mechanism is introduced to downsample the data, reducing computational complexity. The computation of self-attention is reduced from the original O(N²) complexity (where N is the number of tokens) to O(N × M), where M is the number of tokens after dimensionality reduction. For example, when the dimensionality reduction ratio is R (i.e., M = N / R), the computational cost can be reduced by nearly R times. This allows the model to process high-resolution feature maps with acceptable computational overhead.

[0028] In practical use, when the first image feature extraction module performs feature extraction and convolution calculation on the one-dimensional sequence data to obtain the calculation result of the first image feature extraction module, the second VITBlock in the backbone network may include: a batch normalization layer, through which the global features are normalized to obtain normalized global features; a linear transformation is performed on the normalized global features to obtain transformed features; a sliding window calculation is performed on the transformed features through the first convolution layer, and the sliding window calculation result is accumulated through matrix multiplication to obtain convolutional features; an activation function layer is used to perform activation calculation on the convolutional features to obtain activated features; and a sliding window calculation is performed on the activated features through the second convolution layer, and the sliding window calculation result is accumulated through matrix multiplication to obtain the calculation result of the first image feature extraction module. In one example, the batch normalization layer can be a Batch Normalization Layer (BN), a type of normalization layer in neural networks. It normalizes the input data of each batch during training (mean 0, variance 1) and introduces learnable parameters for scaling and translation to address internal covariate shift issues, thereby accelerating model convergence, improving generalization ability, and reducing dependence on hyperparameters. The first and second convolutional layers can be used for convolution calculations. The activation function layer can be ReLU6, an improved ReLU (Rectified Linear Function) activation function. By limiting the output to the [0,6] interval, it prevents the gradient vanishing problem caused by excessively large activation values ​​and is commonly used in mobile devices to improve the numerical stability of low-precision calculations. In this application, MLP (Multilayer Perception) operations are performed after Attention, namely BN, Conv (convolutional layer), ReLU6, and Conv. The Conv operation, originally a linear operation, has been replaced with a 1x1 Conv operation that has the same operational principle but faster computation speed.

[0029] In practical use, the features to be encoded are input into the Encoder for processing to obtain the final recognition result. The Encoder may include: PosEmbedding (position embedding module), two VITBlocks (image feature extraction modules), BN (batch normalization module), and a Linear (fully connected module). PosEmbedding, mentioned above, is the component in the original Transformer used for labeling the information positions of one-dimensional sequences. The VITBlock structure is largely the same, except that the Conv module used for feature vector scaling is removed. In this application, the model processes all elements in the input sequence in parallel. The computation of the self-attention mechanism can be matrix-based. This parallel architecture breaks through the computational bottleneck of RNNs (Recurrent Neural Networks), resulting in an order-of-magnitude improvement in the training and inference speed of the model and a significant reduction in response time. Furthermore, the Transformer can directly compute the dependency between any two elements in the sequence, regardless of their distance. This fundamentally overcomes the long-range dependency problem of RNNs, enabling the model to more accurately capture global contextual information, thereby significantly improving the feature extraction capability and accuracy on complex sequence tasks. Meanwhile, in deep learning models, especially Transformer and its variants, layer normalization has become a standard technique for stabilizing the training process and improving model performance. Layer Normalization (LN) effectively alleviates the internal covariate bias problem by normalizing the feature dimensions of individual samples. However, the computation process of LN involves calculating the mean and variance of the feature vector for each sample, which introduces significant computational overhead when the sequence length is long or the feature dimension is high. In real-time applications requiring low latency and high throughput (such as online inference and autonomous driving perception), the computational latency introduced by LN can become a bottleneck in system performance, limiting the model's response speed. By replacing LN with Batch Normalization (BN), the normalization operation of the BN layer during the model inference stage is simplified to one subtraction and one division, with a computational complexity far lower than that of the LN layer, which needs to independently calculate the mean and variance for each sample. This computational simplification directly reduces the latency of a single forward propagation, thereby achieving a faster model response time. Experiments show that this modification can reduce the response time by approximately 20% without sacrificing performance.

[0030] This application's solution innovatively integrates a Transformer into the backbone network, leveraging the Transformer's ability to extract global information to effectively improve the accuracy of license plate character recognition. The improvement is even more pronounced in complex scenarios such as blurry, tilted, and nighttime conditions. Simultaneously, the sequence recognition network is also replaced with a Transformer architecture, utilizing its parallel computing capabilities to reduce computation time and simplify deployment. Furthermore, optimizations such as feature map compression, matrix segmentation, and operator replacement are performed for embedded devices, making the model easier to deploy on embedded devices and resulting in shorter response times.

[0031] A second aspect of this application provides a license plate character recognition device based on deep learning, see [link to relevant documentation]. Figure 2 , Figure 2 A schematic diagram of a deep learning-based license plate character recognition device provided in this application embodiment includes: Image acquisition module 201 is used to acquire the image of the license plate to be recognized; The backbone network module 202 is used to input the license plate image to be identified into the backbone network and calculate the features to be encoded. The backbone network includes multiple network layers connected in sequence. Each network layer includes: a patch embedding module, a first image feature extraction module, a spatial location awareness module, and a second image feature extraction module. The processing procedure of each network layer includes: expanding the input data one-dimensionally through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information annotation and feature extraction on the calculation result of the first image feature extraction module through the spatial location awareness module to obtain the output data of the spatial location awareness module; and performing feature extraction and convolution calculation on the output data of the spatial location awareness module through the second image feature extraction module to obtain the calculation result of the second image feature extraction module. The feature encoding module 203 is used to input the feature to be encoded into the encoder to obtain a license plate recognition string. The encoder includes a position embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module. The encoder's processing includes: performing one-dimensional position encoding on the feature to be encoded through the position embedding module to obtain position encoded features; sequentially inputting the position encoded features into the third image feature extraction module and the fourth image feature extraction module to extract global features through an attention mechanism; calculating the global features through the batch normalization module to obtain the encoding result; and inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string.

[0032] In one possible implementation, the backbone network module is specifically used to downsample the license plate image to be identified through the patch embedding module to obtain downsampled image data; expand the downsampled image data to obtain image data of a specified size; and expand the image data of the specified size into the one-dimensional sequence data.

[0033] In one possible implementation, the backbone network module is specifically used to extract correlation information between characters in the license plate from the one-dimensional sequence data through the first image feature extraction module; when the license plate image to be identified is distorted and / or blurred, global information is extracted from the one-dimensional sequence data through the first image feature extraction module, wherein the calculation result of the first image feature extraction module includes: correlation information between characters in the license plate and global information.

[0034] In one possible implementation, the backbone network module is specifically used to downsample the one-dimensional sequence data using a spatial reduction attention mechanism through the first image feature extraction module to obtain downsampled sequence data, wherein the length of the downsampled sequence data is less than the length of the one-dimensional sequence data; and to extract global features from the downsampled sequence data using an attention mechanism to obtain the calculation result of the first image feature extraction module.

[0035] In one possible implementation, the feature encoding module is specifically used to input the encoding result into the fully connected module, and calculate a three-dimensional feature tensor through the fully connected module, wherein the dimensions of the three-dimensional feature tensor correspond to the sequence length, the total number of character categories, and the batch size, respectively; and perform connection-time classification decoding on the three-dimensional feature tensor to obtain the license plate recognition string.

[0036] As can be seen, the solution proposed in this application can not only expand the image data of the specified specifications into one-dimensional sequence data through the backbone network, and then perform feature extraction and convolution calculation through the visual core layer to reduce the computational complexity, but also reduce the computational complexity by normalizing and convolution calculation on the features to be encoded, thus solving the problem of high computational complexity in the license plate recognition process and improving response speed and efficiency.

[0037] In another aspect of the embodiments of this application, an electronic device is also provided, see [link to relevant documentation]. Figure 3 ,include: Memory 301 is used to store computer programs; When processor 302 executes a program stored in memory, it implements the following: Acquire the image of the license plate to be identified; The license plate image to be identified is input into the backbone network to calculate the features to be encoded. The backbone network includes multiple sequentially connected network layers. Each network layer includes: a patch embedding module, a first image feature extraction module, a spatial location awareness module, and a second image feature extraction module. The processing procedure of each network layer includes: expanding the input data one-dimensionally through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information annotation and feature extraction on the calculation result of the first image feature extraction module through the spatial location awareness module to obtain the output data of the spatial location awareness module; and performing feature extraction and convolution calculation on the output data of the spatial location awareness module through the second image feature extraction module to obtain the calculation result of the second image feature extraction module. The features to be encoded are input into an encoder to obtain a license plate recognition string. The encoder includes a location embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module, and a fully connected module. The encoder's processing includes: performing one-dimensional location encoding on the features to be encoded through the location embedding module to obtain location-encoded features; sequentially inputting the location-encoded features into the third image feature extraction module and the fourth image feature extraction module to extract global features through an attention mechanism; calculating the global features through the batch normalization module to obtain the encoding result; and inputting the encoding result into the fully connected module for temporal classification decoding to obtain the license plate recognition string.

[0038] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0039] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0040] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0041] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0042] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described deep learning-based license plate character recognition methods.

[0043] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, enables the computer to implement any of the above-described deep learning-based license plate character recognition methods.

[0044] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0045] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0046] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For related parts, please refer to the description of the method embodiments.

[0047] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A deep learning-based license plate character recognition method, characterized in that, The method comprises: obtaining a to-be-recognized license plate image; inputting the to-be-recognized license plate image into a backbone network to obtain to-be-encoded features; the backbone network comprises a plurality of network layers connected in sequence, and each network layer comprises a patch embedding module, a first image feature extraction module, a spatial position perception module and a second image feature extraction module; wherein the processing process of each network layer comprises: performing one-dimensional expansion on input data through the patch embedding module to obtain one-dimensional sequence data; performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module to obtain the calculation result of the first image feature extraction module; performing information labeling and feature extraction on the calculation result of the first image feature extraction module through the spatial position perception module to obtain the output data of the spatial position perception module; and performing feature extraction and convolution calculation on the output data of the spatial position perception module through the second image feature extraction module to obtain the calculation result of the second image feature extraction module; inputting the to-be-encoded features into an encoder to obtain a license plate recognition string; the encoder comprises a position embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module and a full connection module; wherein the processing process of the encoder comprises: performing one-dimensional position encoding on the to-be-encoded features through the position embedding module to obtain position encoding features; inputting the position encoding features into the third image feature extraction module and the fourth image feature extraction module in sequence to extract global features through an attention mechanism; performing calculation on the global features through the batch normalization module to obtain an encoding result; and inputting the encoding result into the full connection module for time series classification decoding to obtain the license plate recognition string.

2. The method of claim 1, wherein, The one-dimensional sequence data obtained by performing one-dimensional expansion on the input data through the patch embedding module comprises: performing down-sampling on the to-be-recognized license plate image through the patch embedding module to obtain down-sampled image data; expanding the down-sampled image data to obtain image data of a specified specification; and expanding the image data of the specified specification into the one-dimensional sequence data.

3. The method of claim 1, wherein, The calculation result of the first image feature extraction module obtained by performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module comprises: extracting correlation information between characters in the license plate from the one-dimensional sequence data through the first image feature extraction module; when the to-be-recognized license plate image is distorted and / or blurred, performing global information extraction on the one-dimensional sequence data through the first image feature extraction module, wherein the calculation result of the first image feature extraction module comprises correlation information between characters in the license plate and global information.

4. The method of claim 1, wherein, The calculation result of the first image feature extraction module obtained by performing feature extraction and convolution calculation on the one-dimensional sequence data through the first image feature extraction module comprises: The first image feature extraction module is used for performing down-sampling on the one-dimensional sequence data by using a spatial reduction attention mechanism, so as to obtain down-sampled sequence data, wherein the length of the down-sampled sequence data is less than the length of the one-dimensional sequence data. The down-sampled sequence data is used for performing global feature extraction by using an attention mechanism, so as to obtain a calculation result of the first image feature extraction module.

5. The method of claim 1, wherein, The encoding result is input into the full connection module to perform time sequence classification decoding, so as to obtain the license plate recognition string. The encoding result is input into the full connection module to obtain a three-dimensional feature tensor by calculation of the full connection module, wherein the dimensions of the three-dimensional feature tensor correspond to sequence length, total number of character categories and batch size respectively. The three-dimensional feature tensor is used for connection time sequence classification decoding, so as to obtain the license plate recognition string.

6. A deep learning-based license plate character recognition device, characterized by, The image acquisition module is configured to acquire a to-be-recognized license plate image. The backbone network module is configured to input the to-be-recognized license plate image into a backbone network to obtain to-be-encoded features; the backbone network comprises a plurality of network layers connected in sequence, each network layer comprising a patch embedding module, a first image feature extraction module, a spatial position perception module and a second image feature extraction module; wherein the processing process of each network layer comprises: the patch embedding module is used for one-dimensional expansion on input data to obtain one-dimensional sequence data; the first image feature extraction module is used for feature extraction and convolution calculation on the one-dimensional sequence data to obtain a calculation result of the first image feature extraction module; the spatial position perception module is used for information labeling and feature extraction on the calculation result of the first image feature extraction module to obtain output data of the spatial position perception module; and the second image feature extraction module is used for feature extraction and convolution calculation on the output data of the spatial position perception module to obtain a calculation result of the second image feature extraction module. The feature encoding module is configured to input the to-be-encoded features into an encoder to obtain a license plate recognition string; the encoder comprises a position embedding module, a third image feature extraction module, a fourth image feature extraction module, a batch normalization module and a full connection module; wherein the processing process of the encoder comprises: the position embedding module is used for one-dimensional position encoding on the to-be-encoded features to obtain position encoding features; the position encoding features are sequentially input into the third image feature extraction module and the fourth image feature extraction module to extract global features by using an attention mechanism; the batch normalization module is used for calculation on the global features to obtain an encoding result; and the encoding result is input into the full connection module to perform time sequence classification decoding, so as to obtain the license plate recognition string.

7. The apparatus of claim 6, wherein ​ The trunk network module is specifically configured to perform down-sampling on the to-be-recognized license plate image through the patch embedding module to obtain down-sampled image data, perform expansion on the down-sampled image data to obtain image data of a specified specification, and expand the image data of the specified specification into the one-dimensional sequence data.

8. The apparatus of claim 6, wherein, The trunk network module is specifically configured to perform correlation information extraction between characters in a license plate on the one-dimensional sequence data through the first image feature extraction module. When the to-be-recognized license plate image is distorted and / or blurred, the first image feature extraction module is used to perform global information extraction on the one-dimensional sequence data, wherein the calculation result of the first image feature extraction module includes correlation information between characters in a license plate and global information.

9. An electronic device, comprising: Comprising: a memory for storing a computer program; a processor for executing the program stored on the memory to implement the method of any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-5.

Images