Strong generalization multi-type license plate recognition system based on decoupling of packet features

Abstract

The application discloses a strong generalization multi-type license plate recognition system based on decoupling of grouping features, which comprises a feature extraction module containing a backbone network, a group module, an indirect supervision module and a feature decoupling decoding module containing a character decoder, the application fits global license plate image feature maps of real license plates through feature jitter, then extracts license plate type information as type noise from instance features through a two-stage processing mode of extraction and decoupling of fused grouping features containing implicit multi-dimensional visual representation and explicit license plate category information, removes the type noise from the instance features through grouping feature decoupling, and finally predicts license plate content by using a character decoder based on a single character attention mechanism. The application can be well generalized to a real scene and greatly reduce the cost while ensuring the performance.

Description

Technical Field

[0001] This invention relates to a technology in the field of image processing, specifically a highly generalizable multi-type license plate recognition system based on grouped feature decoupling. Background Technology

[0002] Existing license plate recognition technologies tend to group different types of license plates into the same set, ignoring the category information between various license plate types. Furthermore, due to the lack of existing datasets for recognizing multiple types of license plates, most current algorithms are trained using synthetic data. However, the inherent domain differences between synthetic and real data make it difficult for license plate recognition models trained on synthetic datasets to generalize to real license plates. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies, such as the neglect of differences between different types of license plates leading to convergence to suboptimal solutions and low accuracy in recognizing real-world license plates. It proposes a highly generalizable multi-type license plate recognition system based on grouped feature decoupling. For cross-category classification tasks, it improves the robustness of the license plate recognizer to different types of license plates by extracting and decoupling grouped information. Furthermore, it enhances zero-shot domain generalization through feature jittering. The results trained on computer-synthesized non-real data can generalize well to real-world scenarios, significantly reducing costs while ensuring performance.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a highly generalizable multi-type license plate recognition system based on grouped feature decoupling, comprising: a feature extraction module including a backbone network, a group module, an indirect supervision module, and a feature decoupling decoding module including a character decoder. Specifically: the feature extraction module extracts multi-scale features from the original image based on the license plate image, representing different receptive domain features, and fuses multiple multi-scale features of different scales to form a global license plate image feature map; the group module extracts fused grouped features of the multi-dimensional license plate image from the global license plate image feature map for implicit supervision, and then generates instance features containing the fused grouped features and the features to be decoded for grouped feature decoupling; the indirect supervision module extracts the license plate category information from the fused grouped features for explicit supervision to obtain license plate category classification; the feature decoupling decoding module decouples the features to be decoded from the instance features, and obtains a decoupled single-character vector through the single-character attention map output by the character decoder, thereby realizing license plate recognition.

[0006] This invention relates to a highly generalizable multi-type license plate recognition method based on the above system. First, the global license plate image feature map of the real license plate is fitted by feature dithering. Then, through a two-stage processing method of extraction and decoupling of fused group features containing implicit multi-dimensional visual representation and explicit license plate category information, the license plate type information is extracted from the instance features as type noise. After the type noise is removed from the instance features by decoupling the group features, the license plate type noise is removed from the instance features. Finally, a character decoder based on a single character attention mechanism is used to predict the license plate content. Technical effect

[0007] This invention extracts grouping information from the multidimensional visual representation and category information of license plates through a parallel approach of implicit and explicit supervision. By constraining the grouping information to be orthogonal to instance features, the grouping information is decoupled from the instance features, allowing the network to focus more on the character-level information of the license plate features. Feature dithering simulates inputs from different domains at the feature level, reducing the performance degradation caused by domain differences between synthetic and real data. Compared with existing technologies, the grouping information extracted in this invention allows the network to optimize independently for different types of license plates. This decoupling removes noise introduced by license plate category information, allowing the network to focus more on unified character-level features and improving network performance. Feature dithering transforms real data from invisible domains into visible domain data, enabling models trained on purely synthetic datasets to be well transferred to real data, significantly reducing annotation costs. Attached Figure Description

[0008] Figure 1 This is a flowchart of the present invention;

[0009] Figure 2 This is a schematic diagram of the system of the present invention;

[0010] Figure 3 A schematic diagram of a character decoder;

[0011] Figure 4 This is a flowchart of an implementation example; Detailed Implementation

[0012] like Figure 2As shown in the figure, this embodiment relates to a highly generalizable multi-type license plate recognition system based on grouped feature decoupling. The system includes a feature extraction module with a backbone network, a grouping module, an indirect supervision module, and a feature decoupling decoding module. Specifically: the feature extraction backbone network extracts multi-scale features from the original image based on the license plate image, representing different receptive domain features, and fuses these multi-scale features to form a global license plate image feature map; the grouping module extracts fused grouped features of the multi-dimensional license plate image from the global license plate image feature map for implicit supervision, and then generates instance features containing the fused grouped features and the features to be decoded for grouped feature decoupling; the feature decoupling decoding module decouples the fused grouped features from the instance features, enabling the character decoder to focus more on the character-level features of the license plate and suppress noise caused by the license plate grouping information, obtaining the features to be decoded. Through the single-character attention map output by the character decoder, a decoupled single-character vector is obtained, thus achieving license plate recognition.

[0013] The feature extraction backbone network includes: a feature fusion unit and shallow feature extraction units, mid-level feature extraction units, and deep feature extraction units connected to it, wherein: the shallow feature extraction unit extracts shallow features f based on the input license plate image X. L The middle-layer feature extraction unit extracts features based on the shallow-layer features f. L Feature extraction is performed to obtain the mid-level feature f. M Deep feature extraction unit, layer features f M Deep features f are obtained by feature extraction. S Feature fusion unit fuses shallow features f L Mid-layer features f M Deep features f S The global license plate image feature map f is obtained.

[0014] The shallow feature extraction unit includes: a first convolutional module composed of a set of residual blocks, wherein: the input license plate image of the first convolutional module is X∈R C×H×W Output shallow features f L ∈R 32×(H / 2)×(W / 2) Where: W is the width of the license plate image, H is the height of the license plate image, and C is the image color channel. In this embodiment, they are 64, 256, and 3 respectively.

[0015] The mid-layer feature extraction unit includes: second, third, and fourth convolutional modules, each composed of a set of residual blocks, wherein: the second convolutional module takes shallow features f as input. L ∈R 32×(H / 2)×(W / 2) Output Input to the third convolution module Output Input of the fourth convolution module Output mid-level features f M ∈R 128×(H / 2)×(W / 2) .

[0016] The deep feature extraction unit includes: fifth, sixth, and seventh convolutional modules, each composed of a set of residual blocks, wherein the fifth convolutional module takes as input the intermediate layer feature f. M ∈R 128×(H / 2)×(W / 2) Output Input of the sixth convolution module Output Input of the seventh convolution module Output deep features

[0017] The feature fusion unit fuses multi-scale features to generate a fused feature map f∈R. 512 ×(H / 4)×(W / 4) Wherein: a new feature map f = Conv(Conv(f) containing feature information from three scales of the field of view L )+f M )+f S First, the shallow features f L After convolution operation, the size and the middle layer feature f are adjusted. M After addition, and then undergoing another convolution operation to reduce the size and increase the number of channels, it is combined with the deep feature f. S By adding them together, the problem of inconsistent license plate character sizes can be solved.

[0018] In this embodiment, the first to seventh convolutional blocks simulate data from different domains by using feature jitter on the input features, in order to reduce the domain differences between synthetic and real data. Specifically: in: The first to seventh convolutional blocks represent the features of the input, i∈[1,7], α~N(1,θ) α ), β~N(1,θ β The weighting parameters are randomly initialized according to a Gaussian distribution, and the hyperparameter θ is... α and θ β It is the boundary of the characteristic jitter.

[0019] In this embodiment, the boundaries of the characteristic jitter are preferably set to 0.8 and 0.3, respectively.

[0020] The group module includes an instance feature extraction unit, a group perception unit, and a group decision unit. Specifically: the instance feature extraction unit extracts instance features from the fused feature map f through 3×3 convolutions; these instance features are then pooled and processed by two fully connected layers to obtain an N-dimensional vector; the group perception unit performs multiple sets of 3×3 convolutions on the fused feature map f to obtain group perception feature results; the group decision unit weights and sums the above group perception features based on the N-dimensional vector obtained from the instance feature mapping to obtain fused group features. Self-distributed labels are introduced at the N-dimensional vector as supervision to generate uniformly distributed group labels. This structure enables the network to autonomously group license plates based on the multi-dimensional visual representation of the license plate image itself, thereby obtaining group perception features. Unlike the explicit semantic information brought by license plate categories, group perception features often represent higher-dimensional information, such as the environment. Although such grouping results are visually difficult to understand, they can distinguish different license plates from the perspective of implicit information and enrich the information contained in the fused group features.

[0021] The indirect supervision module uses license plate category as additional supervision information for the grouping module. It classifies license plate categories based on the fused grouping features of the grouping module to fully utilize category supervision information, i.e., explicit information. Simultaneously, this supervision does not force each group to be restricted to a specific license plate category; that is, it is indirect supervision. This indirect supervision module includes several convolutional blocks and a fully connected layer. The convolutional blocks perform 3×3 convolution and pooling based on the fused grouping features to obtain smoothed fused grouping information. The fully connected layer predicts the license plate type based on the smoothed fused grouping information, obtaining the predicted license plate type.

[0022] The feature decoupling and decoding module includes a category decoupling module and a character decoder, wherein the category decoding module performs loss function calculation, specifically the orthogonal constraint loss. orth =Dot(F ins ,F fused ), where: F ins F represents instance features. fused The fused grouped features are represented by Dot(·,·), which represents the dot product. By adding orthogonal constraints to the instance features and the fused grouped features, we obtain the features to be decoded that focus on character-level features.

[0023] like Figure 3 As shown, the character decoder is a decoder based on a decoupled attention mechanism to simultaneously recognize license plate characters of indeterminate line length. Specifically, it involves first dividing the features to be decoded, F ∈ R... C×H×W The data is fed into the single-character attention module to obtain T single-character attention maps α. t ∈R 1×H×WWhere T represents the number of characters in the license plate image, which is set to 9 in this embodiment; with α t As weights, the features at different positions in the feature F to be decoded are weighted and summed to obtain the decoupled single-character feature vector. c t ∈R 512×1 The single character feature vector c decoupled from T t The data is sent to a classification head to predict the character classification result. This character decoder includes 4 convolutional layers, 4 deconvolutional layers, and 9 fully connected layers. The convolutional layers downsample the features to be decoded to obtain extracted decoded features. The deconvolutional layers upsample the extracted decoded features to obtain T single-character attention maps. The fully connected layers use the single-character feature vector, obtained by weighting and summing the original features to be decoded from the T single-character attention maps, to predict the license plate character and obtain the license plate character prediction result.

[0024] Through specific practical experiments, in a single NVIDIA RTX3090 environment, using PyTorch as the inference framework, and a synthetic dataset containing multiple types of license plates in China as the training dataset, the network was trained using self-distribution loss, orthogonal constraint loss, attention map error, and string prediction error as supervisory losses. Specifically, the attention map error is used to constrain the consistency between the single-character attention map obtained by the multi-type license plate recognition system with a non-grouped feature decoupling module and the single-character attention map obtained by the multi-type license plate recognition system with a grouped feature decoupling module during the training process.

[0025] Dataset accuracy testing: The robustness of a model is one of the most fundamental indicators for evaluating the quality of an algorithm. Although this embodiment is designed for multi-class license plate recognition tasks, it should also achieve equally excellent performance on single-class tasks. The results are shown in Table 1, and comparable results are obtained compared with the best results in existing single-class license plate recognition tasks.

[0026] Table 1. Accuracy Test of CCPD Blue License Plate

[0027] The accuracy of mixed-type license plate recognition tests is shown in Table 2. In multi-type license plate recognition tasks, this embodiment achieves results far exceeding those of existing methods.

[0028] Table 2 Accuracy Test of Mixed Types of License Plates

[0029] The accuracy of the mixed-type license plate ablation experiment is shown in Table 3. The introduction of group and indirect supervision greatly improves the performance of the license plate recognizer in multi-class cases, while group feature decoupling and feature dithering push the recognizer's performance to an even higher level. Therefore, this embodiment achieves performance far exceeding the baseline (+9.5%).

[0030] Table 3. Accuracy of Mixed Type License Plate Testing

[0031] For multi-class license plate recognition tasks, the model's generalization to invisible categories is crucial because license plates in the real world often vary by country and region. Given the high cost of data collection, a good model should be able to generalize from visible categories to multiple invisible categories.

[0032] The generalization capability of this embodiment is shown in Table 4, where the occlusion category represents the type of license plate that is not visible during the training process.

[0033] Table 4 shows the model generalization test accuracy. Here, the occlusion category refers to the category of license plates that are not included in the training.

[0034] Obscuring a category of license plates does indeed lead to a decrease in average accuracy, but it remains somewhat universal for invisible categories, meaning the performance decrease is not significant.

[0035] Compared to existing technologies, this method significantly improves the performance of license plate recognition in multi-class scenarios by introducing group and indirect supervision, resulting in a 5% performance improvement. Furthermore, group feature decoupling allows the network to focus more on character-level features, leading to a 1.4% performance improvement. Feature dithering reduces the domain difference between synthetic and real data, transforming real data from an invisible domain to a visible domain, thus further improving model accuracy by 3.1%. Ultimately, this invention achieves a 9.5% performance improvement compared to the baseline network.

[0036] This invention achieves high recognition accuracy, reaching 99.4% on the CCPC public dataset. On a multi-category license plate dataset, by extracting and decoupling grouped features, the model focuses more on character-level features of the license plate, ignoring license plate type noise. This allows the model to be compatible with multiple types of license plates and maintain high recognition performance even for license plates not shown in the training set. Feature dithering simulates feature inputs from different domains in the feature domain, transforming the invisible domain of the real license plate image into the visible domain, reducing the domain difference between the synthetic and real domains. Therefore, it effectively avoids the performance degradation caused by domain differences between synthetic and real datasets. Ultimately, this invention achieves 87.1% accuracy on a multi-category license plate dataset with purely synthetic data.

[0037] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A highly generalizable multi-type license plate recognition system based on grouped feature decoupling, characterized in that, include: The system comprises a feature extraction module, a grouping module, an indirect supervision module, and a feature decoupling and decoding module containing a character decoder. Specifically: the feature extraction module extracts multi-scale features from the original license plate image, representing different receptive domain features, and fuses these multi-scale features to form a global license plate image feature map; the grouping module extracts fused grouping features from the global license plate image feature map for implicit supervision, and then generates instance features containing the fused grouping features and the features to be decoded for decoupling the grouping features; the indirect supervision module extracts license plate category information from the fused grouping features for explicit supervision to obtain license plate category classification; and the feature decoupling and decoding module decouples the features to be decoded from the instance features, and obtains a decoupled single-character vector through the single-character attention map output by the character decoder, thus achieving license plate recognition. The aforementioned feature extraction backbone network simulates data from different domains through feature jittering to reduce the domain differences between synthetic and real data. Specifically: ,in: These are the features output by the first through seventh convolutional modules. , , To randomly initialize the weighting parameters and hyperparameters according to a Gaussian distribution and It is the boundary of the characteristic jitter.

2. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 1, characterized in that, The feature extraction backbone network includes: a feature fusion unit and shallow feature extraction units, mid-level feature extraction units, and deep feature extraction units connected to it, wherein: the shallow feature extraction unit is based on the input license plate image. shallow features are obtained by feature extraction. The middle-layer feature extraction unit extracts features based on shallow-layer features. Feature extraction is performed to obtain mid-level features. The deep feature extraction unit extracts features based on the mid-layer features. Deep features are obtained through feature extraction. Feature fusion unit fuses shallow features Mid-layer characteristics and deep features Obtain global license plate image feature map .

3. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 2, characterized in that, The shallow feature extraction unit includes: a first convolutional module composed of a set of residual blocks, wherein the input to the first convolutional module is a license plate image. Output shallow features ,in: For the width of the license plate image, For license plate image height, For image color channels.

4. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 2, characterized in that, The mid-layer feature extraction unit includes: second, third, and fourth convolutional modules, each composed of a set of residual blocks, wherein: the second convolutional module takes shallow features as input. Output Input of the third convolution module Output The fourth convolution module input Output mid-level features .

5. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 2, characterized in that, The deep feature extraction unit includes: fifth, sixth, and seventh convolutional modules, each composed of a set of residual blocks, wherein the fifth convolutional module takes into account the mid-layer features. Output Input of the sixth convolution module Output The seventh convolution module input Output deep features .

6. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 2, characterized in that, The feature fusion unit fuses multi-scale features to generate a fused feature map. Among them: a new feature map containing feature information from three scales of the field of view. First, shallow features The size and middle-layer features are adjusted after convolution operations. The components are added together, then subjected to another convolution operation to reduce their size and increase the number of channels before being combined with deeper features. By adding them together, the problem of inconsistent license plate character sizes can be solved.

7. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 1, characterized in that, The group module includes: an instance feature extraction unit, a group awareness unit, and a group decision unit, wherein: the instance feature extraction unit extracts features based on the fused feature map. The instance features are obtained through 3×3 convolution, and then pooled and processed by two fully connected layers to obtain an N-dimensional vector. The group perception unit then uses the fused feature map... Multiple sets of 3×3 convolutions are performed to obtain group-aware feature results. The group decision unit performs a weighted summation of the above group-aware features based on the N-dimensional vector obtained by the instance feature mapping to obtain fused group features. Self-distributed labels are introduced at the N-dimensional vector as supervision to generate uniformly distributed group labels.

8. The highly generalizable multi-type license plate recognition system based on grouped feature decoupling according to claim 1, characterized in that, The feature decoupling and decoding module includes a category decoupling module and a character decoder, wherein the category decoupling module performs loss function calculation, specifically an orthogonal constraint loss. ,in: Representing instance characteristics, Indicates the fusion grouping features, The dot product is represented by adding orthogonal constraints to the instance features and the fused grouped features to obtain the features to be decoded that focus on character-level features; The character decoder described is based on a decoupled attention mechanism to simultaneously recognize license plate characters of varying lines and lengths. Specifically, it first processes the features to be decoded... The input is fed into a single-character attention module, and the result is... Zhang's single-character attention map ,in: Represents the number of characters in the license plate image, in As weights, the features to be decoded We obtain a decoupled single-character feature vector by weighted summation of features at different positions. , ,Will Decoupled single character feature vector The data is sent to a classification head to predict the character classification result. This character decoder includes 4 convolutional layers, 4 deconvolutional layers, and 9 fully connected layers. Specifically, the convolutional layers downsample the features to be decoded to obtain extracted decoded features, and the deconvolutional layers upsample the extracted decoded features to obtain... The T single-character attention maps are used to perform license plate character prediction in the fully connected layer based on the single-character feature vector obtained by weighted summation of the original features to be decoded from the T single-character attention maps.

9. A highly generalizable multi-type license plate recognition method based on the system described in any one of claims 1-8, characterized in that, First, a global license plate image feature map of the real license plate is fitted by feature dithering. Then, through a two-stage processing method of extraction and decoupling of fused group features containing implicit multidimensional visual representation and explicit license plate category information, the license plate type information is extracted from the instance features as type noise. After the type noise is removed from the instance features by decoupling the group features, the license plate type noise is removed from the instance features. Finally, a character decoder based on a single character attention mechanism is used to predict the license plate content.

Images