License plate recognition method and device

Abstract

The present application discloses a license plate recognition method and device. The method includes: training a feature extraction network and a target detection network based on a basic category group to obtain a basic feature extraction network and a basic target detection network; the number of character images of each basic category in a balanced character dataset is the same as the number of character images of each novel category; initializing the parameters of the feature extraction network based on the novel category group in the balanced character dataset to obtain an initialized feature extraction network; fusing the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network; extracting features from the balanced character dataset based on the full feature extraction network to obtain multiple balanced feature maps; updating the parameters of the basic target detection network based on the multiple balanced feature maps to obtain a full target detection network. Using the full feature extraction network and the full target detection network, the license plate image to be tested is detected and recognized to obtain a license plate character sequence.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a license plate recognition method and device. Background Art

[0002] License plate recognition technology plays an important role in many tasks such as urban traffic management, vehicle identification, parking fee management, and violation handling. However, it is still a challenging task due to the influence of many factors such as lighting conditions, weather conditions, image clarity, license plate shooting angle, and license plate color.

[0003] However, traditional recognition methods require detection and recognition based on a large amount of license plate data. However, for special types of license plate data, such as license plates from the Hong Kong Special Administrative Region of my country, license plates from the Macao Special Administrative Region of my country, military license plates, temporary license plates related to non-motor vehicle management, or special license plates used for autonomous vehicle testing, there is less license plate data and fewer channels for collection. Compared with common types of license plate data, the proportion of special types of license plate data is small, resulting in the traditional recognition method being prone to omission or misidentification of special types of license plate data with less data during the recognition process, which in turn leads to low recognition accuracy of traditional recognition methods. Summary of the Invention

[0004] The purpose of this application is to solve the technical problem of low recognition accuracy caused by missed recognition or incorrect recognition during the recognition of special types of license plate data by traditional recognition methods. To achieve the above purpose, this application provides a license plate recognition method and device.

[0005] This application provides a license plate recognition method, comprising:

[0006] Obtaining a basic category group and a novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position;

[0007] Training the feature extraction network and the target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network;

[0008] constructing a balanced character dataset based on the basic category group and the novel category group, wherein the number of character images of each basic category in the balanced character dataset is the same as the number of character images of each novel category;

[0009] Initializing parameters of the feature extraction network according to the novel category group in the balanced character dataset to obtain an initialized feature extraction network, and fusing the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network;

[0010] Performing feature extraction on the balanced character dataset according to the full feature extraction network to obtain a plurality of balanced feature maps, and updating parameters of the basic object detection network according to the plurality of balanced feature maps to obtain a full object detection network;

[0011] According to the full feature extraction network and the full target detection network, the license plate image to be tested is detected and recognized to obtain a license plate character sequence.

[0012] In one embodiment, before obtaining the basic category group and the novel category group, the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position, the method further includes:

[0013] Acquire multiple license plate images, and obtain multiple character images based on each license plate image;

[0014] The plurality of character images are screened according to the number of character categories corresponding to each character image to obtain the basic category group and the novel category group.

[0015] In one embodiment, the training of the feature extraction network and the target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network includes:

[0016] Inputting the character images of the basic category group into the feature extraction network and outputting a plurality of basic feature maps;

[0017] Inputting the multiple basic feature maps into the object detection network, and outputting a predicted basic character category and a predicted basic character position of each basic character image;

[0018] A basic loss function is constructed according to the predicted basic character category, the predicted basic character position, the real character category and the real character position, and the feature extraction network and the target detection network are trained to obtain the basic feature extraction network and the basic target detection network.

[0019] In one embodiment, the step of extracting features from the balanced character dataset based on the full feature extraction network to obtain a plurality of balanced feature maps, and updating the parameters of the basic object detection network based on the plurality of balanced feature maps to obtain a full object detection network includes:

[0020] Inputting the balanced character data set into the full feature extraction network and outputting the plurality of balanced feature maps;

[0021] Inputting the multiple balanced feature maps into the basic object detection network, and outputting the predicted full character category and predicted full character position of each character image;

[0022] A full loss function is constructed according to the predicted full character category, the predicted full character position, the real character category and the real character position, and the parameters of the basic object detection network are updated to obtain the full object detection network.

[0023] In one embodiment, constructing a full loss function based on the predicted full character category, the predicted full character position, the true character category, and the true character position, updating the parameters of the basic object detection network, and obtaining the full object detection network includes:

[0024] Constructing a full classification loss function based on the predicted full character categories and the true character categories;

[0025] A full position regression loss function is constructed based on the predicted full character position and the true character position. The full position regression loss function is:

[0026]

[0027] Among them, b pre represents the center coordinates of the predicted full character position, b gt represents the center point coordinates of the real character position, ρ represents the Euclidean distance between the center point coordinates of the predicted full character position and the center point coordinates of the real character position, c represents the diagonal distance of the minimum circumscribed rectangular box containing the predicted full character position and the real character position, and IoU represents the IoU intersection-over-union ratio between the predicted full character position and the real character position.

[0028] In one embodiment, the basic category group and the novel category group are obtained, the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, each of the character images includes a real character category and a real character position, and the ratio of the number of categories in the basic category group to the number of categories in the novel category group is in a range of 3:1 to 4:1.

[0029] In one embodiment, the present application provides a license plate recognition device, comprising:

[0030] a data acquisition module, configured to acquire a basic category group and a novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position;

[0031] A basic network generation module, configured to train a feature extraction network and a target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network;

[0032] a balanced character data set acquisition module, configured to construct a balanced character data set based on the basic category group and the novel category group, wherein the number of character images of each basic category in the balanced character data set is the same as the number of character images of each novel category;

[0033] a full feature extraction network generation module, configured to initialize parameters of the feature extraction network according to the novel category group in the balanced character dataset to obtain an initialized feature extraction network, and fuse the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network;

[0034] a full object detection network generation module, configured to perform feature extraction on the balanced character dataset according to the full feature extraction network to obtain a plurality of balanced feature maps, and update the parameters of the basic object detection network according to the plurality of balanced feature maps to obtain a full object detection network;

[0035] The recognition module is used to detect and recognize the license plate image to be tested based on the full feature extraction network and the full target detection network to obtain the license plate character sequence.

[0036] In one embodiment, the apparatus further comprises:

[0037] A character image acquisition module is used to acquire a plurality of license plate images, and obtain a plurality of character images according to each of the license plate images;

[0038] The grouping module is configured to screen the plurality of character images according to the number of character categories corresponding to each character image, to obtain the basic category group and the novel category group.

[0039] In one embodiment, the basic network generation module includes:

[0040] A basic feature map acquisition module, configured to input the character images of the basic category group into the feature extraction network and output a plurality of basic feature maps;

[0041] A predicted basic character information acquisition module is used to input the multiple basic feature maps into the object detection network and output a predicted basic character category and a predicted basic character position of each basic character image;

[0042] The basic network model training module is used to construct a basic loss function based on the predicted basic character category, the predicted basic character position, the real character category and the real character position, train the feature extraction network and the target detection network, and obtain the basic feature extraction network and the basic target detection network.

[0043] In one embodiment, the full object detection network generation module includes:

[0044] A balanced feature map acquisition module, configured to input the balanced character dataset into the full feature extraction network and output the plurality of balanced feature maps;

[0045] A predicted full-character information acquisition module, configured to input the plurality of balanced feature maps into the basic object detection network and output a predicted full-character category and a predicted full-character position for each character image;

[0046] The full-target detection network training module is used to construct a full loss function based on the predicted full-character category, the predicted full-character position, the real character category and the real character position, update the parameters of the basic target detection network, and obtain the full-target detection network.

[0047] In one embodiment, the full object detection network training module includes:

[0048] A full classification loss function construction module, configured to construct a full classification loss function based on the predicted full character categories and the true character categories;

[0049] The full position regression loss function construction module is used to construct a full position regression loss function based on the predicted full character position and the true character position. The full position regression loss function is:

[0050]

[0051] Among them, b pred represents the center point coordinates of the predicted full character position, bgt represents the center point coordinates of the real character position, ρ represents the Euclidean distance between the center point coordinates of the predicted full character position and the center point coordinates of the real character position, c represents the diagonal distance of the minimum circumscribed rectangular box containing the predicted full character position and the real character position, and IoU represents the IoU intersection-over-union ratio between the predicted full character position and the real character position.

[0052] In one embodiment, in the data acquisition module, the ratio of the number of categories in the basic category group to the number of categories in the novel category group ranges from 3:1 to 4:1.

[0053] In the above-mentioned license plate recognition method and device, all character category data are divided into a basic category group and a novel category group, and a balanced character data set is constructed based on the basic category group and the novel category group, thereby realizing the learning of the basic stage and the fine-tuning stage in the network model training process, obtaining a balanced and stable full-feature extraction network and a full-target detection network, and realizing the detection and recognition of the license plate image to be tested. In addition, the license plate recognition method provided in this application fine-tunes the basic target detection network based on the balanced character data set, so that the proportion of special type license plate data and common type license plate data is balanced, which can achieve better recognition effect for special type license plate data with small data volume, avoid the problem of missed recognition or incorrect recognition of special type license plate data with small data volume during the recognition process, and improve the detection and recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of the steps of the license plate recognition method provided by this application.

[0055] Figure 2 It is a structural diagram of the license plate recognition device provided in this application. DETAILED DESCRIPTION

[0056] The technical solution of the present application is further described in detail below through the accompanying drawings and examples.

[0057] See Figure 1 , this application provides a license plate recognition method, comprising:

[0058] S10, obtaining a basic category group and a novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position;

[0059] S20, training the feature extraction network and the target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network;

[0060] S30, constructing a balanced character dataset based on the basic category group and the novel category group, wherein the number of character images of each basic category in the balanced character dataset is the same as the number of character images of each novel category;

[0061] S40, initializing parameters of a feature extraction network according to the novel category group in the balanced character dataset to obtain an initialized feature extraction network, and fusing the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network;

[0062] S50, extracting features from the balanced character dataset using the full feature extraction network to obtain a plurality of balanced feature maps, and updating parameters of the basic object detection network based on the plurality of balanced feature maps to obtain a full object detection network;

[0063] S60, detecting and identifying the license plate image to be tested based on the full feature extraction network and the full target detection network to obtain a license plate character sequence.

[0064] In S10, each character image can be understood as an image containing character information. The character information may include character categories and character positions. The character categories include at least 86 license plate character categories, including 10 Arabic numerals (0-9), 24 uppercase letters (AZ, excluding I and O) and 52 Chinese characters. New license plate character categories may also be added according to actual conditions, and are not limited to 86 categories. All license plate character categories are divided into basic category groups and novel category groups. The number of categories contained in the basic category group is greater than the number of categories contained in the novel category group. For example, the basic category group contains 66 categories, and the novel category group contains 20 categories. The basic category group includes multiple basic categories, each basic category includes multiple character images, forming a basic category data set. The novel category group includes multiple novel categories, each novel category includes multiple character images, forming a novel category data set. The character position includes the coordinates of the rectangular box representing the character position.

[0065] In S20, the feature extraction network can extract features from multiple character images in the basic category group. The feature extraction network includes but is not limited to using convolutional neural networks such as ResNet and VGG to extract image features. The target detection network can perform target category classification and target position regression on the image features extracted by the feature extraction network. The target detection network includes but is not limited to using a one-stage target detection network such as SSD, YOLO, or a two-stage target detection network such as Faster RCNN. In this step, the basic feature extraction network and the basic target detection network are both trained based on the image data of the basic category group.

[0066] In S30, the number of character images corresponding to each category in the balanced character data set is the same, which can be understood as the number of character images corresponding to each basic category in the basic category group is the same as the number of character images corresponding to each novel category in the novel category group, which can ensure that the amount of data for each category in the balanced character data set is the same. For example, the number of character images corresponding to the 86 license plate character categories in the balanced character data set is N. Since the number of character images of each basic category in the basic category group in S10 is greater than the number of character images of each novel category in the novel category group, a part of the character images can be selected from each basic category, and a part of the character images can be selected from each novel category, so that the number of each category is the same, forming a balanced character data set. The balanced character data set constructed in this step covers all character categories and reduces the number of samples for model training. The amount of data for each category in the balanced character data set is the same, which can form a balanced character data set, so that the proportion of special type license plate data and common type license plate data is balanced, and will not focus on a certain type of license plate. Therefore, the network model formed based on the training of the balanced character data set can be applied to any type of license plate, avoiding the problem of missed recognition or incorrect recognition of special types of license plate data with less data, thereby improving the recognition accuracy of the recognition method.

[0067] In S40, by initializing the parameters of the feature extraction network by balancing multiple character images in the novel category group in the character dataset, useful information can be learned during the training process to form an initialized feature extraction network. The initialized feature extraction network is derived from the novel category group in the balanced character dataset, and the basic feature extraction network is derived from the basic category group. By fusing the initialized feature extraction network with the basic feature extraction network, the character images in the basic category group and the character images in the novel category group can be fully utilized to obtain a full feature extraction network. The full feature extraction network can be applied to various different types of character images.

[0068] In S50, feature extraction is performed on an equal number of character images from each base category and a plurality of character images from each novel category using the full feature extraction network to balance the training data. Multiple balanced feature maps are then extracted from the balanced full-category dataset to obtain the parameters of the basic object detection network. The parameters of the basic object detection network are adjusted and updated based on the multiple balanced feature maps to obtain a full-category object detection network for all-category data. The full-category object detection network is trained and optimized based on the balanced character dataset and can be applied to character images of different categories.

[0069] Through the license plate recognition method provided by the present application, all character category data are divided into a basic category group and a novel category group, and a balanced character data set is constructed based on the basic category group and the novel category group, thereby realizing the learning of the basic stage and the fine-tuning stage in the network model training process, obtaining a balanced and stable full-feature extraction network and a full-target detection network, and realizing the detection and recognition of the license plate image to be tested. In addition, the license plate recognition method provided by the present application fine-tunes the basic target detection network based on the balanced character data set, so that the proportion of special type license plate data and common type license plate data is balanced, which can achieve better recognition effect for special type license plate data with small amount of data, avoid the problem of missed recognition or incorrect recognition of special type license plate data with small amount of data during the recognition process, and improve the detection and recognition accuracy.

[0070] In one embodiment, at S10, obtaining a basic category group and a novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position, the license plate recognition method further comprises:

[0071] S08, acquiring multiple license plate images, and obtaining multiple character images based on each license plate image;

[0072] S09 , screening the multiple character images according to the number of character categories corresponding to each character image to obtain a basic category group and a novel category group.

[0073] In this embodiment, a dataset is constructed based on multiple license plate images. Image frames containing motor vehicle and non-motor vehicle license plates are acquired from surveillance video. License plate detection is performed on the image frames using a license plate object detection algorithm to obtain location information of the license plate region within the image frames. Based on the location information of the license plate region, the image frames are captured and saved to obtain license plate images. Alternatively, the image frames are manually captured and saved to obtain license plate images. In each license plate image, a 2D object detection and annotation tool is used to annotate each character within the license plate image using a rectangular box, obtaining a corresponding annotation label for each license plate image, indicating the character category and position. The multiple license plate images include both motor vehicle and non-motor vehicle license plates. The multiple license plate images are derived from images of motor vehicle and non-motor vehicle license plates captured through real-world road traffic surveillance, encompassing images captured from different perspectives, backgrounds, and lighting intensities. The multiple license plate images include different types of license plates, such as cars, new energy vehicles, buses, motorcycles, electric vehicles, and large trucks. License plate object detection algorithms include, but are not limited to, two-stage or single-stage object detection methods such as Faster-RCNN, YOLO, and SSD.

[0074] One character image corresponds to one character category. The number of character images corresponding to each character category is statistically calculated to obtain the number corresponding to each character category. Based on the number corresponding to each character category, the total number of character images corresponding to all character categories is calculated, and then the proportion of the number of each character category to the total number of all character categories is obtained, thereby obtaining the number ratio of each character category. The number ratios of each character category are compared, and multiple character images are screened and sorted. One character image corresponds to one license plate character, one character category, and one character position. There is a one-to-one correspondence between character images, license plate characters, character categories, and character positions. The basic category group includes multiple basic categories, and each basic category includes multiple character images. The novel category group includes multiple novel categories, and each novel category includes multiple character images.

[0075] For example, the total number of character categories is 86. We sort the characters based on their proportion, assigning the 20 categories with the smallest proportion to the novel category group and the 66 categories with the largest proportion to the basic category group. Thus, the novel category group includes 20 character categories, and the basic category group includes 66 character categories. Each character category corresponds to multiple character images.

[0076] In one embodiment, multiple character images in the basic category group and the novel category group are preprocessed. Preprocessing includes image size unification and data enhancement. Image size unification can be understood as uniform processing of the sizes of multiple character images, for example, unifying the feature size of the character image to 64*256*3 (H*W*C, height*width*number of channels). Data enhancement includes data processing such as rotation transformation, brightness transformation and cropping of multiple character images. By performing rotation transformation on multiple character images, license plate images with different rotation angles can be obtained by rotation, which can simulate the tilted license plate data encountered in real shooting scenes. By performing brightness transformation on multiple character images, character images under different brightness conditions in the morning, noon and evening in real scenes can be simulated. By performing random cropping on multiple character images, character image data under occlusion conditions encountered in real scenes can be simulated.

[0077] In one embodiment, S10, a basic category group and a novel category group are obtained, the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, each character image includes a real character category and a real character position, and the ratio of the number of categories in the basic category group to the number of categories in the novel category group is in a range of 3:1 to 4:1.

[0078] In this embodiment, the ratio of the number of categories in the basic category group to the number of categories in the novel category group is set to a range of 3:1 to 4:1, so that most of the character types in all categories that can be collected are used for training in the basic learning stage, and a few character types are used to fine-tune the model formed by the full feature extraction network and the full target detection network in the learning stage to obtain a more stable and accurate recognition model.

[0079] In one embodiment, S20, training the feature extraction network and the object detection network according to the basic category group to obtain a basic feature extraction network and a basic object detection network includes:

[0080] S210, inputting the character images of the basic category group into a feature extraction network, and outputting a plurality of basic feature maps;

[0081] S220, inputting the multiple basic feature maps into the object detection network, and outputting the predicted basic character category and predicted basic character position of each basic character image;

[0082] S230, constructing a basic loss function according to the predicted basic character category, the predicted basic character position, the actual character category, and the actual character position, training the feature extraction network and the target detection network to obtain a basic feature extraction network and a basic target detection network.

[0083] In this embodiment, the feature extraction network includes but is not limited to using convolutional neural networks such as ResNet and VGG to extract image features of multiple basic character images and obtain multiple basic feature maps. The target detection network includes but is not limited to two-stage or single-stage target detection networks such as Faster-RCNN, YOLO, and SSD. The target detection network includes multiple convolutional layers, normalization layers, and nonlinear activation layers. The normalization layer includes but is not limited to normalization operations such as instance normalization layer and adaptive instance normalization layer. The nonlinear activation layer includes but is not limited to nonlinear activation functions such as ReLu and Leaky ReLu. The target detection network is used to detect character categories and character positions of multiple basic feature maps. According to the predicted basic character category, predicted basic character position, true character category, and true character position output by the target detection network, a basic loss function is constructed, and the network parameters of the feature extraction network and the target detection network are adjusted and optimized to obtain a basic feature extraction network and a basic target detection network trained by multiple basic character images.

[0084] In one embodiment, the object detection network can adopt a one-stage YOLO-based object detection network structure, with the output being (H, W, C+5). (H, W) represents the size of the output feature map, C represents the output classification result, and 5 represents the prediction result of position regression. (x, y, w, h, confidence) represents the prediction result of position regression. x and y represent the center point coordinates of the predicted box, w and h represent the width and height of the predicted box, and confidence represents the confidence level of the predicted box.

[0085] The basic loss function includes a basic classification loss function and a basic position regression loss function. The basic classification loss function is constructed based on the predicted basic character category and the actual character category. The basic classification loss function includes but is not limited to using the Cross Entropy loss function, the Focal Loss loss function, etc. In this embodiment, the basic classification loss function uses the cross entropy classification loss function to train the network model. The cross entropy classification loss function is as follows:

[0086]

[0087] A basic position regression loss function is constructed based on the predicted basic character positions and the actual character positions. Basic position regression loss functions include, but are not limited to, the L1 Loss mean absolute error loss function, the L2 Loss mean square error loss function, the IoU Loss intersection over union loss function, and the like. In this embodiment, the basic position regression loss function uses the DIoU Loss position regression loss function to regress the license plate character positions. The DIoU Loss position regression loss function is shown below:

[0088]

[0089] Among them, b pred represents the center coordinates of the predicted box of the predicted basic character position, bgt represents the center coordinates of the annotated box of the actual character position, ρ represents the Euclidean distance between the center points of the predicted box of the predicted basic character position and the center points of the annotated box of the actual character position, and c represents the diagonal distance of the minimum bounding rectangle that can contain both the predicted box and the actual annotated box. IoU represents the Intersection over Union (IoU) ratio between the predicted box and the actual annotated box, P represents the predicted box, G represents the actual annotated box, and IoU is expressed as:

[0090] That is, the ratio of the intersection and union between the predicted box and the true annotation box

[0091] By training the basic feature extraction network and the basic target detection network formed by the basic training set formed by multiple character images of the basic category group in this embodiment, the corresponding parameters of the basic feature extraction network and the basic target detection network can be obtained, completing the basic learning stage of the license plate character type of the basic category group.

[0092] In one embodiment, the feature extraction network uses the backbone network of ResNet18 to extract features from the character images of the basic category group. A one-stage YOLO-based target detection network structure is used to detect the character category and character position of multiple basic feature maps that have passed through the ResNet18 backbone network. When using the backbone network of ResNet18 to extract features from license plate data, a feature fusion method can be added to fuse the features located between different layers of the network, so as to achieve fusion between high- and low-level features, thereby improving detection accuracy. Feature fusion methods include but are not limited to the use of FPN (Feature Pyramid Network), ASPP (Atrous Spatial Pooling Pyramid), etc. The accuracy and running speed of the network model are improved by combining the ResNet18 backbone network, the feature fusion method, and the one-stage YOLO-based target detection network structure.

[0093] In one embodiment, S30, a balanced character data set is constructed based on the basic category group and the novel category group, and the number of character images of each basic category in the balanced character data set is the same as the number of character images of each novel category, and N targets are extracted from the basic category group and the novel category group as the full category data set. N character images are extracted from each basic category in the basic category group, and N character images are extracted from each novel category in the novel category group, so that the amount of data for each category in the balanced character data set is the same. N is a positive integer and can be set to 10, 20, etc. The amount of data for each category in the balanced character data set is the same, and a balanced character data set can be formed, so that the proportion of special type license plate data and common type license plate data is balanced, and no emphasis is placed on a certain type of license plate. Therefore, the network model formed by training based on the balanced character data set can be applied to any type of license plate, avoiding the problem of missed recognition or incorrect recognition of special type license plate data with less data, thereby improving the recognition accuracy of the recognition method.

[0094] In one embodiment, S40, the parameters of the feature extraction network are initialized according to the novel category group in the balanced character data set to obtain an initialized feature extraction network, and the initialized feature extraction network is fused with the basic feature extraction network to obtain a full feature extraction network. The initialized feature extraction network and the basic feature extraction network are spliced ​​by the string connection function (that is, the concat() function) to achieve fusion and obtain the full feature extraction network.

[0095] In one embodiment, S50, feature extraction is performed on the balanced character dataset using the full feature extraction network to obtain a plurality of balanced feature maps, and parameters of the basic object detection network are updated based on the plurality of balanced feature maps to obtain the full object detection network, including:

[0096] S510, inputting the balanced character data set into the full feature extraction network and outputting a plurality of balanced feature maps;

[0097] S520, inputting the multiple balanced feature maps into a basic object detection network, and outputting a predicted full-character category and a predicted full-character position for each character image;

[0098] S530, constructing a full loss function based on the predicted full character categories, predicted full character positions, true character categories, and true character positions, updating the parameters of the basic object detection network, and obtaining a full object detection network.

[0099] In this embodiment, the full feature extraction network combines data from the novel category group with data from the basic category group, enabling feature extraction for any type of license plate data, broadening its applicability. Based on the trained full feature extraction network, feature extraction is performed on individual character images in a balanced character dataset, yielding multiple balanced feature maps. These multiple balanced feature maps reflect the balanced nature of the balanced character dataset. Furthermore, these multiple balanced feature maps are used to train, optimize, and update the basic object detection network, making it more applicable to a variety of license plate data types.

[0100] The license plate recognition method provided by the present application first fixes the parameters of the basic feature extraction network and initializes the network parameters according to the novel category group data in the balanced character data set to form an initialized feature extraction network. The basic feature extraction network and the initialized feature extraction network are subjected to feature fusion to obtain a full feature extraction network based on the full category data set. Based on the full feature extraction network and the balanced character data set, the parameters of the basic target detection network are updated and optimized to generate a full target detection network. Thus, both the full feature extraction network and the full target detection network are network model training based on full category data, and the model formed by the full feature extraction network and the full target detection network is fine-tuned to obtain a more stable and accurate recognition model.

[0101] In one embodiment, S530 constructs a full loss function based on the predicted full character category, the predicted full character position, the true character category, and the true character position, and updates the parameters of the basic object detection network to obtain the full object detection network, including:

[0102] S531, constructing a full classification loss function based on the predicted full character categories and the true character categories;

[0103] S532: Construct a full position regression loss function based on the predicted full character position and the actual character position. The full position regression loss function is:

[0104]

[0105] Among them, bPre represents the center point coordinates of the predicted full character position, b gt represents the center point coordinates of the true character position, ρ represents the Euclidean distance between the center point coordinates of the predicted full character position and the center point coordinates of the true character position, c represents the diagonal distance of the minimum circumscribed rectangular box containing the predicted full character position and the true character position, and IoU represents the IoU intersection-over-union ratio between the predicted full character position and the true character position.

[0106] In this embodiment, the full loss function includes a full classification loss function and a full position regression loss function. The full classification loss function is constructed based on the predicted full character categories and the actual character categories. The full classification loss function includes but is not limited to the use of Cross Entropy loss function, Focal Loss loss function, etc. In this embodiment, the basic classification loss function uses the cross entropy classification loss function to train the network model. The full position regression loss function is constructed based on the predicted full character positions and the actual character positions. The full position regression loss function includes but is not limited to the use of L1Loss mean absolute error loss function, L2 Loss mean square error loss function, IoU Loss intersection over union loss function, etc. In this embodiment, the full position regression loss function uses DIoU Loss position regression loss function to regress the license plate character position. The DIoU Loss position regression loss function is as follows:

[0107]

[0108] Among them, bPre represents the center coordinate of the prediction box of the predicted full character position, b gt represents the coordinates of the center point of the annotation box of the actual character position, ρ represents the Euclidean distance between the center point of the predicted box of the predicted full character position and the center point of the annotation box of the actual character position, and c represents the diagonal distance of the minimum bounding rectangle that can simultaneously contain the predicted box and the actual annotation box. IoU represents the Intersection over Union (IoU) between the predicted box and the actual annotation box, P represents the predicted box, G represents the actual annotation box, and IoU is expressed as:

[0109] That is, the ratio of the intersection and union between the predicted box and the true annotation box

[0110] By balancing the character data set, a balanced full-category data set of license plate character data is formed, which can be used to fine-tune the basic target detection network and realize the stage of learning all categories of license plate character data.

[0111] In one embodiment, S60, after detecting and recognizing the license plate image to be tested and obtaining the license plate character sequence based on the full feature extraction network and the full target detection network, the method further includes:

[0112] The non-maximum suppression algorithm is used to remove some redundant detection frames of the license plate image after it passes through the full feature extraction network and the full target detection network, and the character category and character detection frame corresponding to the license plate image are obtained;

[0113] If the number of character detection frames is greater than 8, they are sorted from large to small according to their confidence, and 8 character detection frames are output;

[0114] According to the positions of the 8 character detection frames, output the license plate character sequence in order from left to right;

[0115] Alternatively, the license plate character sequence is output according to the positions of the 8 character detection frames in the order of top to bottom and from left to right.

[0116] See Figure 2 In one embodiment, the present application provides a license plate recognition device 100. The license plate recognition device 100 includes a data acquisition module 10, a basic network generation module 20, a balanced character dataset acquisition module 30, a full feature extraction network generation module 40, a full target detection network generation module 50, and a recognition module 60. The data acquisition module 10 is used to obtain a basic category group and a novel category group, the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position. The basic network generation module 20 is used to train the feature extraction network and the target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network. The balanced character dataset acquisition module 30 is used to construct a balanced character dataset based on the basic category group and the novel category group, and the number of character images of each basic category in the balanced character dataset is the same as the number of character images of each novel category.

[0117] The full feature extraction network generation module 40 is used to initialize the parameters of the feature extraction network based on the novel category group in the balanced character dataset to obtain an initialized feature extraction network, and then fuse the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network. The full target detection network generation module 50 is used to extract features from the balanced character dataset based on the full feature extraction network to obtain multiple balanced feature maps, and update the parameters of the basic target detection network based on the multiple balanced feature maps to obtain a full target detection network. The recognition module 60 is used to detect and recognize the license plate image to be tested based on the full feature extraction network and the full target detection network to obtain the license plate character sequence.

[0118] In this embodiment, the description of the data acquisition module 10 can be referred to S10 in the above embodiment. The description of the basic network generation module 20 can be referred to S20 in the above embodiment. The description of the balanced character dataset acquisition module 30 can be referred to S30 in the above embodiment. The description of the full feature extraction network generation module 40 can be referred to S40 in the above embodiment. The description of the full object detection network generation module 50 can be referred to S50 in the above embodiment. The description of the recognition module 60 can be referred to S60 in the above embodiment.

[0119] In one embodiment, the license plate recognition device 100 further includes a character image acquisition module (not labeled in the figure) and a grouping module (not labeled in the figure). The character image acquisition module is configured to acquire multiple license plate images and, based on each license plate image, obtain multiple character images. The grouping module is configured to filter the multiple character images based on the number of character categories corresponding to each character image to obtain basic category groups and novel category groups.

[0120] In this embodiment, the description of the character image acquisition module can refer to the description of S08 in the above embodiment. The description of the grouping module can refer to the description of S09 in the above embodiment.

[0121] In one embodiment, the basic network generation module 20 includes a basic feature map acquisition module (not marked in the figure), a predicted basic character information acquisition module (not marked in the figure) and a basic network model training module (not marked in the figure). The basic feature map acquisition module is used to input the character image of the basic category group into the feature extraction network and output a plurality of basic feature maps. The predicted basic character information acquisition module is used to input a plurality of basic feature maps into the target detection network and output the predicted basic character category and predicted basic character position of each basic character image. The basic network model training module is used to construct a basic loss function based on the predicted basic character category, the predicted basic character position, the real character category and the real character position, train the feature extraction network and the target detection network, and obtain a basic feature extraction network and a basic target detection network.

[0122] In this embodiment, the description of the basic feature map acquisition module can refer to the description of S210 in the above embodiment. The description of the predicted basic character information acquisition module can refer to the description of S220 in the above embodiment. The description of the basic network model training module can refer to the description of S230 in the above embodiment.

[0123] In one embodiment, the full target detection network generation module 50 includes a balanced feature map acquisition module (not marked in the figure), a predicted full character information acquisition module (not marked in the figure) and a full target detection network training module (not marked in the figure). The balanced feature map acquisition module is used to input the balanced character data set into the full feature extraction network and output a plurality of balanced feature maps. The predicted full character information acquisition module is used to input a plurality of balanced feature maps into the basic target detection network and output the predicted full character category and predicted full character position of each character image. The full target detection network training module is used to construct a full loss function based on the predicted full character category, predicted full character position, true character category and true character position, update the parameters of the basic target detection network, and obtain the full target detection network.

[0124] In this embodiment, the description of the balanced feature map acquisition module can refer to the description of S510 in the above embodiment. The description of the predicted full character information acquisition module can refer to the description of S520 in the above embodiment. The description of the full object detection network training module can refer to the description of S530 in the above embodiment.

[0125] In one embodiment, the full object detection network training module includes a full classification loss function construction module and a full position regression loss function construction module. The full classification loss function construction module is used to construct a full classification loss function based on the predicted full character categories and the actual character categories. The full position regression loss function construction module is used to construct a full position regression loss function based on the predicted full character positions and the actual character positions. The full position regression loss function is:

[0126]

[0127] Among them, b pred Indicates the center coordinates of the predicted full character position, b gt represents the center point coordinates of the real character position, ρ represents the Euclidean distance between the center point coordinates of the predicted full character position and the center point coordinates of the real character position, c represents the diagonal distance of the minimum circumscribed rectangular box containing the predicted full character position and the real character position, and IoU represents the IoU intersection-over-union ratio between the predicted full character position and the real character position.

[0128] In this embodiment, the description of the full classification loss function construction module can refer to the description of S531 in the above embodiment. The description of the full position regression loss function construction module can refer to the description of S532 in the above embodiment.

[0129] In one embodiment, in the data acquisition module 10 , the ratio of the number of categories in the basic category group to the number of categories in the novel category group ranges from 3:1 to 4:1.

[0130] In this embodiment, for the description of the ratio range of the number of categories in the basic category group to the number of categories in the novel category group, reference may be made to the description in the above embodiment.

[0131] In the various embodiments described above, the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of this disclosure. The accompanying method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0132] Those skilled in the art will also appreciate that the various illustrative logical blocks, modules, and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, modules, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0133] The various illustrative logic blocks described in the embodiments of the present application, or modules can be implemented or operated by the design of a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination thereof. The general-purpose processor can be a microprocessor, alternatively, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration to implement.

[0134] The steps of the methods or algorithms described in the embodiments of the present application can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be provided in an ASIC, which can be provided in a user terminal. Alternatively, the processor and the storage medium can also be provided in different components in the user terminal.

[0135] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A license plate recognition method, characterized in that: include: Obtaining a basic category group and a novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position; Training the feature extraction network and the target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network; constructing a balanced character dataset based on the basic category group and the novel category group, wherein the number of character images of each basic category in the balanced character dataset is the same as the number of character images of each novel category; Initializing parameters of the feature extraction network according to the novel category group in the balanced character dataset to obtain an initialized feature extraction network, and fusing the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network; Performing feature extraction on the balanced character dataset according to the full feature extraction network to obtain a plurality of balanced feature maps, and updating parameters of the basic object detection network according to the plurality of balanced feature maps to obtain a full object detection network; According to the full feature extraction network and the full target detection network, the license plate image to be tested is detected and recognized to obtain a license plate character sequence.

2. The license plate recognition method according to claim 1, characterized in that: Before obtaining the basic category group and the novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position, the method further comprises: Acquire multiple license plate images, and obtain multiple character images based on each license plate image; The plurality of character images are screened according to the number of character categories corresponding to each character image to obtain the basic category group and the novel category group.

3. The license plate recognition method according to claim 1, characterized in that: The step of training the feature extraction network and the target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network includes: Inputting the character images of the basic category group into the feature extraction network and outputting a plurality of basic feature maps; Inputting the plurality of basic feature maps into the object detection network, and outputting a predicted basic character category and a predicted basic character position of each basic character image; A basic loss function is constructed according to the predicted basic character category, the predicted basic character position, the real character category and the real character position, and the feature extraction network and the target detection network are trained to obtain the basic feature extraction network and the basic target detection network.

4. The license plate recognition method according to claim 1, characterized in that: The method further comprises: extracting features from the balanced character dataset according to the full feature extraction network to obtain a plurality of balanced feature maps; and updating parameters of the basic object detection network according to the plurality of balanced feature maps to obtain a full object detection network, including: Inputting the balanced character data set into the full feature extraction network and outputting the plurality of balanced feature maps; Inputting the multiple balanced feature maps into the basic object detection network, and outputting the predicted full character category and predicted full character position of each character image; A full loss function is constructed according to the predicted full character category, the predicted full character position, the real character category and the real character position, and the parameters of the basic object detection network are updated to obtain the full object detection network.

5. The license plate recognition method according to claim 4, characterized in that: The method constructs a full loss function according to the predicted full character category, the predicted full character position, the true character category, and the true character position, updates the parameters of the basic object detection network, and obtains the full object detection network, including: Constructing a full classification loss function based on the predicted full character categories and the true character categories; A full position regression loss function is constructed based on the predicted full character position and the true character position. The full position regression loss function is: Among them, b pred Indicates the center coordinates of the predicted full character position, b gt represents the center point coordinates of the real character position, ρ represents the Euclidean distance between the center point coordinates of the predicted full character position and the center point coordinates of the real character position, c represents the diagonal distance of the minimum circumscribed rectangular box containing the predicted full character position and the real character position, and IoU represents the IoU intersection-over-union ratio between the predicted full character position and the real character position.

6. The license plate recognition method according to claim 1, characterized in that: The basic category group and the novel category group are obtained, the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, each of the character images includes a real character category and a real character position, and the ratio of the number of categories in the basic category group to the number of categories in the novel category group is in a range of 3:1 to 4:

1.

7. A license plate recognition device, characterized in that: include: a data acquisition module, configured to acquire a basic category group and a novel category group, wherein the number of categories in the basic category group is greater than the number of categories in the novel category group, and the number of character images of each basic category in the basic category group is greater than the number of character images of each novel category in the novel category group, and each character image includes a real character category and a real character position; A basic network generation module, configured to train a feature extraction network and a target detection network according to the basic category group to obtain a basic feature extraction network and a basic target detection network; a balanced character data set acquisition module, configured to construct a balanced character data set based on the basic category group and the novel category group, wherein the number of character images of each basic category in the balanced character data set is the same as the number of character images of each novel category; a full feature extraction network generation module, configured to initialize parameters of the feature extraction network according to the novel category group in the balanced character dataset to obtain an initialized feature extraction network, and fuse the initialized feature extraction network with the basic feature extraction network to obtain a full feature extraction network; a full object detection network generation module, configured to perform feature extraction on the balanced character dataset according to the full feature extraction network to obtain a plurality of balanced feature maps, and update the parameters of the basic object detection network according to the plurality of balanced feature maps to obtain a full object detection network; The recognition module is used to detect and recognize the license plate image to be tested based on the full feature extraction network and the full target detection network to obtain the license plate character sequence.

8. The license plate recognition device according to claim 7, characterized in that: The device further comprises: A character image acquisition module is used to acquire multiple license plate images and obtain multiple character images based on each license plate image; The grouping module is configured to screen the plurality of character images according to the number of character categories corresponding to each character image, to obtain the basic category group and the novel category group.

9. The license plate recognition device according to claim 7, characterized in that: The basic network generation module includes: A basic feature map acquisition module, configured to input the character images of the basic category group into the feature extraction network and output a plurality of basic feature maps; A predicted basic character information acquisition module, configured to input the plurality of basic feature maps into the object detection network and output a predicted basic character category and a predicted basic character position of each basic character image; The basic network model training module is used to construct a basic loss function based on the predicted basic character category, the predicted basic character position, the real character category and the real character position, train the feature extraction network and the target detection network, and obtain the basic feature extraction network and the basic target detection network.

10. The license plate recognition device according to claim 7, characterized in that: The full target detection network generation module includes: A balanced feature map acquisition module, configured to input the balanced character dataset into the full feature extraction network and output the plurality of balanced feature maps; A predicted full-character information acquisition module, configured to input the plurality of balanced feature maps into the basic object detection network and output a predicted full-character category and a predicted full-character position for each character image; The full-target detection network training module is used to construct a full loss function based on the predicted full-character category, the predicted full-character position, the real character category and the real character position, update the parameters of the basic target detection network, and obtain the full-target detection network.

11. The license plate recognition device according to claim 10, characterized in that: The full target detection network training module includes: A full classification loss function construction module, configured to construct a full classification loss function based on the predicted full character categories and the true character categories; The full position regression loss function construction module is used to construct a full position regression loss function based on the predicted full character position and the true character position. The full position regression loss function is: Among them, b pred Indicates the center coordinates of the predicted full character position, b gt represents the center point coordinates of the real character position, ρ represents the Euclidean distance between the center point coordinates of the predicted full character position and the center point coordinates of the real character position, c represents the diagonal distance of the minimum circumscribed rectangular box containing the predicted full character position and the real character position, and IoU represents the IoU intersection-over-union ratio between the predicted full character position and the real character position.

12. The license plate recognition device according to claim 7, characterized in that: In the data acquisition module, the ratio of the number of categories in the basic category group to the number of categories in the novel category group ranges from 3:1 to 4:1.

Images