Single-stage multi-type multi-license-plate license plate detection and recognition integrated method

Through the single-stage multi-type multi-licence plate detection and recognition method of deep learning, the parallel operation of license plate detection and character recognition is achieved using feature extraction and attention modules, solving the problem of low license plate recognition efficiency in open scenarios, and realizing multiple single-time recognition Uniform identification of license plates and multiple types of license plates.

CN118982822AActive Publication Date: 2024-11-19PEKING UNIV
View PDF 13 Cites 0 Cited by

Patent Information

Application Number
CN202411021562.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-19
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

The existing automatic license plate recognition method is difficult to identify in open scenarios. The traditional method is cascading, resulting in low operational efficiency, unable to process license plate detection and recognition tasks in parallel, and unable to identify multiple license plates at the same time.

Method used

Using a single-stage multi-type multi-license plate detection and recognition method based on deep learning, the parallel operation of license plate detection and character recognition is realized through feature extraction backbone network and attention module, and a single network structure is used to identify multiple types of license plates, and through detection of branches and recognition The fusion of the branch outputs the license plate position and character sequence.

Benefits of technology

It improves network efficiency, realizes detection and identification of multiple license plates in a single recognition, reduces the complexity of network design, supports unified identification of multiple types of license plates, and avoids multiple recognition and cropping operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982822A_ABST
    Figure CN118982822A_ABST
Patent Text Reader

Abstract

The invention discloses a single-stage multi-type multi-license-plate license plate detection and recognition integrated method. The method comprises the following steps: firstly, performing feature extraction on a to-be-processed image possibly containing multiple license plates through a feature extraction backbone network to obtain a feature map; sending the feature map into an attention module to obtain an attention map of the license plate and the character sequence; sending the feature map into a detection branch to predict the position of the license plate in the image, sending the feature map and the attention map into an identification branch to predict character categories, and carrying out parallel operation on the detection branch and the identification branch; and finally, output results of the detection branch and the identification branch are fused to obtain the position and the character sequence of each license plate in the image. According to the method, multiple networks do not need to be cascaded, license plate detection and character recognition tasks are achieved in a single network in parallel, multiple license plates can be recognized at a time, and the network efficiency is improved; meanwhile, unified recognition of multiple types of license plates is realized under a single network structure through an attention mechanism, and the network design complexity is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technologies such as automatic license plate recognition, target detection, and deep learning, and specifically to a single-stage, multi-type, and multi-license plate integrated license plate detection and recognition method. Background Art

[0002] Automatic license plate recognition is an important part of the intelligent transportation system and is widely used in life. Existing license plate recognition methods have mature applications and good performance in fixed scenarios, such as fixed lighting, fixed angles, fixed distances, etc. However, real-world scenarios sometimes cannot provide a stable application environment, especially in open environments such as the field of autonomous driving. License plates in open scenes have different lighting conditions, different angles, different clarity, etc., making it difficult for traditional methods to recognize in open scenes. Traditional automatic license plate recognition methods detect and recognize by manually extracting image features. With the rapid development of deep learning, deep neural networks have become the main method for automatic license plate recognition, and have better performance than traditional methods.

[0003] The goal of automatic license plate recognition is to detect the license plate position in the image and identify the corresponding license plate characters. Therefore, the automatic license plate recognition task can be divided into a license plate detection task and a license plate character recognition task. The existing method adopts a two-stage cascade method of first detecting the license plate position and then cropping the license plate image or features for license plate character recognition, such as CN117649644A, CN114782770A, CN114581904A, and CN108509954A. Due to the cascade method, the license plate detection task and the license plate character recognition task cannot be performed in parallel, which reduces the operating efficiency. At the same time, the recognition by cropping the license plate results in that only a single license plate can be recognized each time. Therefore, if there are multiple license plates in the image, the existing method needs to perform multiple recognitions according to the number of license plates, which increases the time required for operation.

[0004] License plates have many types, including different colors, different numbers of characters, and different numbers of character rows. For the recognition of different types of license plates, a method of type prediction followed by processing can be used, such as CN114581897A and CN115424250A. This type of method uses different recognition networks for different types of license plates, resulting in a complex structure. The existing method of identifying different types of license plates under a unified architecture uses a method of dividing the license plate character sequence into individual characters for individual recognition and then splicing them. CN113554030A uses a soft attention method to divide the character sequence into individual characters and then classify the characters. CN112016556A uses an attention mechanism to fuse semantic information and position information, obtain character position and type, and then fuse them. Existing multi-type license plate recognition methods can only recognize a single license plate, and cannot recognize multiple license plates at the same time. Summary of the invention

[0005] The purpose of the present invention is to provide a single-stage, multi-type, and multi-license plate integrated license plate detection and recognition method based on deep learning, which does not require cascading multiple networks and implements license plate detection and character recognition tasks in parallel within a single network. In a multi-license plate scenario, multiple license plates can be recognized at a time to improve network efficiency. At the same time, the attention mechanism is used to achieve unified recognition of multiple types of license plates under a single network structure, reducing the complexity of network design.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A single-stage multi-type and multi-license plate detection and recognition integrated method based on deep learning, comprising:

[0008] The feature extraction backbone network is used to extract features from the image to be processed that may contain multiple license plates, and the corresponding feature map is obtained;

[0009] The extracted feature map is sent to the attention module to obtain the attention map of the license plate and character sequence;

[0010] The extracted feature map is sent to the detection branch to predict the position of the license plate in the image, and the extracted feature map and attention map are sent to the recognition branch to predict the character category. The detection branch and the recognition branch perform parallel operations;

[0011] The output results of the detection branch and the recognition branch are fused to obtain the position and character sequence of each license plate in the image.

[0012] The single stage means that the method of the present invention uses a single network to achieve license plate detection and character recognition tasks without cascading multiple networks.

[0013] The multiple types mentioned refer to that the method of the present invention can recognize multiple license plate types, including different license plate colors, different character lengths and different numbers of character rows.

[0014] The multiple license plates mentioned above mean that the method of the present invention can simultaneously detect and identify multiple license plates in an image.

[0015] The integration of detection and recognition means that no correction and cropping methods are used in the middle of the network of the method of the present invention, and the license plate detection and character recognition tasks can be calculated in parallel.

[0016] In the above-mentioned license plate detection and recognition integrated method, the feature extraction backbone network uses a multi-layer residual network ResBlock (residual block) to stack and perform one or more downsampling to output a feature map F b .

[0017] In the above-mentioned license plate detection and recognition integrated method, the attention module performs up- and down-sampling and jump connection based on the UNet structure. In some embodiments of the present invention, the attention module performs up- and down-sampling and jump connection on the input feature map F b Use convolution to perform multiple downsampling, use interpolation to perform multiple upsampling, and use convolution to fuse the stacked information; finally, use the classification head to classify each point on the feature map into a character or a license plate, and generate multiple character attention maps and a license plate attention map. The number of character attention maps depends on the maximum number of characters contained in a license plate. The output of the attention module is H a and W a Respectively represent the height and width of the attention map size, M lp represents the license plate attention map, The attention map for each character index. The character attention map for each index contains the attention hotspots of the characters at the corresponding index position of each license plate in the image. The license plate attention map contains the license plate attention hotspots of each license plate in the image.

[0018] In the above license plate detection and recognition integrated method, the detection branch predicts the position information of each license plate in the image. The recognition branch predicts the character of each index of each license plate in the image by combining the extracted feature map and the generated attention map. The character recognition network shares parameters for each index of the character sequence.

[0019] In some embodiments of the present invention, the recognition branch divides the image into H r *W r Identification grid (H r and W r is a positive integer), when the license plate image falls within a certain recognition grid, the recognition grid is responsible for recognizing the characters of the license plate. The detection branch divides the image into H d *W d A detection grid (H d and W d is a positive integer), and the anchor point is used as the coordinate reference point. The anchor point is the center point of the detection grid. The output of the detection branch is H d *W d *(l, t, r, b, c), l, t, r, b are the distances from the anchor point to the left, top, right, and bottom borders of the bounding box of the target to be detected; c is the detection confidence, and its label is the IoU value between the predicted bounding box and the true bounding box. The recognition branch combines the feature map and the attention map by point multiplication to predict the character category. The feature representation of the input recognition branch is That is, the sum of the characteristics of the character and the characteristics of part of the license plate.

[0020] Preferably, the detection network of the detection branch uses multiple layers of SE ResBlock to be stacked, and downsampling is performed once or multiple times; the recognition network of the recognition branch uses multiple layers of SE ResBlock to be stacked, and downsampling is performed once or multiple times.

[0021] In the above-mentioned integrated license plate detection and recognition method, the output results of the detection branch and the recognition branch are fused in the following way: first, the character results predicted by the recognition branch are spliced ​​in the order of their indexes, and then the spliced ​​character sequence is assigned to the position predicted by the detection branch, and finally, redundant results are removed according to the confidence of detection and recognition.

[0022] The integrated license plate detection and recognition method of the present invention uses back propagation to adjust and learn network parameters through a loss function when training the network. The loss function includes three loss functions: attention, detection and recognition.

[0023] It can be seen from the technical solution provided by the present invention that, on the one hand, recognition does not rely on the detected position information for cropping, and the detection and recognition tasks can be performed in parallel; on the other hand, through the attention mechanism, the character sequences of different types of license plates are divided into separate characters according to the index for recognition, and a single structure is used to achieve unified recognition of different types of license plates; finally, both the detection and recognition branches support multi-license plate scenarios, and there is no need to perform multiple recognitions based on the number of license plates, and all prediction results can be obtained in one run. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 The figure is a flow chart of the integrated method for license plate detection and recognition of the present invention.

[0026] Figure 2 A schematic diagram of a network structure provided by an embodiment of the present invention.

[0027] Figure 3 Examples of attention maps for different types of license plates provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] This embodiment provides a single-stage multi-type and multi-license plate integrated license plate detection and recognition method based on deep learning, which can detect and recognize multiple license plates and multiple types of license plates in one image using a single network. Figure 1 The main components and working process of the method are provided. Figure 2 The network structure of this embodiment is provided, and its main process is as follows:

[0030] 1. Perform feature extraction on the image to be processed through the feature extraction backbone network to obtain the corresponding feature map.

[0031] The feature extraction backbone network uses a 7-layer residual network ResBlock for stacking and performs 2 times downsampling. The output feature map of the feature extraction backbone network is F b , size is H f *W f *C f , which are the height, width and number of channels of the feature map respectively.

[0032] 2. Obtain eight character attention maps and one license plate attention map through the attention module combined with the feature map.

[0033] The attention module is based on the UNet structure for up- and down-sampling and skip connection. Figure 2 As shown, in this embodiment, the input feature map F b Convolution is used to perform three downsamplings, interpolation is used to perform three upsamplings, and convolution is used to fuse the stacked information. Finally, the classification head is used to classify each point on the feature map into a character or a license plate, generating eight character attention maps and one license plate attention map. The output of the attention module is H a and W a Respectively represent the height and width of the attention map size, M lp The attention map representing the license plate, The attention map for each character index. Figure 3 As shown, contains the attention hotspots of the ith character of all license plates in the image; M lp The above contains the attention hotspots of all license plates in the image. The generation of character attention is only related to the index of the character, which unifies the processing logic of different types of license plates.

[0034] 3. License plate recognition is performed by combining the feature map and attention map through the recognition branch.

[0035] The recognition branch combines the extracted feature map and the generated attention map to predict the character category. r *W r Identification grid (H r and W r is a positive integer), when the license plate image falls within a certain recognition grid, the recognition grid is responsible for recognizing the characters of the license plate. The feature map and the attention map are combined by point multiplication. This embodiment combines single character attention and license plate attention to better recognize characters. Figure 2 As shown, the features of the input recognition branch can be expressed as That is, the sum of the characteristics of the character and the characteristics of the license plate. Figure 2 As shown in the figure, the recognition network of the recognition branch is stacked with 4 layers of SE ResBlock and downsampled 4 times. At the same time, the recognition network of each character index shares parameters.

[0036] 4. Perform license plate detection by combining the detection branch with the feature map.

[0037] The detection branch combines the feature map to predict the location information of the license plate in the image. d *W d A detection grid (H d and W d is a positive integer), and the anchor point is used as the coordinate reference point. The anchor point is the center point of the detection grid. The output of the detection branch is H d *W d *(l, t, r, b, c), l, t, r, b are the distances from the anchor point to the left, top, right, and bottom borders of the bounding box of the target to be detected; c is the detection confidence, and its label is the IoU value between the predicted bounding box and the real bounding box. Figure 2 As shown in Figure 1, the detection network of the detection branch is stacked with 6 layers of SE ResBlock and downsampled twice.

[0038] 5. Integrate recognition and detection information through fusion modules.

[0039] The fusion module fuses the character category information of the recognition branch and the license plate position information of the detection branch. This embodiment uses the fusion module to match information according to the corresponding position. When the detection anchor point falls within a certain recognition grid, the recognition result of the recognition grid is assigned to these detection anchor points. The confidence score of each license plate is the integration of the detection confidence and the recognition confidence: c is the detection confidence, is the recognition probability of each character of the license plate. Finally, non-maximum suppression is performed on the output to remove redundant results.

[0040] 6. Adjust and learn the network parameters through the loss function.

[0041] The loss function includes three loss functions: attention, detection and recognition. The attention loss function of this embodiment is a weighted binary cross entropy loss; the detection loss function consists of positioning loss and confidence loss; the recognition loss function is a classification cross entropy loss. The network parameters are adjusted and learned through back propagation.

[0042] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A deep learning-based license plate detection and recognition integrated method uses a single network to simultaneously detect and recognize multiple license plates of multiple types in an image, including the following steps: 1) Extract features from the image that may contain multiple license plates through the feature extraction backbone network to obtain the corresponding feature map; 2) Send the feature map extracted in step 1) to the attention module to obtain the attention map of the license plate and character sequence; 3) The feature map extracted in step 1) is sent to the detection branch to predict the position of the license plate in the image, and the feature map extracted in step 1) and the attention map obtained in step 2) are sent to the recognition branch to predict the character category. The detection branch and the recognition branch perform parallel operations; 4) The output results of the detection branch and the recognition branch are fused to obtain the position and character sequence of each license plate in the image.

2. The integrated license plate detection and recognition method according to claim 1, characterized in that: In step 1), the feature extraction backbone network is stacked with a multi-layer residual network ResBlock and downsampled once or multiple times, and the output size is H f *W f *C f The feature map F b , where H f , W f , C f are the height, width and number of channels of the feature map respectively.

3. The integrated license plate detection and recognition method according to claim 1, characterized in that: In step 2), the attention module performs up- and down-sampling and jump connections based on the UNet structure.

4. The integrated license plate detection and recognition method according to claim 3, characterized in that: In step 2), the attention module focuses on the input feature map F b Use convolution for multiple downsampling, use interpolation for multiple upsampling, and use convolution to fuse the stacked information; finally, use the classification head to classify each point on the feature map into a character or a license plate, and generate multiple character attention maps and a license plate attention map. The number of character attention maps depends on the maximum number of characters contained in a license plate; the output of the attention module is H a and W a Respectively represent the height and width of the attention map size, M lp represents the license plate attention map, Attention map for each character index; contains the attention hotspots of the characters corresponding to the index position of each license plate in the image, M lp contains the license plate attention hotspots for each license plate in the image.

5. The integrated license plate detection and recognition method according to claim 4, characterized in that: In step 3), the recognition branch divides the image into H r *W r recognition grid, H r and W r is a positive integer. When the license plate image falls within a certain recognition grid, the recognition grid is responsible for recognizing the characters of the license plate. The detection branch divides the image into H d *W d detection grid, H d and W d is a positive integer, and the anchor point is used as the coordinate reference point. The anchor point is the center point of the detection grid. The output of the detection branch is H d *W d *(l, t, r, b, c), l, t, r, b are the distances from the anchor point to the left, top, right, and bottom borders of the bounding box of the target to be detected; c is the detection confidence, and its label is the IoU value between the predicted bounding box and the real bounding box; the recognition branch combines the feature map and the attention map by point multiplication to predict the character category, and the feature representation of the input recognition branch is That is, the sum of the characteristics of the character and the characteristics of part of the license plate.

6. The integrated license plate detection and recognition method according to claim 5, characterized in that: In step 4), the fusion module performs information matching according to the corresponding position. When the detection anchor point falls within a certain recognition grid, the recognition result of the recognition grid is assigned to the detection anchor point; the confidence score of each license plate is the integration of the detection confidence and the recognition confidence: Where c is the detection confidence, is the recognition probability of each character of the license plate; finally, non-maximum suppression is performed on the output to remove redundant results.

7. The integrated license plate detection and recognition method according to claim 1, characterized in that: In step 3), the detection network of the detection branch uses multiple layers of SE ResBlock to stack and perform one or more downsampling; the recognition network of the recognition branch uses multiple layers of SE ResBlock to stack and perform one or more downsampling, and the recognition network of each character index shares parameters.

8. The integrated license plate detection and recognition method according to claim 1, characterized in that: In step 4), the character results predicted by the recognition branch are first concatenated according to their index order, and then the concatenated character sequence is assigned to the position predicted by the detection branch, and finally, redundant results are removed according to the confidence of detection and recognition.

9. The integrated license plate detection and recognition method according to claim 1, characterized in that: When training the network, the network parameters are adjusted and learned using back propagation through the loss function, and the loss function includes three parts: attention loss function, detection loss function and recognition loss function.

10. The integrated license plate detection and recognition method according to claim 9, characterized in that: The attention loss function is a weighted binary cross entropy loss; the detection loss function consists of a positioning loss and a confidence loss; and the recognition loss function is a classification cross entropy loss.

Citation Information

Patent Citations

  • Multi-vehicle-license-plate dynamic identification method of real-time traffic scene

    CN108509954A

  • Multi-type license plate recognition method

    CN112016556A

  • Multi-type license plate recognition method and system based on single character attention

    CN113554030A

  • Multi-type and multi-license-plate license plate recognition method, device and system

    CN114581897A

  • End-to-end license plate detection and recognition method based on deep learning

    CN114581904A