License plate detection method based on inter-plate cooperative learning mechanism

Abstract

This invention belongs to the field of license plate detection technology and relates to a license plate detection method based on a collaborative learning mechanism among license plate images. It leverages the strong consistency among license plate detection images in a dataset and employs an inter-scale collaborative attention mechanism to learn the consistency between the output features of different network layers in different license plate detection images. This ensures the interactivity of license plate features during the feature extraction stage, enabling the sharing of common features. Then, a semantic collaborative attention mechanism is used to learn the consistency between the high-level output semantics of different license plate detection images, allowing the network to narrow the search range of license plate features based on semantic features. Finally, a semantic-scale collaborative learning mechanism fuses the relationships between the network's semantic context, improving the accuracy of license plate detection in complex scenes. This method can be used not only for license plate detection in complex scenes but also for object segmentation and detection in complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of license plate detection technology and relates to a license plate detection method based on a collaborative learning mechanism among license plates. Background Technology

[0002] Especially in the field of license plate recognition, more and more artificial intelligence-based algorithms have been proposed, achieving excellent performance and have been widely applied in daily life.

[0003] Traditional license plate recognition algorithms are mostly manually designed feature extraction algorithms, which are very limited in application in complex and ever-changing real-life scenarios, especially in challenging situations (rainy or snowy weather, high / low contrast). This results in very limited performance of license plate detection algorithms. With the emergence of large-scale license plate recognition datasets, deep learning-based methods have gradually replaced traditional methods. However, deep learning-based approaches still have many problems. This is because deep learning is data-driven, and its performance depends on the diversity of the dataset. However, existing algorithms do not fully mine the dataset, often only considering the detection of a single image. The focus is mainly on network design to improve detection accuracy. Moreover, single-image feature mining can easily lead to the loss of dataset diversity information and insufficient mining of mutual information between license plates in the dataset. Therefore, when the dataset changes, the network's generalization performance is low, resulting in low license plate recognition accuracy in complex scenarios. Therefore, a new license plate detection method is needed to improve the accuracy of license plate detection in complex scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and to design a license plate detection method based on a collaborative learning mechanism among license plates.

[0005] To achieve the above objectives, this invention first extracts output features from different layers of the backbone network as multi-scale features. Based on these multi-scale features, a scale-based collaborative attention mechanism is used to learn the consistency between the output features of different network layers in different license plate detection images, thereby ensuring the interactivity between license plate features during the feature extraction stage. Then, a semantic collaborative attention mechanism is used to learn the consistency between the semantic outputs of different high-level license plate detection images, ensuring that the network can narrow the search range for license plate features using semantic features. Finally, a semantic-scale collaborative learning mechanism is used to fuse the relationships between the semantic context of the network, improving the accuracy of license plate detection in complex scenes. The specific steps include:

[0006] (1) Collect license plate images in different complex scenarios to construct a license plate detection dataset. Each image contains the vertex coordinates of the license plate and the character annotation information of the license plate. The constructed license plate detection dataset is divided into a training set, a validation set and a test set.

[0007] (2) Input the two license plate images in the training set into the basic network respectively, take the intermediate layer feature output of the basic network as multi-scale features, and then transform the side outputs of different layers of the network into a feature layer of the same dimension through the convolutional layer. Then, aggregate the multi-scale features through downsampling and upsampling operations to obtain multi-scale aggregated features.

[0008] (3) Based on the multi-scale aggregation features obtained in step (2), the rich scale information interaction between license plates is learned through the scale collaborative attention mechanism. The scale information interaction adopts the collaborative relationship modeling method, that is, the scale information interaction is carried out by learning the similarity matrix between license plates, thereby improving the consistency between license plates, while filtering out the background interference information of the license plate area, and obtaining the refined foreground area of ​​the license plate.

[0009] (4) The semantic collaborative attention mechanism is used to solve the consistency of the high-level regions between license plates, highlighting the license plate regions with the same semantics, and obtaining the most discriminative regional features of the license plate;

[0010] (5) The semantic scale collaborative learning mechanism is used to collaboratively learn the refined foreground region of the license plate obtained in step (3) and the most discriminative region features of the license plate obtained in step (4) to obtain refined license plate detection region features;

[0011] (6) The refined license plate detection area features are transformed into license plate classification information and license plate coordinate information. The aggregated features are then weighted and fused with the initial features to enhance the common license plate area features and filter out background interference information.

[0012] (7) Use the training set constructed in step (1) to train the license plate detection network;

[0013] (8) Use the test set constructed in step (1) to test the license plate detection network trained in step (7), output the license plate category confidence and output the coordinate location information;

[0014] (9) Based on the results output by the license plate detection network, the characters in the license plate are identified and the license plate character information is output by using the LSTM-based license plate recognition algorithm to complete the license plate detection.

[0015] As a further technical solution of the present invention, the basic network in step (2) is a VggNet network, and the convolutional kernel of the convolutional layer is The multi-scale features of the two images are as follows: and Based on multi-scale features The obtained multi-scale aggregated features: Where Con represents the feature overlay operation, and / represents upsampling or downsampling operations on features at different scales. Using the same operation, multi-scale aggregated features are obtained. .

[0016] As a further technical solution of the present invention, the specific process of step (3) is as follows:

[0017] First, calculate the inter-scale relation matrix to obtain ,

[0018] in, It's matrix multiplication; Norm is a dimension-based normalization function. It is the L2 normalization function, which normalizes the numerical values ​​of the relation matrix. The relation matrix is ​​sparsified to prevent noise interference between license plate information, and then multi-scale feature aggregation is performed. , Enhanced , :

[0019]

[0020] in, It is based on the multiplication between matrix elements, and The calculation method is defined as follows:

[0021]

[0022] The definition of the function and The definition of functions is the same in both languages.

[0023] As a further technical solution of the present invention, the specific process of step (4) is as follows:

[0024] First, calculate the semantic relation matrix to obtain:

[0025] ,

[0026] Then, semantic features and Enhancement results:

[0027]

[0028] The definition is as follows:

[0029]

[0030] The definitions of each symbol and The definition of functions is the same in both languages.

[0031] As a further technical solution of the present invention, the specific process of step (5) is as follows:

[0032]

[0033]

[0034] The calculation method and The calculation method is the same.

[0035] As a further technical solution of the present invention, the process of weighted fusion of the aggregated features and the initial features in step (6) is as follows:

[0036]

[0037] The calculation method is as follows: .

[0038] As a further technical solution of the present invention, the specific process of step (7) training the license plate detection network is as follows: the two image data of the training set are... and Input the network, and get the license plate numbers respectively. Category confidence and regression coordinate position and license plate Category confidence and regression coordinate position Category confidence and Determine if the current branch prediction is a license plate and regress to the coordinate position. and Given the coordinates of the four vertices of the license plate, FocalLoss is used to calculate the license plate value. and Loss, Smooth L1 Loss calculation of license plate and The error is calculated by saving the model parameters with the highest accuracy after 55 training iterations on the complete training set. These parameters are the trained model parameters, where W, H, and N are the width, height, and number of images, respectively.

[0039] As a further technical solution of the present invention, the specific process of step (8) testing the license plate detection network is as follows: The test set image data... and The input is fed into the network, and the model parameters trained in step (7) are loaded. The network outputs the confidence score of the license plate category. and regression coordinate position Then, low-confidence license plates are filtered out by thresholding, and finally, non-maximum suppression (NMS) is used to remove redundant license plate detection boxes from the network output.

[0040] Compared with existing technologies, this invention is based on the strong consistency among license plate detection images in a license plate detection dataset. It learns the consistency between the output features of different license plate detection images' network layers through an inter-scale collaborative attention mechanism, thereby ensuring the interactivity of license plate features during the feature extraction stage. Through this interaction, common features are shared. Then, a semantic collaborative attention mechanism learns the consistency between the high-level output semantics of different license plate detection images, ensuring the network can narrow the search range of license plate features using semantic features. Finally, a semantic-scale collaborative learning mechanism fuses the relationships between the network's semantic context, improving the accuracy of license plate detection in complex scenes. This invention can be used not only for license plate detection in complex scenes but also for object segmentation and detection in complex scenes, achieving an accuracy of 97.3% on the CCPD dataset. Attached Figure Description

[0041] Figure 1 This is a structural diagram of the scale / semantic collaborative attention mechanism described in this invention.

[0042] Figure 2 This is a structural diagram of the semantic scale collaborative learning mechanism described in this invention.

[0043] Figure 3 This is a diagram of the entire network structure for license plate detection in this invention.

[0044] Figure 4 This is a flowchart of the license plate detection process of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0046] Example:

[0047] This embodiment uses, as follows: Figure 3 The network shown and Figure 4 The process shown implements license plate detection, specifically including the following steps:

[0048] (1) Construct a license plate detection dataset:

[0049] A license plate detection dataset was constructed by collecting license plate detection images in different complex scenarios. Each image contains the vertex coordinates of the license plate and the character annotation information of the license plate. The constructed license plate detection dataset was divided into a training set, a validation set and a test set.

[0050] (2) Multi-scale feature extraction:

[0051] To obtain multi-scale information about license plates, two license plate images (I1, I2) from the training set are input into the base network (VggNet), and the intermediate layer features (Conv0->4) of the base network are output as multi-scale features. , Then, the side outputs of different layers of the network (Conv_0->4) are passed through the convolutional layer Conv( Transformed into a unified dimension feature layer Through downsampling and upsampling operations ( Aggregate multi-scale features.

[0052]

[0053] Where Con represents the feature overlay operation, and / represents the upsampling or downsampling operation applied to features at different scales. Using the same operation, multi-scale aggregated features are obtained. .

[0054] (3) Inter-scale collaborative attention mechanism:

[0055] Extracted multi-scale features , Building upon this foundation, a rich scale information interaction mechanism is employed between license plates to learn the interactions between them. This inter-scale interaction utilizes a collaborative relationship modeling approach, specifically learning the similarity matrix between license plates. This mutual learning of license plate features enhances consistency while filtering out background interference in the license plate region.

[0056] ,

[0057] in, It's matrix multiplication; Norm is a dimension-based normalization function. It is the L2 normalization function, which normalizes the numerical values ​​of the relation matrix. Sparsification of the relationship matrix prevents interference from noisy data in the license plate information.

[0058] After obtaining the inter-scale relation matrix, multi-scale features are analyzed. , Enhance,

[0059]

[0060] in, It is based on the multiplication between matrix elements, and The calculation method is defined as follows:

[0061]

[0062] The definition of the function and The definition of functions in Chinese is the same;

[0063] (4) Semantic collaborative attention mechanism:

[0064] In license plate detection, high-level semantic information is crucial for license plate localization compared to low-level features. It can significantly narrow down the problem domain for license plate detection. Furthermore, by solving for consistency among high-level regions of license plates, it is possible to highlight license plate regions with the same semantics. Specifically:

[0065] ,

[0066] The definitions of each symbol are as follows: As defined in the text, after obtaining the semantic relation matrix, semantic features are... and Enhance,

[0067]

[0068] and The definition is as follows:

[0069]

[0070] The definitions of each symbol and The definition of functions in Chinese is the same;

[0071] (5) Semantic scale collaborative learning mechanism:

[0072] To fully integrate license plate context information, the features obtained in steps (3) and (4) are collaboratively learned to obtain refined license plate detection region features:

[0073]

[0074]

[0075] for The calculation method and The calculation method is the same;

[0076] (6) Refined output of license plate location:

[0077] The refined license plate region features are transformed into license plate classification information and license plate coordinate information. The aggregated features are then weighted and fused with the initial features to enhance common license plate region features and filter out background interference information.

[0078]

[0079] and The calculation method is as follows: ;

[0080] (7) Training the network:

[0081] The license plate detection network is trained using the training set constructed in step (1), with image data as the input. and Output license plate Category confidence and regression coordinate position and license plate Category confidence and regression coordinate position Category confidence and Determine if the current branch prediction is a license plate and regress to the coordinate position. and Given the coordinates of the four vertices of the license plate, this invention uses FocalLoss to calculate the license plate coordinates. and Loss, Smooth L1 Loss calculation of license plate and The error is calculated by saving the model parameters with the highest accuracy after training iterations on the complete training set (55 times) using the set epoach.

[0082] (8) Test network:

[0083] The license plate detection network was tested using the test set constructed in step (1), and the license plate detection image data was processed. and The input is fed into the network proposed in this embodiment, and the trained model parameters are loaded. The network output is the confidence score of the license plate category. and regression coordinate position Then, low-confidence license plates are filtered out by thresholding, and finally, non-maximum suppression (NMS) is used to remove redundant license plate detection boxes in the network output.

[0084] (9) License plate recognition:

[0085] Based on the results output by the license plate detection network, the LSTM-based license plate recognition algorithm is used to identify the characters in the license plate and output the license plate character information.

[0086] This embodiment, based on multi-scale features, learns the consistency between the output features of different license plate detection network layers through an inter-scale collaborative attention mechanism, thereby ensuring the interactivity between license plate features in the feature extraction stage. Then, it learns the consistency between the high-level output semantics of different license plate detection images through a semantic collaborative attention mechanism, thereby ensuring that the network can narrow the search range of license plate features by relying on semantic features. Finally, it integrates the relationship between the semantic context of the network through a semantic scale collaborative learning mechanism, thereby improving the accuracy of license plate detection in complex scenes.

[0087] Algorithms and network structures not described in detail in this article are general techniques in this field.

[0088] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A license plate detection method based on a collaborative learning mechanism among license plates, characterized in that, Specifically, the steps include the following: (1) Collect license plate images in different complex scenarios to construct a license plate detection dataset. Each image contains the vertex coordinates of the license plate and the character annotation information of the license plate. The constructed license plate detection dataset is divided into a training set, a validation set and a test set. (2) Input the two license plate images in the training set into the basic network respectively, take the intermediate layer feature output of the basic network as multi-scale features, and then transform the side outputs of different layers of the network into a feature layer of the same dimension through the convolutional layer. Then, aggregate the multi-scale features through downsampling and upsampling operations to obtain multi-scale aggregated features. (3) Based on the multi-scale aggregated features obtained in step (2), the rich scale information interaction between license plates is learned through the scale-to-scale collaborative attention mechanism. The scale information interaction adopts the collaborative relationship modeling method, that is, the scale information interaction is carried out by learning the similarity matrix between license plates, thereby improving the consistency between license plates and filtering out the background interference information of the license plate area to obtain the refined foreground area of ​​the license plate; the specific process is as follows: First, calculate the inter-scale relation matrix to obtain , in, It's matrix multiplication; Norm is a dimension-based normalization function. It is the L2 normalization function, which normalizes the numerical values ​​of the relation matrix. The relation matrix is ​​sparsified to prevent noise interference between license plate information, and then multi-scale feature aggregation is performed. , Enhanced , : ， in, It is based on the multiplication between matrix elements, and The calculation method is defined as follows: ， The definition of the function and The definition of functions in Chinese is the same; (4) The semantic collaborative attention mechanism is used to solve the consistency of the high-level regions between license plates, highlighting the license plate regions with the same semantics, and obtaining the most discriminative regional features of the license plate; (5) The semantic scale collaborative learning mechanism is used to collaboratively learn the refined foreground region of the license plate obtained in step (3) and the most discriminative region features of the license plate obtained in step (4) to obtain refined license plate detection region features; (6) The refined license plate detection area features are transformed into license plate classification information and license plate coordinate information. The aggregated features are then weighted and fused with the initial features to enhance the common license plate area features and filter out background interference information. (7) Use the training set constructed in step (1) to train the license plate detection network; (8) Use the test set constructed in step (1) to test the license plate detection network trained in step (7), output the license plate category confidence and output the coordinate location information; (9) Based on the results output by the license plate detection network, the characters in the license plate are identified and the license plate character information is output by using the LSTM-based license plate recognition algorithm to complete the license plate detection.

2. The license plate detection method based on a collaborative learning mechanism among license plates according to claim 1, characterized in that, The basic network in step (2) is the VggNet network, and the convolutional kernels of the convolutional layers are... The multi-scale features of the two images are as follows: and Based on multi-scale features The obtained multi-scale aggregated features: Where Con represents the feature overlay operation, and / represents upsampling or downsampling operations on features at different scales. Using the same operation, multi-scale aggregated features are obtained. .

3. The license plate detection method based on a collaborative learning mechanism among license plates according to claim 2, characterized in that, The specific process of step (4) is as follows: First, calculate the semantic relation matrix to obtain: , Then, semantic features and Enhancement results: ， The definition is as follows: ， The definitions of each symbol and The definition of functions is the same in both languages.

4. The license plate detection method based on a collaborative learning mechanism among license plates according to claim 3, characterized in that, The specific process of step (5) is as follows: ， ， The calculation method and The calculation method is the same.

5. The license plate detection method based on a collaborative learning mechanism among license plates according to claim 4, characterized in that, Step (6) involves weighted fusion of the aggregated features with the initial features as follows: ， The calculation method is as follows: .

6. The license plate detection method based on a collaborative learning mechanism among license plates according to claim 5, characterized in that, Step (7) The specific process of training the license plate detection network is as follows: The two image data sets in the training set are... and Input the network, and get the license plate numbers respectively. Category confidence and regression coordinate position and license plate Category confidence and regression coordinate position Category confidence and Determine if the current branch prediction is a license plate and regress to the coordinate position. and Given the coordinates of the four vertices of the license plate, FocalLoss is used to calculate the license plate value. and Loss, Smooth L1 Loss calculation of license plate and The error is calculated by saving the model parameters with the highest accuracy after 55 training iterations on the complete training set. These parameters are the trained model parameters, where W, H, and N are the width, height, and number of images, respectively.

7. The license plate detection method based on a collaborative learning mechanism among license plates according to claim 6, characterized in that, Step (8) The specific process of testing the license plate detection network is as follows: The test set image data... and The input is fed into the network, and the model parameters trained in step (7) are loaded. The network outputs the confidence score of the license plate category. and regression coordinate position Then, low-confidence license plates are filtered out using a threshold, and finally, nonmaximum suppression is used to remove redundant license plate detection boxes from the network output.

Images