Universal license plate recognition method and device

Abstract

The invention discloses a universal license plate recognition method and device. The method comprises the steps that a to-be-recognized target image input by a target object and configured regional information are acquired; analyzing the target image by using the vehicle detection model to obtain a vehicle detection frame and a vehicle type; analyzing the vehicle area image corresponding to the vehicle detection frame by using the license plate detection model to obtain a license plate detection frame; a target candidate character set matched with the vehicle type and the regional information is determined from the multiple candidate character sets, and each candidate character set comprises preset license plate characters; and comprehensively analyzing the license plate area image corresponding to the license plate detection frame and the target candidate character set by using a license plate recognition model to obtain a license plate character recognition result, the license plate recognition model at least comprising a cross attention module. The technical problem that a traditional license plate recognition model generally can only perform accurate license plate recognition in a fixed scene and lacks universality is solved.

Description

Technical Field

[0001] This application relates to the field of license plate detection technology, and more specifically, to a general license plate recognition method and apparatus. Background Technology

[0002] License plate recognition technology, as a crucial component of intelligent transportation systems, plays a key role in urban traffic management, vehicle safety monitoring, and parking automation. In recent years, with the deepening development of smart city and smart transportation concepts, the demand for license plate recognition technology has been increasing, especially in the management of non-motorized vehicles, where its importance has become increasingly prominent. Non-motorized vehicles, such as electric bicycles and motorcycles, have become a new challenge for urban traffic management due to their large numbers and complex driving behaviors. However, the diversity of non-motorized vehicle license plates, including their small size, inconsistent styles, and non-fixed character sets, has brought unprecedented challenges to traditional license plate recognition systems.

[0003] Traditional license plate recognition solutions are mostly based on fixed license plate styles and limited character sets, such as yolov8-plate. While they perform well in recognizing standard motor vehicle license plates, their recognition rate drops significantly when dealing with non-motorized vehicle license plates, especially those with large size differences, multiple lines of characters, or special characters. For example, motor vehicle license plates are usually single or double lines, while some non-motorized vehicle license plates may contain more lines of characters, requiring the recognition system to be flexible enough to adapt to varying line counts. Furthermore, some non-motorized vehicle license plates may contain numbers, letters, and specific symbols or Chinese characters, necessitating a broader character set. Simultaneously, due to the diversity of non-motorized vehicle license plates, traditional OCR (Optical Character Recognition) technology struggles to guarantee accuracy with limited training data, and the recognition rate further decreases under complex lighting conditions and with damaged license plates.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a general license plate recognition method and apparatus to at least solve the technical problem that traditional license plate recognition models can only perform accurate license plate recognition in fixed scenarios and lack versatility.

[0006] According to one aspect of the embodiments of this application, a general license plate recognition method is provided, comprising: acquiring a target image to be recognized and configured regional information input by a target object; analyzing the target image using a vehicle detection model to obtain a vehicle detection box and a vehicle type; analyzing the vehicle region image corresponding to the vehicle detection box using a license plate detection model to obtain a license plate detection box; determining a set of target candidate characters that match the vehicle type and regional information from multiple candidate character sets, wherein each candidate character set includes a preset license plate character; and comprehensively analyzing the license plate region image corresponding to the license plate detection box and the set of target candidate characters using a license plate recognition model to obtain a license plate character recognition result, wherein the license plate recognition model includes at least a cross-attention module.

[0007] Optionally, determining a target candidate character set that matches vehicle type and regional information from multiple candidate character sets includes: determining a first candidate character set that matches vehicle type and a second candidate character set that matches regional information from a pre-configured candidate character database, wherein the candidate character database stores multiple candidate character sets corresponding to different vehicle types and multiple candidate character sets corresponding to different regional information; and determining the union of the first candidate character set and the second candidate character set as the target candidate character set.

[0008] Optionally, the method for determining the license plate region image includes: determining the angle between the horizontal boundary of the license plate detection box in the vehicle region image and the horizontal direction, and performing a radial transformation on the first region image corresponding to the license plate detection box in the vehicle region image based on the angle to obtain a second region image including the front-view license plate; analyzing the second region image using an image text detection model to determine the third region image corresponding to each line of horizontal text and the fourth region image corresponding to each column of vertical text in the second region image; for each fourth region image, determining the fifth region image corresponding to each character in the fourth region image, and stitching multiple fifth region images horizontally in sequence to obtain a sixth region image; and using all the third region images and the sixth region images as the license plate region image.

[0009] Optionally, the license plate recognition model is used to comprehensively analyze the license plate region image corresponding to the license plate detection box and the target candidate character set to obtain the license plate character recognition result. This includes: extracting image features corresponding to the license plate region image using the image encoding module in the license plate recognition model; extracting character features corresponding to each character in the target candidate character set using the character encoding module in the license plate recognition model; fusing image features and character features using the cross-attention module in the license plate recognition model to obtain fused features; performing dimensional transformation, information integration, and expression enhancement on the fused features using the linear mapping module in the license plate recognition model to obtain the linear mapping result; and calculating the probability distribution of the linear mapping result using the output module in the license plate recognition model to obtain the license plate character recognition result.

[0010] Optionally, the image features and character features are fused using the cross-attention module in the license plate recognition model to obtain fused features. This includes: inputting the image features and character features into the cross-attention module, wherein the cross-attention module is used to calculate the fused features according to the following formula: , In the formula, A represents the attention matrix. Represents the image feature matrix. The transpose of the key matrix representing character features. A value matrix representing character features. Indicates fusion features, This is a preset scaling factor used to prevent gradient explosion.

[0011] Optionally, the training process of the license plate recognition model includes: constructing an initial model, which includes an image encoding module, a character encoding module, a cross-attention module, a linear mapping module, and an output module; acquiring multiple license plate images and the vehicle type, geographical location, and character information corresponding to each license plate image; for each license plate image, determining a set of candidate characters that match the vehicle type and geographical location information corresponding to the license plate image, using the license plate image and the set of candidate characters as a set of training samples, and using the character information included in the license plate image as the corresponding sample label; iteratively training the initial model using multiple sets of training samples and sample labels to obtain the license plate recognition model, wherein, during the training process, a connection-time classification loss function is constructed based on the difference between the model output and the sample labels, and the model parameters are adjusted based on the backpropagation mechanism.

[0012] Optionally, the license plate detection model's analysis results of the vehicle region image also include: license plate color; the method further includes: unifying the license plate color and license plate character recognition results as the license plate recognition result; and feeding back the license plate recognition result to the target object.

[0013] According to another aspect of the embodiments of this application, a universal license plate recognition device is also provided, comprising: an acquisition module for acquiring a target image to be recognized and configured regional information input by a target object; a first detection module for analyzing the target image using a vehicle detection model to obtain a vehicle detection box and a vehicle type; a second detection module for analyzing the vehicle region image corresponding to the vehicle detection box using a license plate detection model to obtain a license plate detection box; a matching module for determining a set of target candidate characters that match the vehicle type and regional information from multiple candidate character sets, wherein each candidate character set includes a preset license plate character; and a recognition module for comprehensively analyzing the license plate region image corresponding to the license plate detection box and the set of target candidate characters using a license plate recognition model to obtain a license plate character recognition result, wherein the license plate recognition model includes at least a cross-attention module.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product comprising: a computer program, wherein the computer program, when executed by a processor, implements the above-described general license plate recognition method.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described general license plate recognition method through the computer program.

[0016] In this embodiment, the target image to be identified and the configured regional information are first acquired. Then, a vehicle detection model is used to analyze the target image to obtain the vehicle detection box and vehicle type. Next, a license plate detection model is used to analyze the region image corresponding to the vehicle detection box to obtain the license plate detection box. Simultaneously, a set of target candidate characters matching the vehicle type and regional information is determined from multiple candidate character sets containing preset characters. Finally, a license plate recognition model with a cross-attention mechanism is used to comprehensively analyze the region image corresponding to the license plate detection box and the set of target candidate characters to obtain the license plate character recognition result. By matching the set of target candidate characters, the limitation of fixed character sets in related license plate recognition systems can be overcome, improving the system's recognition flexibility and versatility. The license plate recognition model with a cross-attention mechanism can adaptively adjust the attention to characters based on the features of the input license plate image, achieving fine-grained alignment between the image and text, and improving the accuracy of license plate character recognition. This solution effectively solves the technical problem that traditional license plate recognition models can only perform accurate license plate recognition in fixed scenarios and lack versatility. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart illustrating an optional universal license plate recognition method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the structure of an optional universal license plate recognition device according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0024] Connectionist Temporal Classification (CTC) loss function: This is a loss function specifically designed for sequence prediction tasks, especially those with different input and output lengths. It can handle the problem of mismatched input and output sequence lengths, while allowing the network model to ignore blank labels that are not important for recognition, thus enabling end-to-end training.

[0025] Example 1

[0026] According to an embodiment of this application, a general license plate recognition method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] Figure 1 This is a flowchart illustrating a general license plate recognition method according to an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0028] Step S102: Obtain the target image to be identified and the configured geographic information input by the target object;

[0029] Step S104: Analyze the target image using the vehicle detection model to obtain the vehicle detection box and vehicle type;

[0030] Step S106: Analyze the vehicle region image corresponding to the vehicle detection box using the license plate detection model to obtain the license plate detection box;

[0031] Step S108: Determine a target candidate character set that matches the vehicle type and geographical information from multiple candidate character sets, wherein each candidate character set includes preset license plate characters;

[0032] Step S110: The license plate recognition model is used to comprehensively analyze the license plate area image corresponding to the license plate detection box and the target candidate character set to obtain the license plate character recognition result. The license plate recognition model includes at least a cross-attention module.

[0033] The following section explains each step of the general license plate recognition method in conjunction with the specific implementation process.

[0034] As an optional implementation, in step S102, the target image to be identified input by the target object can be an image of vehicles on the road captured by a road monitoring camera. Specific vehicle types include, but are not limited to, cars, trucks, buses, motorcycles, bicycles, electric vehicles, etc.

[0035] Considering that the design format and numbering rules for license plates of various types of vehicles may vary in different regions, the target user can also configure the regional information corresponding to the image to be recognized, such as the country and province, so as to select the character set that matches the regional information as the constraint for license plate character recognition and improve the accuracy of recognition.

[0036] As an optional implementation, in step S104, a pre-trained vehicle detection model can be used to analyze the target image to obtain vehicle detection boxes and vehicle types. The vehicle detection model can be a computer vision deep learning model such as YOLO (YouOnly Look Once), SSD (Single Shot MultiBox Detector), RCNN (Region with Convolutional Neural Network), or DETR (Detection Transformer).

[0037] Optionally, when training the vehicle detection model, images of various types of vehicles can be collected as training samples, and the vehicle bounding box and vehicle type in each image can be labeled as corresponding sample labels. By iteratively training the initial model using multiple sets of training samples and sample labels, a vehicle detection model that can accurately identify vehicle detection boxes and vehicle types can be obtained.

[0038] As an optional implementation, in step S106, a pre-trained license plate detection model can be used to analyze the vehicle region image corresponding to the vehicle detection box to obtain the license plate detection box. The vehicle region image corresponding to the vehicle detection box can be a sub-image cropped from the target image containing only the vehicle detection box region. The license plate detection model can also be a computer vision deep learning model such as YOLO, SSD, RCNN, or DETR.

[0039] Considering that license plate color can reflect vehicle type and region affiliation to some extent—for example, blue and green license plates can distinguish between gasoline-powered vehicles and new energy vehicles, while yellow license plates can identify commercial transport vehicles or special vehicles—license plate color is also an important indicator for license plate recognition. In this embodiment, when analyzing the vehicle region image corresponding to the vehicle detection box using the license plate detection model, the license plate color can be determined simultaneously and used as one of the factors in the final license plate recognition result.

[0040] To achieve the above analysis, when training the license plate detection model, images of license plates of various colors and styles can be collected as training samples. The bounding box and color of the license plate in each image can be labeled as corresponding sample labels. By iteratively training the initial model using multiple sets of training samples and sample labels, a license plate detection model that can accurately identify the license plate detection box and license plate color can be obtained.

[0041] As an optional implementation, in step S108, the target candidate character set can be determined in the following way: a first candidate character set matching the vehicle type and a second candidate character set matching the regional information are determined from a pre-configured candidate character database, wherein the candidate character database stores multiple candidate character sets corresponding to different vehicle types and multiple candidate character sets corresponding to different regional information; then the union of the first candidate character set and the second candidate character set is determined as the target candidate character set.

[0042] Understandably, configuring different candidate character sets based on the characteristics of license plates from different vehicle types and regions, forming a candidate character database, can ensure both the specificity of license plate characters in a single candidate character set and the breadth of license plate characters in the entire candidate character database. In practical applications, by determining the target candidate character set that matches the current vehicle type and region information as a constraint condition for the output characters of the subsequent license plate recognition model, the flexibility, versatility, and accuracy of license plate character recognition can be effectively improved.

[0043] As an optional implementation, in step S110, a pre-trained license plate recognition model containing a cross-attention module can be used to comprehensively analyze the license plate region image corresponding to the license plate detection box and the target candidate character set to obtain the license plate character recognition result.

[0044] To improve the accuracy of character recognition for different license plate styles, this application proposes the following steps to determine the license plate area image:

[0045] S1. Determine the angle between the horizontal boundary of the license plate detection box in the vehicle region image and the horizontal direction, and perform a radial transformation on the first region image corresponding to the license plate detection box in the vehicle region image based on the angle to obtain a second region image including the front-view license plate.

[0046] Among them, the radiative transformation is a mathematical model that describes the projection relationship between two planes. It is especially applicable to the relationship between a camera and the plane being photographed. It can map a point on one plane to another. In license plate recognition, the radiative transformation can map the detected tilted license plate area into a rectangular frontal license plate area to facilitate subsequent character recognition.

[0047] S2, using an image text detection model to analyze the second region image, determine the third region image corresponding to each row of horizontal text and the fourth region image corresponding to each column of vertical text in the second region image.

[0048] The image text detection model used here can also be a deep learning model for computer vision such as YOLO, SSD, RCNN, or DETR. This model can accurately identify each horizontal line and each vertical column of text in the front-facing license plate area, ensuring universality for recognizing various license plate styles. It should be noted that the model here only needs to identify the regions corresponding to the horizontal and vertical text; it does not need to perform fine-grained character recognition.

[0049] S3. For each fourth region image, determine the fifth region image corresponding to each character in the fourth region image, and stitch multiple fifth region images horizontally in sequence to obtain the sixth region image.

[0050] By breaking down each column of vertical text character by character and then reassembling them sequentially into horizontal text, subsequent character recognition can be further facilitated.

[0051] S4, use all the third and sixth region images as license plate region images.

[0052] Traditional OCR models based on CNN (Convolutional Neural Network) determine all possible output characters during training, without an explicit mechanism to enforce constraints during inference. Their models only input one modality of image during inference, without image-text feature fusion process, and the character set is fixed by default and cannot be changed. Furthermore, the general OCR character set based on large models is extremely large, possessing massive world knowledge and image-text feature alignment, but it focuses on the alignment of image and text semantic features, and the output is uncontrollable, resulting in huge deployment and inference costs.

[0053] To overcome the above problems, this application provides a license plate recognition model based on a multimodal cross-attention mechanism. This model uses a set of candidate characters as an explicit and dynamic conditional input, and utilizes the cross-attention mechanism to perform fine-grained text feature alignment and fusion of image features and character features, as well as output constraints, to ensure the universality of license plate recognition and the accuracy of the recognition results.

[0054] Alternatively, the license plate recognition model can be trained using the following steps:

[0055] S1, Construct the initial model, which includes: an image encoding module, a character encoding module, a cross-attention module, a linear mapping module, and an output module.

[0056] The image encoding module can employ networks such as ResNet (Residual Network) and ViT (Vision Transformer) to extract image features corresponding to the license plate area image.

[0057] The character encoding module can employ networks such as BPE (Byte Pair Encoding) and CLIP (Contrastive Language-Image Pre-training) to extract character features corresponding to each character in the target candidate character set. For example, BPE can be used to tokenize each character in the candidate character set, and then CLIP can be used to extract the corresponding character features.

[0058] The cross-attention module is a core component of the license plate recognition model in this application embodiment, used to fuse image features extracted by the image encoding module and character features extracted by the character encoding module. Its working principle formula is as follows:

[0059]

[0060]

[0061] In the formula, A represents the attention matrix. Represents the image feature matrix. Key matrix representing character features The transpose of the matrix, A value matrix representing character features. Indicates fusion features, This is a preset scaling factor used to prevent gradient explosion.

[0062] The linear mapping module is used to perform dimensional transformation, information integration, and expression enhancement on the fused features to obtain the linear mapping result. It is usually a fully connected layer that can convert the model's internal representation into specific category labels.

[0063] The output module is used to calculate the probability distribution of the linear mapping result output by the linear mapping module to obtain the license plate character recognition result, which can be calculated and output using the softmax activation function.

[0064] S2, obtain multiple license plate images and the vehicle type, geographical location, and character information corresponding to each license plate image.

[0065] S3. For each license plate image, determine the candidate character set that matches the vehicle type and geographical location information corresponding to the license plate image. Use the license plate image and the candidate character set as a set of training samples, and use the character information included in the license plate image as the corresponding sample label.

[0066] It should be noted that the license plate images used as training samples are images that have been preprocessed from ordinary license plate images using the method described above for determining license plate region images. Each license plate image is labeled with vehicle type, region information, and included character information. The method described above for determining the target candidate character set can be used to determine the candidate character set corresponding to each license plate image. Then, each license plate image and its corresponding candidate character set are used as a set of training samples, and the character information in the license plate image is used as the corresponding sample label.

[0067] S4. The initial model is iteratively trained using multiple sets of training samples and sample labels to obtain a license plate recognition model. During the training process, a connection-time classification loss function can be constructed based on the difference between the model output and the sample labels, and the model parameters can be adjusted based on the backpropagation mechanism.

[0068] Specifically, in each training batch during the training process, the license plate recognition model can obtain the predicted character sequences output by analyzing multiple training samples within the current batch. A connection-time classification loss function is constructed based on the differences between each predicted character sequence and the character sequences in the corresponding sample labels. The gradient of the model parameters is calculated using a backpropagation mechanism, and then the model parameters are adjusted based on gradient descent optimization algorithms (such as stochastic gradient descent, Adam, etc.). Through continuous iterative training until a preset termination condition is met (such as the number of iterations reaching a preset threshold, or the loss function converging), a license plate recognition model capable of accurately recognizing license plate characters can be obtained.

[0069] In practical applications, after inputting the license plate region image corresponding to the license plate detection box and the target candidate character set into the trained license plate recognition model, the final license plate character recognition result can be obtained through the following analysis steps:

[0070] S1, use the image encoding module in the license plate recognition model to extract the image features corresponding to the license plate area image;

[0071] S2, use the character encoding module in the license plate recognition model to extract the character features corresponding to each character in the target candidate character set;

[0072] S3 utilizes the cross-attention module in the license plate recognition model to fuse image features and character features to obtain fused features;

[0073] S4. The linear mapping module in the license plate recognition model is used to perform dimensional transformation, information integration and expression enhancement on the fused features to obtain the linear mapping result.

[0074] S5 uses the output module in the license plate recognition model to calculate the probability distribution of the linear mapping result, and obtains the license plate character recognition result.

[0075] As an optional implementation, the license plate color and license plate character recognition results can be unified as the license plate recognition result in the final output, and then the license plate recognition result can be fed back to the target object.

[0076] In this embodiment, the target image to be identified and the configured regional information are first acquired. Then, a vehicle detection model is used to analyze the target image to obtain the vehicle detection box and vehicle type. Next, a license plate detection model is used to analyze the region image corresponding to the vehicle detection box to obtain the license plate detection box. Simultaneously, a set of target candidate characters matching the vehicle type and regional information is determined from multiple candidate character sets containing preset characters. Finally, a license plate recognition model with a cross-attention mechanism is used to comprehensively analyze the region image corresponding to the license plate detection box and the set of target candidate characters to obtain the license plate character recognition result. By matching the set of target candidate characters, the limitation of fixed character sets in related license plate recognition systems can be overcome, improving the system's recognition flexibility and versatility. The license plate recognition model with a cross-attention mechanism can adaptively adjust the attention to characters based on the features of the input license plate image, achieving fine-grained alignment between the image and text, and improving the accuracy of license plate character recognition. This solution effectively solves the technical problem that traditional license plate recognition models can only perform accurate license plate recognition in fixed scenarios and lack versatility.

[0077] Example 2

[0078] According to an embodiment of this application, a universal license plate recognition device for implementing the universal license plate recognition method in Embodiment 1 is also provided, such as... Figure 2 As shown, the universal license plate recognition device includes at least: an acquisition module 21, a first detection module 22, a second detection module 23, a matching module 24, and a recognition module 25, wherein:

[0079] The acquisition module 21 is used to acquire the target image to be identified and the configured regional information input by the target object;

[0080] The first detection module 22 is used to analyze the target image using a vehicle detection model to obtain the vehicle detection box and vehicle type;

[0081] The second detection module 23 is used to analyze the vehicle region image corresponding to the vehicle detection box using the license plate detection model to obtain the license plate detection box.

[0082] The matching module 24 is used to determine a target candidate character set that matches the vehicle type and geographical information from multiple candidate character sets, wherein each candidate character set includes preset license plate characters;

[0083] The recognition module 25 is used to perform comprehensive analysis on the license plate region image corresponding to the license plate detection box and the target candidate character set using the license plate recognition model to obtain the license plate character recognition result. The license plate recognition model includes at least a cross-attention module.

[0084] The following section explains the functions of each module of the universal license plate recognition device in conjunction with the specific implementation process.

[0085] As an optional implementation, the target image to be identified acquired by the acquisition module can be an image of a road containing vehicles captured by a road surveillance camera. Specific vehicle types include, but are not limited to, cars, trucks, buses, motorcycles, bicycles, and electric vehicles. Considering that the design format and numbering rules for license plates of various vehicles may differ in different regions, the acquisition module can also acquire the regional information corresponding to the target object's configuration image, such as its country and province, while acquiring the target image. This allows for the subsequent selection of a character set matching the regional information as a constraint for license plate character recognition, thereby improving recognition accuracy.

[0086] Optionally, the first detection module can use a pre-trained vehicle detection model to analyze the target image to obtain vehicle detection boxes and vehicle types.

[0087] The vehicle detection model can be trained in the following way: collect images of various types of vehicles as training samples, and label the vehicle bounding box and vehicle type in each image as corresponding sample labels. Use multiple sets of training samples and sample labels to iteratively train the initial model, and a vehicle detection model that can accurately identify vehicle detection boxes and vehicle types can be obtained.

[0088] Optionally, the second detection module can use a pre-trained license plate detection model to analyze the vehicle region image corresponding to the vehicle detection box to obtain the license plate detection box. The vehicle region image corresponding to the vehicle detection box can be a sub-image cropped from the target image that contains only the vehicle detection box region.

[0089] Considering that license plate color can also reflect the type and region of a vehicle to some extent, in this embodiment of the application, when the second detection module analyzes the vehicle area image corresponding to the vehicle detection box using the license plate detection model, it can simultaneously determine the license plate color and use it as one of the final license plate recognition results.

[0090] To achieve the above analysis, the license plate detection model can be trained in the following way: collect images of license plates of different colors and styles as training samples, and label the license plate bounding box and license plate color in each image as corresponding sample labels. Use multiple sets of training samples and sample labels to iteratively train the initial model, and a license plate detection model that can accurately identify the license plate detection box and license plate color can be obtained.

[0091] As an optional implementation, the matching module can determine the target candidate character set in the following way: determine a first candidate character set that matches the vehicle type and a second candidate character set that matches the regional information from a pre-configured candidate character database, wherein the candidate character database stores multiple candidate character sets corresponding to different vehicle types and multiple candidate character sets corresponding to different regional information; then determine the union of the first candidate character set and the second candidate character set as the target candidate character set.

[0092] As an optional implementation, the recognition module can use a pre-trained license plate recognition model that includes a cross-attention module to comprehensively analyze the license plate region image corresponding to the license plate detection box and the target candidate character set to obtain the license plate character recognition result.

[0093] To improve the accuracy of character recognition for different license plate styles, the recognition module can determine the license plate region image in the following way: determine the angle between the horizontal boundary of the license plate detection box in the vehicle region image and the horizontal direction, and perform a radial transformation on the first region image corresponding to the license plate detection box in the vehicle region image based on this angle to obtain a second region image including the front-view license plate; analyze the second region image using an image text detection model to determine the third region image corresponding to each row of horizontal text and the fourth region image corresponding to each column of vertical text in the second region image; for each fourth region image, determine the fifth region image corresponding to each character in the fourth region image, and stitch multiple fifth region images horizontally in sequence to obtain a sixth region image; use all the third region images and the sixth region images as the license plate region image.

[0094] To overcome the problems of traditional models lacking explicit mechanisms to enforce constraints during inference, only using an image as the input modality during inference, lacking a text-image feature fusion process, and having a fixed and unchangeable character set by default, this application provides a license plate recognition model based on a multimodal cross-attention mechanism. This model uses the candidate character set as an explicit and dynamic conditional input, and utilizes the cross-attention mechanism to perform fine-grained text feature alignment and fusion of image features and character features, as well as output constraints, to ensure the universality of license plate recognition and the accuracy of the recognition results.

[0095] Optionally, the license plate recognition model can be trained as follows: Construct an initial model, which includes an image encoding module, a character encoding module, a cross-attention module, a linear mapping module, and an output module; acquire multiple license plate images and the vehicle type, geographical location, and included character information corresponding to each license plate image; for each license plate image, determine a set of candidate characters matching the vehicle type and geographical location information corresponding to the license plate image, use the license plate image and the candidate character set as a set of training samples, and use the character information included in the license plate image as the corresponding sample label; iteratively train the initial model using multiple sets of training samples and sample labels to obtain the license plate recognition model. During training, a connection-time classification loss function can be constructed based on the difference between the model output and the sample labels, and the model parameters can be adjusted based on the backpropagation mechanism.

[0096] In practical applications, the recognition module can call the trained license plate recognition model to analyze the license plate region image corresponding to the license plate detection box and the target candidate character set in the following ways to obtain the final license plate character recognition result: The image encoding module in the license plate recognition model extracts the image features corresponding to the license plate region image; the character encoding module in the license plate recognition model extracts the character features corresponding to each character in the target candidate character set; the cross-attention module in the license plate recognition model fuses the image features and character features to obtain fused features; the linear mapping module in the license plate recognition model performs dimensional transformation, information integration, and expression enhancement on the fused features to obtain the linear mapping result; and the output module in the license plate recognition model calculates the probability distribution of the linear mapping result to obtain the license plate character recognition result.

[0097] As an optional implementation, the license plate color and license plate character recognition results can be unified as the license plate recognition result in the final output, and then the license plate recognition result can be fed back to the target object.

[0098] It should be noted that each module in the universal license plate recognition device in this application corresponds one-to-one with each implementation step of the universal license plate recognition method in Embodiment 1. Since Embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to Embodiment 1, and will not be elaborated further here.

[0099] Example 3

[0100] According to an embodiment of this application, a computer program product is also provided, which includes a computer program, wherein when the computer program is executed by a processor, it implements the general license plate recognition method in Embodiment 1.

[0101] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the general license plate recognition method in Embodiment 1 by running the computer program.

[0102] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the general license plate recognition method in Embodiment 1 during runtime.

[0103] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the general license plate recognition method of Embodiment 1 through the computer program.

[0104] Specifically, the computer program executes the following steps during runtime: acquiring the target image to be recognized and the configured regional information of the target object input; analyzing the target image using a vehicle detection model to obtain vehicle detection boxes and vehicle types; analyzing the vehicle region image corresponding to the vehicle detection box using a license plate detection model to obtain a license plate detection box; determining a set of target candidate characters that match the vehicle type and regional information from multiple candidate character sets, wherein each candidate character set includes preset license plate characters; and comprehensively analyzing the license plate region image corresponding to the license plate detection box and the target candidate character set using a license plate recognition model to obtain the license plate character recognition result, wherein the license plate recognition model includes at least a cross-attention module.

[0105] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 3 A hardware block diagram of an electronic device for implementing a universal license plate recognition method is shown. Figure 3 As shown, the electronic device 30 may include one or more processors 302 (shown as 302a, 302b, ..., 302n in the figure) (processor 302 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 304 for storing data, and a transmission device 306 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, electronic device 30 may also include... Figure 3 The more or fewer components shown, or having the same Figure 3The different configurations shown.

[0106] It should be noted that the aforementioned one or more processors 302 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element of the electronic device 30. As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0107] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the general license plate recognition method in this embodiment. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, thereby implementing the aforementioned application vulnerability detection method. The memory 304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include memory remotely located relative to the processor 302, and these remote memories can be connected to the electronic device 30 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0108] The transmission device 306 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the electronic device 30. In one example, the transmission device 306 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 306 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0109] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the electronic device 30.

[0110] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0111] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0116] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A general license plate recognition method, characterized by, include: Obtain the target image to be identified and the configured geographic information from the input of the target object; The target image is analyzed using a vehicle detection model to obtain vehicle detection boxes and vehicle types; The vehicle region image corresponding to the vehicle detection box is analyzed using a license plate detection model to obtain the license plate detection box; A target candidate character set matching the vehicle type and the geographical information is determined from multiple candidate character sets, wherein each candidate character set includes a preset license plate character; The license plate recognition model is used to comprehensively analyze the license plate region image corresponding to the license plate detection box and the target candidate character set to obtain the license plate character recognition result. The license plate recognition model includes at least a cross-attention module.

2. The method according to claim 1, characterized in that, Determining a target candidate character set that matches the vehicle type and the geographic information from multiple candidate character sets includes: A first set of candidate characters matching the vehicle type and a second set of candidate characters matching the regional information are determined from a pre-configured candidate character database. The candidate character database stores multiple sets of candidate characters corresponding to different vehicle types and multiple sets of candidate characters corresponding to different regional information. The union of the first candidate character set and the second candidate character set is determined as the target candidate character set.

3. The method according to claim 1, characterized in that, The methods for determining the license plate area image include: Determine the angle between the horizontal boundary of the license plate detection frame in the vehicle region image and the horizontal direction, and perform a radial transformation on the first region image corresponding to the license plate detection frame in the vehicle region image based on the angle to obtain a second region image including the front-view license plate; The image text detection model is used to analyze the second region image to determine the third region image corresponding to each row of horizontal text and the fourth region image corresponding to each column of vertical text in the second region image. For each fourth region image, determine the fifth region image corresponding to each character in the fourth region image, and then stitch multiple fifth region images horizontally in sequence to obtain the sixth region image; All of the third region images and the sixth region images are used as the license plate region images.

4. The method according to claim 1, characterized in that, The license plate recognition model is used to comprehensively analyze the license plate region image corresponding to the license plate detection frame and the target candidate character set to obtain the license plate character recognition result, including: The image features corresponding to the license plate region image are extracted using the image encoding module in the license plate recognition model. The character encoding module in the license plate recognition model is used to extract the character features corresponding to each character in the target candidate character set; The image features and character features are fused using the cross-attention module in the license plate recognition model to obtain fused features; The linear mapping module in the license plate recognition model is used to perform dimensional transformation, information integration, and expression enhancement on the fused features to obtain the linear mapping result. The probability distribution of the linear mapping result is calculated using the output module of the license plate recognition model to obtain the license plate character recognition result.

5. The method according to claim 4, characterized in that, The image features and character features are fused using the cross-attention module in the license plate recognition model to obtain fused features, including: The image features and the character features are input into the cross-attention module, wherein the cross-attention module is used to calculate the fused features according to the following formula: In the formula, A represents the attention matrix. Represents the image feature matrix. The transpose of the key matrix representing character features. A value matrix representing character features. Indicates fusion features, This is a preset scaling factor used to prevent gradient explosion.

6. The method according to claim 1, characterized in that, The training process of the license plate recognition model includes: Construct an initial model, which includes: an image encoding module, a character encoding module, a cross-attention module, a linear mapping module, and an output module; Acquire multiple license plate images and the vehicle type, geographical location, and character information corresponding to each license plate image; For each license plate image, a set of candidate characters matching the vehicle type and geographical location information corresponding to the license plate image is determined. The license plate image and the set of candidate characters are used as a set of training samples, and the character information included in the license plate image is used as the corresponding sample label. The initial model is iteratively trained using multiple sets of training samples and sample labels to obtain the license plate recognition model. During the training process, a connection-time classification loss function is constructed based on the difference between the model output and the sample labels, and the model parameters are adjusted based on the backpropagation mechanism.

7. The method according to claim 1, characterized in that, The analysis results of the license plate detection model on the vehicle region image also include: license plate color; the method further includes: The license plate color and the license plate character recognition results are combined and used as the license plate recognition result; The license plate recognition result is fed back to the target object.

8. A universal license plate recognition device, characterized in that, include: The acquisition module is used to acquire the target image to be identified and the configured geographic information input by the target object; The first detection module is used to analyze the target image using a vehicle detection model to obtain the vehicle detection box and vehicle type; The second detection module is used to analyze the vehicle region image corresponding to the vehicle detection frame using the license plate detection model to obtain the license plate detection frame. A matching module is used to determine a target candidate character set that matches the vehicle type and the geographical information from multiple candidate character sets, wherein each candidate character set includes a preset license plate character; The recognition module is used to perform comprehensive analysis on the license plate region image corresponding to the license plate detection box and the target candidate character set using a license plate recognition model to obtain the license plate character recognition result. The license plate recognition model includes at least a cross-attention module.

9. A computer program product, characterized in that, include: A computer program, wherein when executed by a processor, the computer program implements the general license plate recognition method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the universal license plate recognition method according to any one of claims 1 to 7 through the computer program.

Images