A Deep Learning-Based License Plate Recognition Method and System

Abstract

This invention discloses a license plate recognition method and system based on deep learning, comprising the following steps: acquiring license plate image data to generate an image dataset; performing semantic decomposition to divide the data into region code units, letter units, and number units, and constructing a semantic structure graph; constructing a multi-path character recognition model with an improved PCNN model as the backbone feature structure; inputting the image dataset into the multi-path character recognition model respectively, and outputting region code recognition results, letter recognition results, and number recognition results to generate candidate license plate recognition results; if there are illegal character combinations or missing structural unit information, performing path filtering processing to generate a candidate path set; performing inference imputation processing to generate a complete path set; and merging and outputting the license plate recognition results and the set of confident candidate paths. This invention improves the accuracy of character recognition while enhancing robustness to structural defects, possessing the advantages of high recognition accuracy and strong fault tolerance.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and in particular to a license plate recognition method and system based on deep learning. Background Technology

[0002] With the continuous growth in demand for intelligent transportation systems and urban vehicle management, license plate recognition technology has been widely applied in fields such as electronic police, parking management, and highway toll collection. Existing license plate recognition methods are mainly based on convolutional neural networks or end-to-end image recognition models, relying on the entire image for feature extraction and character decoding. While they perform well under standard, clear image conditions, they still face problems such as unstable recognition accuracy and incomplete character information deconstruction in complex environments.

[0003] In scenarios where license plate images are obscured, damaged, or blurred, whole-image feature modeling methods struggle to accurately extract key character information, leading to errors in character order recognition or category confusion. Existing character segmentation methods often employ projection-based or segmentation-regression-based processing, lacking the ability to model character arrangement patterns and semantic structures, resulting in inaccurate segmentation boundaries and high misclassification rates. Furthermore, traditional path optimization strategies often rely solely on cumulative character confidence for path ranking, failing to fully integrate factors such as structural integrity and semantic continuity, which can easily lead to misjudgments of confidence and reduce the reliability of the final recognition.

[0004] Therefore, how to provide a license plate recognition method and system based on deep learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a license plate recognition method and system based on deep learning. This invention improves the recognizability of character regions by performing brightness equalization and edge enhancement operations on the original license plate image through an image processing module; obtains multiple candidate paths for characters in the license plate region through a candidate extraction module; constructs a structural consistency graph and combines it with character confidence and path scoring mechanisms for multi-factor comprehensive evaluation, improving the structural rationality and character accuracy of the license plate recognition results; introduces an improved PCNN model for path completion and character confidence modeling, enhancing the system's ability to recover blurred and incomplete characters; and finally outputs the license plate recognition results and a set of confidence candidate paths through a confidence fusion module, effectively alleviating problems such as high character misjudgment rate and unreasonable path combinations in traditional methods.

[0006] A license plate recognition method based on deep learning according to an embodiment of the present invention includes the following steps:

[0007] Step 1: Collect license plate image data, perform size normalization, illumination equalization, and edge enhancement processing to generate an image dataset;

[0008] Step 2: Perform semantic decomposition on the image dataset, dividing it into region code units, letter units, and number units, and construct a semantic structure graph;

[0009] Step 3: Construct a multi-path character recognition model based on the semantic structure graph; the multi-path character recognition model includes a region code recognition sub-network, a letter recognition sub-network, and a number recognition sub-network with an improved PCNN model as the backbone feature structure; the improved PCNN model includes a segmented convolutional structure, a positional information fusion structure, and a feature concatenation structure;

[0010] Step 4: Input the image dataset into the region recognition subnetwork, letter recognition subnetwork and number recognition subnetwork respectively, and output the region code recognition result, letter recognition result and number recognition result. Concatenate the results in the preset order to generate candidate license plate recognition results.

[0011] Step 5: If there are illegal character combinations or missing structural unit information in the candidate license plate recognition results, then perform path filtering processing to generate a candidate path set;

[0012] Step 6: Perform inference and imputation processing on the candidate path set to generate a complete path set;

[0013] Step 7: Merge the candidate path set and the completed path set, and output the license plate recognition result and the set of confident alternative paths.

[0014] Optionally, step one specifically includes:

[0015] Acquire color images including the license plate area;

[0016] The acquired color images are subjected to size normalization processing, and the height and width of the images are adjusted according to the set resolution;

[0017] Perform illumination equalization processing on the image after size normalization to adjust the brightness and contrast distribution of the image;

[0018] Edge processing is performed on the image after illumination equalization. Edge contours are extracted based on image gradient information, and filtering methods are applied to process the edge regions to form an image dataset.

[0019] Optionally, step two specifically includes:

[0020] Perform character region localization processing on the image dataset, extracting the character arrangement region according to the license plate boundaries in the image;

[0021] Within the character arrangement area, character blocks are divided, and character boundaries are identified based on horizontal and vertical projection densities.

[0022] The character type recognition process is performed on the divided character regions, and the characters are divided into region code units, letter units and number units according to the encoding standard.

[0023] A semantic structure diagram is established based on the arrangement order of each character unit; the semantic structure diagram includes character category, character position information and arrangement relationship.

[0024] Optionally, step three specifically includes:

[0025] Read the character category information and arrangement order information in the semantic structure diagram to identify region code units, letter units and number units;

[0026] Set up the region code recognition path, letter recognition path, and number recognition path according to the character category classification results;

[0027] Each recognition path is configured with an input interface based on character category and location index, specifying the corresponding character region in the image dataset as the input content.

[0028] For each recognition path, the improved PCNN model is called to construct the recognition structure, and the input dimension, convolution parameters and channel mapping method of the improved PCNN model are set.

[0029] After all recognition paths are constructed, a multi-path character recognition model is formed; the multi-path character recognition model includes a region code recognition sub-network, a letter recognition sub-network, and a number recognition sub-network.

[0030] Optionally, the improved PCNN model includes a segmented convolutional structure, a positional information fusion structure, and a feature concatenation structure, specifically:

[0031] The segmented convolutional structure receives a character image input, constructs a fixed-length window according to the character center position, and divides the character image into three regions along the horizontal direction: left, middle, and right. Convolution operations are performed on the left, middle, and right regions respectively, using the same convolution kernel size and stride parameters. Local feature maps are extracted independently for each region, and three sets of segmented feature vectors are output.

[0032] The location information fusion structure constructs three sets of location signals, corresponding to the spatial location information of the left, middle and right segments, respectively. The location signals are generated into vector representations using a set location encoding method. The three sets of location signals are concatenated with the corresponding segmented feature vectors in the channel dimension to generate a fused feature vector group.

[0033] The feature concatenation structure performs feature alignment on the three sets of fused feature vectors and performs concatenation operations in the sequence dimension to generate fused feature vectors.

[0034] Optionally, step four specifically includes:

[0035] Each image in the image dataset is simultaneously input into the region recognition subnetwork, letter recognition subnetwork, and digit recognition subnetwork;

[0036] After receiving the image, the region recognition subnetwork extracts candidate region information based on the backbone feature structure, performs region character determination operation in combination with the region location distribution features, and outputs the region code recognition result.

[0037] After receiving an image, the letter recognition subnetwork extracts character feature vectors based on the backbone feature structure and outputs letter recognition results through letter recognition operations.

[0038] After receiving an image, the digital recognition subnetwork extracts digital feature vectors based on the backbone feature structure and outputs digital recognition results through digital recognition operations.

[0039] The regional code recognition results, letter recognition results, and number recognition results are concatenated in a preset order to generate candidate license plate recognition results.

[0040] Optionally, step five specifically includes:

[0041] The legality of the candidate license plate recognition results is determined to identify whether there are illegal character combinations or missing structural units. Illegal character combinations include combinations of regional codes and letter codes that do not conform to the provincial coding rules, incorrect order of letter codes and number codes, non-letter characters in letter positions, non-numeric characters in numeric positions, invalid province abbreviations in the regional code, and license plate lengths that do not conform to national standards. Missing structural units include missing regional code characters, missing English letter characters, missing at least one numeric character, and the presence of consecutive blank recognition units or invalid symbol filler characters.

[0042] When it is determined that there are illegal character combinations or missing structural unit information in the candidate license plate recognition results, the path filtering process is initiated.

[0043] Based on the output area code recognition results, letter recognition results, and number recognition results in the candidate license plate recognition results, a path tree structure is constructed. The path tree structure establishes multi-level recognition nodes according to the character position, and the recognition nodes store the candidate recognition value set for the corresponding position.

[0044] Based on the preset character combination rules and license plate structure rules, a combined path traversal operation is performed on the path tree structure to filter out path nodes that do not meet the rule constraints and retain valid path branches that meet the structure conditions.

[0045] All valid path branches that meet the rule requirements are combined into a path sequence, and each path in the path sequence represents a possible arrangement of license plate characters; all combined paths that meet the structural rules are summarized to form a candidate path set.

[0046] Optionally, step six specifically includes:

[0047] Based on each group of paths in the candidate path set, it is determined whether there is missing structural unit information. The missing structural unit information includes missing regional code characters, missing English letter characters, missing at least one digit character, and the presence of blank recognition units or invalid symbol filling characters.

[0048] When missing structural unit information is detected in the candidate path, the reasoning and filling process is invoked. The reasoning and filling process includes performing missing character reasoning operations based on semantic structure rules and historical recognition data. The semantic structure rules include character position type constraints, structural combination order rules and length integrity requirements. The historical recognition data includes standard structural templates, historical license plate character combination frequency data and a mapping table of common misidentified symbols.

[0049] Based on the identified characters, structural positions, and contextual dependencies in the candidate paths, a set of candidate characters for inference at the missing positions is constructed. The structural fit and historical confidence of each candidate character at the corresponding position are calculated. The optimal character is selected to fill the structural missing position, generating a completion process path. All paths after inference filling are combined into a completion path set.

[0050] Optionally, step seven specifically includes:

[0051] Merge the candidate path set and the completed path set to generate a merged path set;

[0052] A multi-factor comprehensive scoring is performed on each path in the path merging set based on its structural integrity, character confidence distribution, and path confidence level to construct a path scoring sequence;

[0053] The path with the highest score in the path score sequence is selected as the license plate recognition result;

[0054] Other high-scoring paths in the path scoring sequence are used as confidence candidate paths, and the license plate recognition result and the set of confidence candidate paths are output.

[0055] Optionally, a deep learning-based license plate recognition system includes the following modules:

[0056] Image processing module: used to acquire license plate image data, perform size normalization, illumination equalization and edge enhancement processing, and generate image datasets;

[0057] Semantic structure module: used to perform semantic decomposition on image datasets, dividing them into region code units, letter units and number units, and constructing a semantic structure graph;

[0058] Model building module: used to build a multi-path character recognition model based on semantic structure graph. The multi-path character recognition model includes a region code recognition sub-network, a letter recognition sub-network and a number recognition sub-network with an improved PCNN model as the backbone feature structure.

[0059] The path generation module is used to input the image dataset into the region code recognition subnetwork, letter recognition subnetwork, and number recognition subnetwork, and output the region code recognition result, letter recognition result, and number recognition result, which are then concatenated to generate candidate license plate recognition results. If there are illegal character combinations or missing structural unit information, path filtering processing is performed to generate a candidate path set and a complete path set.

[0060] The results output module is used to perform a set merging operation on the candidate path set and the completed path set, and to construct a path scoring sequence based on structural integrity, character confidence distribution and path confidence level, and output the license plate recognition result and the set of confidence candidate paths.

[0061] The beneficial effects of this invention are:

[0062] This invention constructs a multi-path character recognition model and introduces an improved PCNN model as the backbone feature structure. Addressing issues such as character adhesion, semantic misalignment, and path interference in license plate character structures involving region codes, letters, and numbers, a semantic decomposition and structure graph construction mechanism is designed to achieve semantic alignment of character units. During the recognition stage, a multi-path branching recognition strategy is employed to extract region code recognition results, letter recognition results, and number recognition results separately, constructing candidate license plate recognition results. To address character misidentification and missing character issues, a path filtering and inference-based completion mechanism is introduced to generate a complete path set, effectively enhancing the model's robustness to structural defects, illegal combinations, and ambiguous characters in complex license plate images, and improving the accuracy of license plate structural integrity recognition. By fusing confidence information to output the final recognition result and a set of confidence-based candidate paths, structural perception, fault-tolerant inference, and semantic recovery are achieved for license plate recognition under various character structure combinations, improving the practicality and intelligence of the license plate recognition system in complex traffic environments. Attached Figure Description

[0063] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0064] Figure 1 This is a flowchart of a deep learning-based license plate recognition method proposed in this invention;

[0065] Figure 2 This is a structural diagram of the multi-path character recognition model proposed in this invention;

[0066] Figure 3 This is a block diagram of a deep learning-based license plate recognition system proposed in this invention. Detailed Implementation

[0067] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0068] refer to Figure 1-2 A deep learning-based license plate recognition method includes the following steps:

[0069] Step 1: Collect license plate image data, perform size normalization, illumination equalization, and edge enhancement processing to generate an image dataset;

[0070] Step 2: Perform semantic decomposition on the image dataset, dividing it into region code units, letter units, and number units, and construct a semantic structure graph;

[0071] Step 3: Construct a multi-path character recognition model based on the semantic structure graph; the multi-path character recognition model includes a region code recognition sub-network, a letter recognition sub-network, and a number recognition sub-network with an improved PCNN model as the backbone feature structure; the improved PCNN model includes a segmented convolutional structure, a positional information fusion structure, and a feature concatenation structure;

[0072] Step 4: Input the image dataset into the region recognition subnetwork, letter recognition subnetwork and number recognition subnetwork respectively, and output the region code recognition result, letter recognition result and number recognition result. Concatenate the results in the preset order to generate candidate license plate recognition results.

[0073] Step 5: If there are illegal character combinations or missing structural unit information in the candidate license plate recognition results, then perform path filtering processing to generate a candidate path set;

[0074] Step 6: Perform inference and imputation processing on the candidate path set to generate a complete path set;

[0075] Step 7: Merge the candidate path set and the completed path set, and output the license plate recognition result and the set of confident alternative paths.

[0076] This implementation improves the accuracy and robustness of license plate recognition by constructing a deep learning-based multi-path character recognition model. By acquiring license plate image data and performing size standardization, illumination equalization, and edge enhancement processing, image quality is improved, ensuring the input image possesses good visual characteristics. Semantic decomposition is performed, dividing the image into region code units, letter units, and number units, and a semantic structure graph is constructed, providing clear structural prior information for the recognition process. Furthermore, a multi-path character recognition model incorporating an improved PCNN model is constructed. Segmented convolutional structures enhance local feature extraction capabilities, and a positional information fusion structure and feature concatenation structure are combined to achieve multi-unit collaborative recognition, effectively enhancing the model's ability to distinguish license plate characters. Simultaneously, by inputting image data into the region code recognition subnetwork, letter recognition subnetwork, and number recognition subnetwork respectively, multiple recognition results are output and sequentially concatenated to generate candidate license plate recognition results. If there are illegal character combinations or missing structural units in the candidate results, a path filtering process is performed to generate a candidate path set. Then, a complete path set is constructed through reasoning and filling processes. Finally, the candidate paths and complete paths are merged to output a stable and reliable license plate recognition result and a corresponding set of confident alternative paths. This achieves fault-tolerant recognition and result enhancement for incomplete license plate information, and improves the overall recognition effect and adaptability of the system.

[0077] In this embodiment, step one specifically includes:

[0078] Color images including the license plate area are acquired. High-resolution imaging equipment is used to capture images of vehicles under different lighting, angles and background conditions to ensure that the standard license plate structure area is completely included in the images.

[0079] The acquired color images are subjected to size standardization processing. The height and width of the images are adjusted uniformly according to the set resolution parameters to give the images a uniform input scale, which facilitates batch calculation and structural alignment in subsequent model processing.

[0080] The image after size standardization is processed by illumination equalization. Based on the statistical features of the image histogram, a combination of brightness equalization and contrast enhancement is used to adjust the distribution of bright and dark areas in the image, alleviate the problem of uneven illumination, and improve the visual clarity of the license plate character area.

[0081] Edge processing is performed on the image after illumination equalization. An edge contour map is constructed based on the image gradient information. The gradient values ​​in the horizontal and vertical directions are extracted by the image derivative estimation method. Bidirectional filtering is performed on the edge region to suppress high-frequency noise interference. Finally, contour information with clear structure is extracted to form an image dataset.

[0082] In this embodiment, step two specifically includes:

[0083] Character region localization processing is performed on the image dataset. Based on the morphological features and geometric parameters of the license plate outline region, the character arrangement region is extracted. The boundary of the main license plate region is identified by combining color clustering algorithm with shape prior rules.

[0084] The character block is divided within the character arrangement area. The character boundaries are identified based on the statistical results of pixel projection density in the horizontal and vertical directions. The interval position between characters is determined by the sliding window and local extremum detection method, and the character area is accurately segmented.

[0085] The character type recognition process is performed on the segmented character regions. The character structure template comparison and deep feature matching methods are used to determine the type of the character. The characters are classified into region code units, letter units and number units in combination with standard encoding rules. During the processing, a character recognition auxiliary network is called to improve the classification robustness.

[0086] A semantic structure diagram is constructed based on the arrangement order of each character unit. The process of constructing the semantic structure diagram includes:

[0087] Construct a character node set, which includes nodes generated from the identified character units. Each node contains information such as character content, character type, and position index in the license plate.

[0088] A semantic edge set is constructed, which includes three types: sequential edges, structural edges, and context edges. Sequential edges represent the order of characters. Structural edges are used to connect regional code characters and first letter characters to represent the subordinate relationship between administrative regions and licensing authorities. Context edges are used to establish the association between a character node and its multiple adjacent characters before and after it to enhance the model's ability to perceive context dependencies.

[0089] Each character node is assigned a node attribute vector, which includes information such as character category label, image feature encoding, position encoding, and recognition confidence. The image feature encoding is composed of local character features extracted by the improved PCNN model.

[0090] Each semantic edge is assigned an edge attribute vector, which includes semantic weight, edge type identifier and structural importance score. The semantic weight can be dynamically updated through the structural learning process to reflect the real logical dependencies between characters.

[0091] The node set, semantic edge set, node attributes and edge attributes are integrated into a structured graph data structure, which is then input into a structural modeling network for semantic relationship modeling processing. The output is a semantic structure graph used to guide subsequent character recognition optimization.

[0092] The final result is a semantic structure graph that expresses the logical structure of the license plate. This graph serves as a structural constraint input to the character recognition optimization module, assisting in the accurate recognition of characters and the verification of the rationality of their arrangement.

[0093] This step uses a combined processing approach of region localization, character segmentation, type recognition, and structural modeling to achieve accurate analysis of the spatial distribution of license plate characters, thereby improving the structural expressiveness of license plate content at both the logical and visual levels.

[0094] In this embodiment, step three specifically includes:

[0095] Read the character category information and arrangement order information in the semantic structure graph, and based on the category label and position information of each character node in the structure graph, identify the character block index range corresponding to the region code unit, letter unit and number unit respectively;

[0096] Based on the character category classification results, set up regional code recognition path, letter recognition path and number recognition path, and configure independent recognition channels for different categories to ensure that the multi-path model structure has category-specific recognition capabilities and feature adaptability;

[0097] The input interface is configured for each recognition path based on character category and position index. The corresponding character region is extracted from the image dataset as the input content of the recognition path. The position mapping mechanism is used to achieve accurate correspondence between character image and model channel.

[0098] For each recognition path, an improved PCNN model is called to construct the recognition structure. The improved PCNN model includes multi-scale convolutional kernels, cross-channel fusion layers, and position embedding structures. By setting the model input dimension, convolution parameters, and channel mapping method, the model's adaptability to character morphological changes and its ability to express spatial features are improved.

[0099] After all recognition paths are constructed, a multi-path character recognition model is formed. The multi-path character recognition model includes a regional code recognition sub-network, a letter recognition sub-network, and a number recognition sub-network. The structure of each sub-network is customized according to the character category, and each performs an independent feature extraction and category determination process, ultimately achieving high-precision recognition output of license plate characters.

[0100] This step establishes a character category-driven multi-path recognition model, structurally separating the recognition channel from the character semantics, which significantly improves the model's recognition accuracy and robustness in handling character diversity and complex background interference.

[0101] In this embodiment, the improved PCNN model includes a segmented convolutional structure, a positional information fusion structure, and a feature concatenation structure, specifically:

[0102] The segmented convolutional structure receives a character image as input. A fixed-length receptive field window is constructed based on the center coordinates of the character image, and the character image is divided into left, middle, and right segments at equal intervals along the horizontal direction. A convolution operation is performed within each segmented region, using the same kernel size, stride, and padding parameters to maintain consistency in the spatial dimensions of the convolution output. Local feature maps are extracted from the convolution results of each region, and these feature maps are flattened to generate three sets of segmented feature vectors. These three sets of feature vectors represent the local structural information of the character image in the left, middle, and right regions, respectively.

[0103] The position information fusion structure constructs position signals for the left, middle, and right segments respectively. The position signals are generated using a set position encoding method. Three sets of position encoding vectors are obtained by performing vector mapping operations on preset position indices. The three sets of position encoding vectors are concatenated with the corresponding segment feature vectors in the channel dimension to obtain three sets of fused feature vectors. During the concatenation process, channel expansion alignment processing is performed to ensure that the position information has sufficient expression space in the feature representation, thereby enhancing the model's ability to perceive the positional differences between the left and right structural parts of the character.

[0104] The feature concatenation structure performs feature alignment on the three sets of fused feature vectors, performs linear transformation on the feature dimensions to maintain consistent length, and performs concatenation operation on the sequence dimension. The connection order is left segment fused vector, middle segment fused vector and right segment fused vector, finally generating a fused feature vector with a unified format. The fused feature vector serves as the input of the subsequent character discrimination module, and has a clear spatial structure order and positional information to enhance feature expression capabilities.

[0105] This step, by constructing a structured segmented convolution and positional information fusion mechanism, enables the improved PCNN model to capture both local details and global positional information during character structure extraction, significantly improving the robustness and accuracy of character recognition against irregular shapes, deformation interference, and positional shifts.

[0106] In this embodiment, step four specifically includes:

[0107] Each image in the image dataset is simultaneously input into the region code recognition subnetwork, the letter recognition subnetwork, and the number recognition subnetwork;

[0108] After receiving the image, the region code recognition subnetwork extracts candidate region information based on the backbone feature structure; performs feature encoding operation to extract regional spatial distribution features and character edge features; calls the region recognition module to perform regional character determination operation, uses the supervision signal obtained by fitting regional character labels and spatial location information as the optimization basis, and outputs the region code recognition result.

[0109] After receiving the image, the letter recognition subnetwork extracts the character image feature vector based on the backbone feature structure; it then performs letter classification processing, using a classification function obtained by fitting the character image and character label to perform letter recognition operation and outputs the letter recognition result.

[0110] After receiving the image, the digit recognition subnetwork extracts the digit character image feature vector based on the backbone feature structure; it then executes the digit recognition module processing, using a recognition function obtained by fitting the digit image features with the numerical label to perform the digit recognition operation and output the digit recognition result.

[0111] The region code recognition result, letter recognition result, and number recognition result are concatenated according to the preset region code-letter-number order in the character sequence structure to generate candidate license plate recognition results.

[0112] This implementation method, through a multi-sub-network recognition approach after structural division, improves the recognition accuracy of different character types while maintaining the consistency of character structure, and enhances the stability and robustness of license plate recognition in complex image scenarios.

[0113] In this embodiment, step five specifically includes:

[0114] The legality of the candidate license plate recognition results is determined to identify any illegal character combinations or missing structural units. Illegal character combinations include combinations of regional codes and letter codes that do not conform to provincial coding rules, incorrect order of letter codes and number codes, non-letter characters in letter positions, non-numeric characters in numeric positions, invalid province abbreviations in regional codes, and license plate lengths that do not conform to national standards. Missing structural units include missing regional code characters, missing English letter characters, missing at least one numeric character, and the presence of consecutive blank recognition units or invalid symbol filler characters.

[0115] When it is determined that there are illegal character combinations or missing structural unit information in the candidate license plate recognition results, the path filtering process is initiated; the path filtering process includes path tree construction, rule-constrained path elimination, and path sequence generation.

[0116] Based on the output area code recognition results, letter recognition results, and number recognition results in the candidate license plate recognition results, a path tree structure is constructed. The path tree structure establishes multi-level recognition nodes according to the character position. Each level of recognition node corresponds to a set position of the license plate character. The recognition node stores the candidate recognition value set for the corresponding position. The recognition value set is composed of the top K candidate characters selected after sorting the output confidence results of the character recognition model according to the high confidence. K is an adjustable parameter, usually with a value of 3 to 5.

[0117] Based on the preset character combination rules and license plate structure rules, a combined path traversal operation is performed on the path tree structure. The combined path traversal operation adopts a depth-first traversal strategy, recursively expanding the path branches from the root node of the path tree. During the traversal, it checks in real time whether the current path meets the character combination specifications. The character combination specifications include that the first character of the area code is limited to the provincial abbreviation character in the administrative region abbreviation, the second character is an English letter character and meets the city code rules, and subsequent characters are arranged in the form of a combination of one uppercase English letter plus four or five digits. If, during the expansion of any node, it is found that the character type does not match the current structure position, the number of characters is insufficient or excessive, or the combination order is inconsistent with the standard, the current path expansion is terminated and the node is marked as an invalid path node.

[0118] All valid path branches that meet the structural conditions are retained and combined to form a path sequence. Each path in the path sequence represents a possible arrangement of license plate characters. A path confidence weighted scoring operation is performed on all path sequences. The path confidence score is the weighted average of the character recognition confidence values ​​in each path. The weights are adjusted based on the importance of the character's location and the historical recognition accuracy. The confidence score is calculated by fitting an objective function constructed by combining offline training sample data and structural rules. The least squares regression method is used to determine the weighting coefficient matrix and embed it into the scoring model.

[0119] The path confidence scores are sorted, and path sequences with scores higher than a set threshold are retained as candidate paths with valid structure and reliable confidence. All paths that pass the screening are summarized to form a candidate path set, which serves as an important input for subsequent character result reordering and final license plate output.

[0120] This step constructs a path tree structure under structural constraints and combines character validity rules and confidence scoring mechanisms to effectively eliminate structurally conflicting paths and paths with low confidence, thereby enhancing the structural correctness and reliability of multi-path license plate recognition. This provides accurate and stable candidate path inputs for final result optimization, improving overall recognition accuracy and reliability.

[0121] In this embodiment, step six specifically includes:

[0122] Based on each group of paths in the candidate path set, determine whether there is missing structural unit information; the missing structural unit information includes missing regional code characters, missing English letter characters, missing at least one digit character, and the presence of blank recognition units or invalid symbol filling characters.

[0123] When missing structural unit information is detected in a candidate path, the inference and filling process is invoked. The inference and filling process includes performing missing character inference operations based on semantic structure rules and historical recognition data. The semantic structure rules include character position type constraints, structural combination order rules, and length integrity requirements. The historical recognition data includes standard structural templates, historical license plate character combination frequency data, and a mapping table of common misidentified symbols.

[0124] Based on the identified characters, structural positions, and contextual dependencies in the candidate paths, a set of candidate characters for inference at the missing positions is constructed. This set generates a collection of candidate characters according to character type derivation rules, with each character type bound to structural position information, a range of possible characters, and legality constraints. After constructing the candidate character set, each missing character position is scored based on structural fit and historical confidence indices. Structural fit represents the degree of matching between the candidate character and the current path structure, while historical confidence represents the historical probability weight of the candidate character appearing at the same structural position. The scoring results are combined using a weighted function to obtain a candidate character priority index.

[0125] After prioritizing the candidate characters, the character with the highest score is selected to fill the current missing position, thus constructing the completion process path. The completion process path inherits the identified character information of the original path and records the inference filling position and corresponding character.

[0126] All paths processed by inference are combined into a complete path set; each path in the complete path set satisfies the semantic structure rules and character integrity requirements, and has the potential to be transformed from a candidate path into the final license plate recognition result.

[0127] This implementation method constructs a missing character reasoning mechanism based on semantic rule constraints and historical data assistance to complete the structure of incomplete license plate paths, thereby improving the completeness and rationality of the final license plate recognition result.

[0128] In this embodiment, step seven specifically includes:

[0129] Perform a set merging operation on the candidate path set and the completed path set to generate a merged path set. The set merging operation retains all original candidate paths and completed paths generated during the reasoning and completion process, ensuring that all possible license plate structure paths participate in subsequent screening and scoring.

[0130] For each path in the path merging set, three scoring factors are extracted: path structural integrity, character confidence distribution, and path confidence level. Path structural integrity is measured by comparing the consistency between the character sequence and the standard license plate structure template. If the character sequence simultaneously satisfies the rules of total number of characters, character category order, and spatial distribution, the path structural integrity score is set to the highest; otherwise, the score is reduced based on the number of inconsistencies. Character confidence distribution constructs sequence features based on the recognition confidence value of each character node in the path, using standard deviation, maximum-minimum difference, and average confidence level as key statistical indicators to characterize the internal recognition stability of the path. The path confidence level constructs an evaluation function based on the path generation method, network prediction strength, and the output results of the auxiliary scoring module, assigning different level labels to manually labeled paths, network main paths, pseudo-labeled paths, and graph structure completion paths, and associating them with corresponding confidence coefficients to form a path confidence level vector.

[0131] The path structure integrity score, character confidence distribution features, and path confidence level vector are input into the multi-factor comprehensive scoring model to construct a path scoring sequence. The multi-factor comprehensive scoring model constructs a scoring label system based on the labeled data, uses the least square error as the loss function for model training, introduces a linear combination structure to fuse various scoring factors, and determines the contribution ratio of different factors in the final score through a weight learning mechanism, outputting the scoring result corresponding to each path.

[0132] The path with the highest score in the path scoring sequence is selected as the final license plate recognition result. Other high-scoring paths in the path scoring sequence, excluding the highest-scoring path, are filtered according to the preset confidence interval threshold and path content similarity standard. If a candidate path score is higher than the set threshold and its consistency with the final license plate recognition result in character category distribution, character arrangement position and character confidence is higher than the similarity judgment standard, then the path is included in the set of confidence candidate paths.

[0133] Output the license plate recognition result and a set of confidence alternative paths; the recognition result is used for structured output, and the set of confidence alternative paths provides multi-path reference information.

[0134] This implementation improves the scientific nature of path assessment and the accuracy of recognition results by establishing a multi-factor scoring system based on structural integrity, character confidence distribution, and path confidence level. In real-world complex environments, it significantly enhances the anti-interference capability and decision robustness of the license plate recognition system.

[0135] refer to Figure 3 It includes the following modules:

[0136] Image processing module: used to acquire license plate image data, perform size normalization, illumination equalization and edge enhancement processing, and generate image datasets;

[0137] Semantic structure module: used to perform semantic decomposition on image datasets, dividing them into region code units, letter units and number units, and constructing a semantic structure graph;

[0138] Model building module: used to build a multi-path character recognition model based on semantic structure graph. The multi-path character recognition model includes a region code recognition sub-network, a letter recognition sub-network and a number recognition sub-network with an improved PCNN model as the backbone feature structure.

[0139] The path generation module is used to input the image dataset into the region code recognition subnetwork, letter recognition subnetwork, and number recognition subnetwork, and output the region code recognition result, letter recognition result, and number recognition result, which are then concatenated to generate candidate license plate recognition results. If there are illegal character combinations or missing structural unit information, path filtering processing is performed to generate a candidate path set and a complete path set.

[0140] The results output module is used to perform a set merging operation on the candidate path set and the completed path set, and to construct a path scoring sequence based on structural integrity, character confidence distribution and path confidence level, and output the license plate recognition result and the set of confidence candidate paths.

[0141] Example 1:

[0142] To verify the feasibility of this invention in practice, it was applied to a smart checkpoint system under a big data command platform. This platform deployed 328 video checkpoint devices in the provincial capital and several counties, collecting vehicle flow information entering and exiting main roads around the clock. In actual operation, due to changes in lighting, weather conditions, camera angle offsets, and the influence of complex fonts, existing license plate recognition models suffer from problems such as characters sticking together and being unable to be separated, disordered letter and number recognition order, and incomplete structure. Especially at night and in rainy or foggy conditions, the character misrecognition rate remains high, seriously affecting the reliability of law enforcement evidence collection.

[0143] This invention proposes a deep learning-based license plate recognition method and system. It employs a multi-path character recognition model, utilizes an improved PCNN network for feature extraction, and introduces a semantic decomposition structure graph and path reasoning completion mechanism to effectively alleviate the problems of character structure misalignment, incomplete recognition, and insufficient fault tolerance in traditional models. In platform deployment, the research team selected three key traffic nodes (the south entrance of the urban elevated road, the west entrance of the ring road, and the north gate of the industrial park) for a three-month field test. The system collected over 2.8 million license plate images as test samples, covering various complex environmental conditions such as daytime, nighttime, rainy days, and backlighting. The system was compared with the currently used commercial vehicle license plate recognition system A to evaluate indicators such as recognition accuracy, structural integrity rate, fault tolerance recovery rate, and false recognition rate.

[0144] In actual deployment, the recognition system is integrated into the existing checkpoint platform. Images are captured by the original cameras and then fed into the model flow of this invention. Each image is categorized into three paths during processing: region code recognition, letter recognition, and number recognition. A structure diagram and confidence screening mechanism are established to address character blurring, occlusion, and abnormal combinations. In images with incomplete recognition, the system automatically constructs candidate paths and performs inference completion operations, thereby enhancing recognition stability. By annotating and verifying each frame of the captured images, a human gold standard is constructed, and comparison data is collected daily to generate detailed statistics.

[0145] To further verify the system performance, the following table compares the recognition accuracy under complex weather conditions during the three-month operation (Table 1) and the recognition fault tolerance and recovery capability (Table 2). All data are based on the comparison and statistics of manually checked samples, with a total sample size of over 100,000 images to ensure authenticity and representativeness.

[0146] Table 1 Comparison of Recognition Accuracy under Complex Weather Conditions

[0147]

[0148] Table 1 shows a comparison of the recognition accuracy of the system of this invention and the current commercial system under various complex weather conditions. The data clearly shows that under ideal conditions such as sunny daytime, the difference between the system of this invention and the commercial system is small, with an improvement of only about 1.42%. However, under non-ideal conditions such as nighttime without lights, rainy / foggy weather, backlighting, and snowy nights, the recognition accuracy of the system of this invention is significantly better than that of the commercial system, with a maximum improvement of 9.66%. Especially under backlighting and rainy / foggy conditions, conventional models are prone to character inversion and local blurring, while the system of this invention, through multi-path recognition and structure perception mechanisms, can effectively preserve semantic information, achieving more accurate character recognition. At the same time, the average character integrity rate remains above 91%, indicating that the system of this invention not only has high recognition accuracy but also stronger ability to restore character structure, resulting in more stable and reliable overall performance.

[0149] Table 2 Comparison of Fault Tolerance and Recovery Capabilities of License Plate Structure Recognition

[0150]

[0151] Table 2 reflects the fault tolerance and recovery capabilities of the system of this invention in character recognition anomaly scenarios, covering four typical problem scenarios: character adhesion, character missingness, disordered order, and occlusion. The results show that the recovery rate of the system of this invention exceeds 88% in all four scenarios, with a recovery rate as high as 94.12% for disordered character order, significantly better than the 79.86% of commercial systems. Furthermore, it also exhibits an overwhelming advantage in the success rate of fuzzy character reasoning, averaging more than 12% higher than commercial systems, indicating that the present invention has stronger reasoning and completion capabilities when handling abnormal structures such as fuzzy, abnormal, and illegal combinations. Regarding the success rate of illegal combination repair, the present invention relies on path filtering and semantic distribution maps to effectively correct disordered characters, with a repair success rate generally exceeding 85%. Overall, the system of this invention has significant technical advantages in handling various complex and non-standard license plate recognition problems in real-world traffic scenarios and has good engineering promotion value.

[0152] This embodiment verifies the recognition accuracy and stability of the present invention in complex real-world scenarios, especially demonstrating strong fault tolerance, recovery, and semantic reasoning capabilities under conditions such as rain, fog, nighttime, and blurred / occluded conditions. Compared to traditional models, the present invention significantly improves the accuracy and completeness of structural recognition, providing stronger protection for intelligent transportation and law enforcement monitoring.

[0153] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep learning-based license plate recognition method, characterized in that, The method comprises the following steps: Step 1: Collecting license plate image data, performing size standardization processing, light balance processing and edge enhancement processing to generate an image data set; Step 2: Performing semantic decomposition operation on the image data set, dividing into regional code unit, letter unit and number unit, and constructing semantic structure graph; Step 3: Reading character category information and arrangement order information in the semantic structure graph, identifying regional code unit, letter unit and number unit; According to the character category division result, set the regional code recognition path, letter recognition path and number recognition path; Based on the character category and position index, configure the input interface for each recognition path, and specify the corresponding character region in the image data set as the input content; For each recognition path, call the improved PCNN model to construct the recognition structure, set the input dimension, convolution parameter and channel mapping mode of the improved PCNN model; After completing the construction of all recognition paths, a multi-path character recognition model is formed; the multi-path character recognition model comprises a regional code recognition subnetwork, a letter recognition subnetwork and a number recognition subnetwork; The improved PCNN model comprises a segmented convolution structure, a position information fusion structure and a feature splicing structure, specifically: The segmented convolution structure receives character image input, constructs a fixed-length window according to the character center position, and divides the character image into three regions of left segment, middle segment and right segment along the horizontal direction; convolution operation is performed on the left segment, middle segment and right segment regions respectively, the same convolution kernel size and step parameter are adopted, local feature maps are extracted independently for each region, and three groups of segmented feature vectors are output; The position information fusion structure constructs three groups of position signals corresponding to the spatial position information of the left segment, middle segment and right segment, and the position signals are generated in the form of vector by setting the position encoding method; the three groups of position signals and the corresponding segmented feature vectors are spliced in the channel dimension to generate a group of fusion feature vectors; The feature splicing structure performs feature alignment processing on the three groups of fusion feature vectors, and connects them in the sequence dimension to generate a fusion feature vector; Step 4: Input the image data set into the regional recognition subnetwork, the letter recognition subnetwork and the number recognition subnetwork respectively, output the regional code recognition result, the letter recognition result and the number recognition result, and splice them according to the preset result order to generate the candidate license plate recognition result; Step 5: If there are illegal character combinations or missing structure unit information in the candidate license plate recognition result, perform path screening processing to generate a candidate path set; Step 6: Perform inference filling processing on the candidate path set to generate a complete path set; Step 7: Merge the candidate path set and the complete path set to output the license plate recognition result and the confidence candidate path set. 2.The method of claim 1, wherein, The step 1 is specifically: Collecting a color image including a license plate region; Performing size standardization processing on the collected color image, adjusting the height and width of the image according to the set resolution; Performing light balance processing on the image after size standardization processing, adjusting the brightness distribution and contrast distribution of the image; Edge processing is performed on the image after the illumination balancing, edge contours are extracted based on image gradient information, and edge regions are processed by applying a filtering method to form an image dataset. 3.The method of claim 2, wherein, The step two is specifically: Character region positioning processing is performed on the image dataset, and a character arrangement region is extracted according to a license plate boundary in the image; Character block division is performed in the character arrangement region, and character boundaries are identified according to horizontal and vertical projection densities; Character type identification processing is performed on the divided character regions, and the characters are divided into area code units, letter units, and number units according to coding specifications; A semantic structure diagram is established according to the arrangement order of each character unit; the semantic structure diagram includes character categories, character position information, and arrangement relationships. 4.The method of claim 3, wherein, The step four is specifically: Each image in the image dataset is simultaneously input to a region recognition subnetwork, a letter recognition subnetwork, and a number recognition subnetwork; After receiving the image, the region recognition subnetwork extracts region candidate information based on a backbone feature structure, performs region character determination operations in combination with region position distribution characteristics, and outputs region code recognition results; After receiving the image, the letter recognition subnetwork extracts character feature vectors based on the backbone feature structure, and outputs letter recognition results through letter recognition operations; After receiving the image, the number recognition subnetwork extracts number feature vectors based on the backbone feature structure, and outputs number recognition results through number recognition operations; The region code recognition results, the letter recognition results, and the number recognition results are spliced in a preset order to generate a candidate license plate recognition result. 5.The method of claim 4, wherein, The step five is specifically: A legality determination operation is performed on the candidate license plate recognition result to determine whether there are illegal character combinations or structure unit missing information in the candidate license plate recognition result; the illegal character combinations include that a region code and a letter code combination does not comply with a province coding rule, a letter code and a number code order are incorrect, a non-letter character appears in a letter position, a non-number character appears in a number position, an invalid province abbreviation appears in a region code, and a total length of the license plate does not comply with a national standard; the structure unit missing information includes missing region code characters, missing English letter characters, missing at least one digit character, existing continuous blank recognition units, or invalid symbol filling characters; When it is determined that there are illegal character combinations or structure unit missing information in the candidate license plate recognition result, a path screening processing procedure is started; Based on the region code recognition results, the letter recognition results, and the number recognition results that have been output in the candidate license plate recognition result, a path tree structure is constructed, the path tree structure establishes multiple levels of recognition nodes according to character positions, and a candidate recognition value set corresponding to a position is stored in a recognition node; According to a preset character combination rule and a license plate structure formation rule, a combination path traversal operation is performed on the path tree structure, path nodes that do not satisfy the rule constraints are screened out, and valid path branches that satisfy the structure conditions are retained; All valid path branches that satisfy the rule requirements are combined into a path sequence, each group of paths in the path sequence represents a possible license plate character arrangement, and all combination paths that satisfy the structure rule are collected to form a candidate path set. 6.The method of claim 5, wherein, The step six is specifically: judging whether there is structure unit missing information in each set of paths in the candidate path set, the structure unit missing information including missing regional code characters, missing English letter characters, missing at least one digit character, existing blank recognition unit or invalid symbol filling character; when detecting that there is structure unit missing information in the candidate path, calling an inference filling processing procedure; the inference filling processing procedure includes performing a missing character inference operation based on semantic structure rules and historical recognition data, the semantic structure rules including character position type constraints, structure combination order rules and length integrity requirements, and the historical recognition data including standard structure templates, historical license plate character combination frequency data and common misrecognized symbol mapping comparison tables; based on the recognized characters, structure positions and context dependency relations in the candidate path, constructing an inference candidate character set for the missing position, calculating the structure adaptation degree and historical confidence of each candidate character at the corresponding position, selecting the optimal character to fill into the structure missing position, and generating a completed process path; combining all paths after performing the inference filling processing into a completed path set.

7. The deep learning-based license plate recognition method of claim 6, wherein, The step seven is specifically: performing set merging on the candidate path set and the completed path set to generate a path merging set; performing multi-factor comprehensive scoring based on the structure integrity, character confidence distribution and path confidence level of each path in the path merging set to construct a path score sequence; selecting the path with the highest score from the path score sequence as the license plate recognition result; outputting the license plate recognition result and the set of confidence alternative paths.

8. A deep learning-based license plate recognition system, performing a deep learning-based license plate recognition method according to any one of claims 1 to 7, characterized in that, The method comprises the following modules: an image processing module, used for collecting license plate image data, performing size standardization processing, light balance processing and edge enhancement processing, and generating an image data set; a semantic structure module, used for performing semantic decomposition operation on the image data set, dividing into regional code units, letter units and number units, and constructing a semantic structure graph; a model construction module, used for constructing a multi-path character recognition model based on the semantic structure graph, the multi-path character recognition model including a regional code recognition subnetwork, a letter recognition subnetwork and a number recognition subnetwork with an improved PCNN model as the backbone feature structure; a path generation module, used for inputting the image data set into the regional code recognition subnetwork, the letter recognition subnetwork and the number recognition subnetwork, outputting regional code recognition results, letter recognition results and number recognition results, and splicing to generate a candidate license plate recognition result; if there is illegal character combination or structure unit missing information, performing path screening processing to generate a candidate path set and a completed path set; a result output module, used for performing set merging operation on the candidate path set and the completed path set, and constructing a path score sequence based on structure integrity, character confidence distribution and path confidence level, and outputting the license plate recognition result and the set of confidence alternative paths.

Images