A method for license plate enlargement recognition based on overall and character region detection

Abstract

This invention discloses a method for recognizing enlarged license plate numbers based on overall and character region detection. The method includes: constructing an improved DBNet overall region detection model, an improved CRAFT character region detection model, and an improved SVTR text recognition model using a license plate enlarged number training set and a publicly available scene text dataset, respectively; detecting the enlarged number test set data using the improved DBNet and CRAFT models to obtain the overall region of the enlarged number and all character text regions, then merging them using a connectivity module to obtain the overall location of the enlarged number and the locations of other text regions that may belong to the same text line; then recognizing the text content in the image using the improved SVTR model; and finally, filtering, merging, and verifying the text content in the image using a pre-set verification module, outputting the final verified license plate number. This invention achieves high-precision recognition of enlarged license plate numbers.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation, and more specifically, to a method for license plate enlargement recognition based on overall and character region detection. Background Technology

[0002] License plate numbers are the most important identity information and unique identifier of a vehicle. Building intelligent urban transportation systems requires accurate and rapid detection and recognition of license plates. While existing detection and recognition technologies for most blue, yellow, and new energy vehicle license plates in China are relatively mature, these methods are prone to misdetection and missed detection of medium and large vehicles such as trucks and dump trucks due to factors such as shooting angle, lighting conditions, license plate position, and license plate dirt. For example, the cargo box of a truck can easily obstruct the detection and recognition of the rear license plate by the checkpoint camera from a high-angle view. According to Article 13 of the "Regulations for the Implementation of the Road Traffic Safety Law of the People's Republic of China," heavy and medium-sized trucks and their trailers, tractors and their trailers, must have enlarged license plate numbers (referred to as enlarged license plate numbers) painted on the rear of their bodies or cargo boxes. Since the size and position of enlarged license plate numbers are more obvious than standard license plates, detecting and recognizing these numbers helps to better obtain vehicle identity information, thereby reducing misdetection and missed detection.

[0003] In the field of intelligent transportation, standard license plates have prescribed dimensions, colors, prominent boundaries, and character styles. Existing technologies rely on these characteristics for location, then obtain license plate information by identifying segmented characters or the entire sequence within the located area. Current license plate detection and recognition methods primarily utilize deep learning, with mainstream open-source systems such as EasyPR and HyperLPR, which have been widely applied in practical problems. However, existing technologies only target standard license plates. Enlarged license plates lack these prescribed characteristics; for example, the color of the enlarged license plate is not fixed, the character spacing is large and varied, and there may be other noise interference such as advertisements on the vehicle body. This leads to situations where general license plate detection and recognition methods and scene text identification methods cannot accurately locate the license plate or provide incomplete location information. Furthermore, due to small data sample sizes and poor scene-specific performance, how to more accurately determine the complete area of ​​the enlarged license plate within the vehicle body, solve the problem of inconsistent character spacing, and accurately identify license plate information from the location bounding box while eliminating false detections remain urgent problems to be solved.

[0004] Therefore, it is necessary to make targeted improvements to address the aforementioned problems in the existing technology. The detection and recognition method based on the whole and character regions of this invention can more effectively achieve license plate recognition for enlarged license plate numbers, thereby providing further technical support for the identification of medium and large vehicles in the field of intelligent transportation.

[0005] Chinese invention application No. 202011532814.1 discloses "A method for detecting and recognizing enlarged license plate numbers". The technical solution is as follows: the area where the enlarged license plate number is located is detected and located to obtain a sample image of the original enlarged license plate number; a recognition network based on deep convolution is used to recognize the characters of the enlarged license plate number; during the training phase, the sample image of the original enlarged license plate number is expanded to obtain a training sample set; then, a recognition network is constructed and features are extracted from the actual enlarged license plate number image to obtain the final text recognition result. Summary of the Invention

[0006] To address the insufficient accuracy of existing license plate enlargement recognition technologies, this invention provides a license plate enlargement recognition method based on overall and character region detection. The technical solution adopted by this invention is as follows:

[0007] A method for license plate enlargement recognition based on overall and character region detection, the method comprising:

[0008] A license plate enlargement dataset and a scene text public dataset are constructed respectively, wherein the license plate enlargement dataset includes a license plate enlargement training set and a license plate enlargement test set;

[0009] Improved DBNet overall region detection model, improved CRAFT character region detection model, and improved SVTR text recognition model were constructed using the license plate enlargement training set and the public scene text dataset, respectively.

[0010] The improved DBNet license plate enlargement area detection model was used to detect the license plate enlargement area in the test set data images and obtain the overall license plate enlargement area in the image.

[0011] The improved CRAFT license plate enlargement character region detection model was used to detect all character text regions in the license plate enlargement test set data images.

[0012] The pre-defined connectivity module merges the entire area of ​​the enlarged license plate number in the image with all character text areas in the image to obtain the overall location of the merged enlarged license plate number, as well as the location of other text areas that may belong to the same text line.

[0013] By improving the SVTR text recognition model, the image regions corresponding to the overall location of the enlarged license plate number and other text regions are identified, and the text content in the image is obtained.

[0014] The preset verification module filters and merges the text content in the image, verifies the text content according to the license plate number encoding rules, and outputs the final verified license plate number.

[0015] As a preferred embodiment, the publicly available scene text dataset includes an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, and a text recognition training set and a text recognition verification set.

[0016] As a preferred approach, methods for constructing improved DBNet global region detection models, improved CRAFT character region detection models, and improved SVTR text recognition models using license plate enlargement training sets and publicly available scene text datasets include:

[0017] The improved DBNet license plate enlargement area detection network is iteratively trained using the overall detection training set, and the model accuracy is verified using the overall detection validation set. The optimal model is saved and retrained using the enlargement training set, and finally the improved DBNet overall area detection model is obtained.

[0018] The improved CRAFT license plate enlargement character region detection network is iteratively trained using the character detection training set, and the accuracy of the model is verified using the character detection validation set. The optimal model is saved and retrained using the enlargement training set, and finally the improved CRAFT character region detection model is obtained.

[0019] The improved SVTR license plate enlargement text recognition network was iteratively trained using the text recognition training set, and the accuracy of the model was verified using the text recognition validation set. The optimal model was saved and retrained using the enlargement training set, and finally the improved SVTR text recognition model was obtained.

[0020] As a preferred embodiment, the improved DBNet overall region detection model includes a first feature extraction module, a feature fusion module, a first classification prediction module, a first binarization module, and an overall region output module connected in sequence.

[0021] The first feature extraction module is used to extract features from the license plate enlargement test set data images to obtain a first primary feature map;

[0022] The feature fusion module is used to obtain several feature maps of different sizes by passing the primary feature map through four 1×1 convolutions, then upsampling from top to bottom, and horizontally connecting the feature maps of the same size generated from bottom to top; the connected feature maps are then passed through four 3×3 convolutions to eliminate the aliasing effect of upsampling, and finally multi-scale feature representations are obtained and connected to obtain the first fused feature map.

[0023] The first classification prediction module is used to input the first fused feature map into a preset first volume integral class network to obtain a target probability map and a threshold map;

[0024] The first binarization module is used to generate a first approximate binary map by using a differentiable binarization method on the target probability map and the threshold map;

[0025] The overall region output module is used to obtain the vertex coordinates of the region polygon based on the first approximate binary image, and then restore these coordinates to the original image size to obtain the overall region of the enlarged license plate number.

[0026] The first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer.

[0027] The first stage layer includes 3 residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation.

[0028] The second stage layer includes three residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0029] The third stage layer includes 6 residual units. The structure of the first 4 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, and 1×1 convolution, batch normalization and hard swish activation function. The structure of the last 2 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, SE module, and 1×1 convolution, batch normalization and hard swish activation function.

[0030] The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation.

[0031] The feature fusion module is a feature pyramid network;

[0032] The first classification prediction module includes a first volume integral network, which includes a target probability map output module and a threshold map output module. The target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution.

[0033] The loss function of the improved DBNet global region detection model is the prediction probability loss. Binarization loss and target area distance loss The weighted sum, where, and Using cross-entropy loss, use Norm loss.

[0034] As a preferred embodiment, the first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer.

[0035] The first stage layer includes 3 residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation.

[0036] The second stage layer includes three residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0037] The third stage layer includes 6 residual units. The structure of the first 4 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, and 1×1 convolution, batch normalization and hard swish activation function. The structure of the last 2 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, SE module, and 1×1 convolution, batch normalization and hard swish activation function.

[0038] The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation.

[0039] The feature fusion module is a feature pyramid network;

[0040] The first classification prediction module includes a first volume integral network, which includes a target probability map output module and a threshold map output module. The target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution.

[0041] The loss function of the improved DBNet global region detection model is the prediction probability loss. Binarization loss and target area distance loss The weighted sum, where, and Using cross-entropy loss, use Norm loss.

[0042] As a preferred embodiment, the improved CRAFT license plate enlargement character region detection model includes a second feature extraction module, a deconvolution module, a second classification prediction module, a second binarization module, and a character region output module connected in sequence.

[0043] The second feature extraction module is used to extract features from the license plate enlargement test set data images to obtain a second primary feature map;

[0044] The deconvolution module is used to deconvolve the second primary feature map to obtain a deconvolution feature map.

[0045] The second classification prediction module is used to input the deconvolution feature map into a preset second-volume integral network to obtain the region score of the corresponding character and the connection score between characters;

[0046] The second binarization module is used to connect and expand the region scores and connection scores between characters using a fixed threshold method to generate a second approximate binary image;

[0047] The character region output module is used to obtain the vertex coordinates of the region polygon based on the second approximate binary image, and then restore these coordinates to the original image size to obtain the target character region output.

[0048] As a preferred embodiment, the second feature extraction module is VGG16_BN, which includes a first slice layer, a second slice layer, a third slice layer, a fourth slice layer, and a fifth slice layer.

[0049] The first slice layer includes 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0050] The second slice layer includes ReLU activation, max pooling, 3×3 convolution, batch normalization, and ReLU activation, 3×3 convolution, and batch normalization.

[0051] The third slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0052] The fourth slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0053] The fifth slice layer includes max pooling, 3×3 convolution, and 1×1 convolution;

[0054] The deconvolution module includes a first deconvolution module, a second deconvolution module, a third deconvolution module, and a fourth deconvolution module;

[0055] The fifth slice layer is connected to the output jump layer of the fourth slice layer, then connected to the output jump layer of the third slice layer through the first deconvolution module, then connected to the output jump layer of the second slice layer through the second deconvolution module, then connected to the output jump layer of the first slice layer through the third deconvolution module, and finally connected to the input of the fourth deconvolution module.

[0056] The second classification prediction module includes a second convolutional integral network, which includes three 3×3 convolutions and two 1×1 convolutions connected in sequence. Except for the last 1×1 convolution, the other convolutional layers are activated by ReLU.

[0057] The loss function of the improved CRAFT license plate enlargement character region detection model is the region score of the character and the connection score between characters. Norm loss weighted sum.

[0058] As a preferred approach, a method for merging the entire area of ​​the enlarged license plate number in the image and all character text areas in the image using a preset connectivity module to obtain the merged overall location of the enlarged license plate number, as well as the location of other text areas that may belong to the same text line, includes:

[0059] Based on the overall region of the license plate enlargement number output by the improved DBNet overall region detection model, it is determined whether the vehicle has been painted with an enlarged license plate number.

[0060] The entire area of ​​the enlarged license plate number is divided into different groups using a pre-defined connectivity module. Within each group, bounding boxes with a similarity score greater than a specified threshold are considered to belong to the same text sequence. The similarity score is calculated using the following formula:

[0061]

[0062]

[0063] in, This indicates that the dimensions of different positioning frames are similar. To indicate the similarity of the ordinate values ​​of the center points of different bounding boxes, set... and For a certain positioning box The height and ordinate of the frame. and This represents another positioning box. The height and ordinate of the frame; The normalized ratio of the height difference between different positioning boxes to their mean is used as a measure of whether the text sizes of the two boxes are similar. The normalized ratio of the difference in the ordinate height of different positioning boxes to the average height of their boxes is used as a measure of whether the ordinate values ​​of the text center points of the two boxes are similar.

[0064] The entire area of ​​the enlarged license plate number in the image and all text character areas in the image are merged, specifically as follows:

[0065] First, use the bounding boxes within the entire area of ​​the license plate enlargement and all character text areas, simultaneously calculating the height, width, and center coordinates of each bounding box; then iterate through the input bounding box list, and when different bounding boxes... and of and If the similarity score is greater than a specified threshold, they are marked as belonging to the same group. The text boxes marked as belonging to the same group are merged, that is, the x-coordinates of the center points of all text boxes in the same group are arranged from left to right, and finally the top left and bottom left coordinates of the first box and the top right and bottom right coordinates of the last box are used as the four vertices of this region to output the connected bounding boxes. Based on the threshold of the similarity score, the similarity results of the character regions can be merged. The setting of different thresholds indicates that the connectivity method allows the characters in the same text sequence to have size differences and vertical fluctuation ranges, so as to fully locate the long text of the research object.

[0066] As a preferred embodiment, the improved SVTR text recognition model includes a third feature extraction module, an encoding module, a transcription module, a decoding module, and a text content output module;

[0067] The third feature extraction module is used to extract features from the image regions corresponding to the overall positioning of the license plate enlargement number and the positioning of other text regions to obtain a third primary feature map.

[0068] The encoding module includes an Im2Seq module and a MultiHead module; the Im2Seq module is used to transpose the third primary feature map channel into a sequence, and then use two layers of convolution and normalization to downsample and obtain a patch; the Im2Seq module includes 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation.

[0069] The core design of the MultiHead module is the Global Mixing block, which is used to perceive the correlation between characters and between characters and the background. Specifically:

[0070]

[0071] in, It is a dimension. It is obtained by linear transformation of the input image region matrix. and After matrix multiplication, softmax normalization is performed to obtain the attention matrix, which is then multiplied by... Matrix multiplication is performed, followed by a linear transformation to output the result; after each linear transformation, 10% of the neurons are randomly dropped out to reduce overfitting;

[0072] The MultiHead module is used to normalize the patch feature input mixture, then perceive the correlation between characters using the Attention method. After performing identity mapping and layer normalization on the Attention method output, the features are fed into a Multilayer Perceptron (MLP) for fusion, finally obtaining the second fused feature map. The specific expression of the Attention method is as follows:

[0073]

[0074] in, It is a dimension. It is obtained by linear transformation of the input image region matrix. and After matrix multiplication, softmax normalization is performed to obtain the attention matrix, which is then multiplied by... Matrix multiplication is performed, followed by a linear transformation to output the result; after each linear transformation, 10% of the neurons are randomly dropped out to reduce overfitting;

[0075] The implementation path of the MLP is: fully connected layer and Swish activation, fully connected layer and Dropout;

[0076] The transcription module is used to perform layer normalization on the second fused feature map, and then merge the features using 3-layer convolution and batch normalization to obtain the merged feature map; wherein the 3-layer convolution and batch normalization include 1×1 convolution, batch normalization and Swish activation, 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation.

[0077] The decoding module is used to obtain the confidence score of each character prediction in the text by applying the Softmax method to the merged feature map;

[0078] The text content output module is used to mark the probability of non-target encoded characters as 0, and to use CTC decoding mapping based on the confidence level predicted for each character in the text to output the text content in the image.

[0079] As a preferred solution, the method of filtering and merging the text content in the image through a preset verification module, verifying the text content according to the license plate number encoding rules, and outputting the final verified license plate number includes:

[0080] The text information is divided into different groups through a preset verification module. The text information in each group is then judged whether to be filtered, merged, and output based on the encoding rules of the license plate number. The design method for calculating the encoding rule score is detailed in the following formula:

[0081]

[0082] in, The encoding rule score for the input text information; settings It is a text string. Indicates the string length. Define as a characteristic variable The value is 1 if the text does not conform to the license plate coding rules, and 0 otherwise. The absolute value of the difference between the current text length and the target text length, plus the indicator variable for whether the current text conforms to the license plate coding rules, is used as the metric. The maximum value of this metric is used as the score of the text regarding the license plate coding rule; the threshold for the rule score is determined by actual needs, and the verified area information will be output as the final result after filtering out false characters;

[0083] Filtering the text content in the image, specifically:

[0084] The improved SVTR text recognition model outputs text content and bounding boxes in a list. The encoding rule score of each text and the height, width and center coordinates of the corresponding bounding box are calculated. The input text list is traversed, and the encoding rule scores of adjacent text sequence combinations are calculated. The text sequence combinations with the highest scores are merged, sorted and returned. The text sequence information that meets the minimum output threshold is returned.

[0085] Based on the threshold judgment of the coding rule score, it can merge the overall regions that are not merged in the overall positioning of the merged license plate enlargement number due to excessive spacing, as well as the overall regions that may belong to the same text line. After filtering out false detection characters with too low scores, the final output is more accurate license plate enlargement number and standard motor vehicle license plate number information.

[0086] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0087] This invention builds upon three models—DBNet, CRAFT, and SVTR—by constructing a DBNet-based overall region detection network, a CRAFT-based character region detection network, and an SVTR-based text recognition network based on the characteristics of enlarged license plate numbers. Through connectivity methods, it merges the localization results of different regions, solving the problem of difficulty in localization or incomplete localization due to excessively large character intervals in the enlarged license plate number. Furthermore, through verification methods, it further filters out interference from other textual noise on the vehicle compartment.

[0088] This invention, based on example data, uses the output of the enlarged license plate region detection network as a classification result to determine whether an enlarged license plate number exists in an image. The text recognition network employs perspective transformation to address the high-angle distortion that may occur in images during typical surveillance scenarios. Compared to general license plate detection and recognition methods, and methods for detecting and recognizing enlarged license plates, this invention proposes a method that cascades both overall and character region detection networks, ultimately outputting the location results of both enlarged and standard vehicle license plates simultaneously.

[0089] The method designed in this invention is cascaded, allowing for application to other specific scenarios by replacing different modules, such as the overall detection network, connectivity module, and verification module. Therefore, the method designed in this invention has better accuracy, robustness, and transferability. Attached Figure Description

[0090] Figure 1 This is a flowchart illustrating a license plate enlargement recognition method based on overall and character region detection provided in this embodiment;

[0091] Figure 2 This embodiment provides a training flowchart for a license plate enlargement recognition method based on overall and character region detection.

[0092] Figure 3 This is a schematic diagram of the overall area detection network for the enlarged license plate number provided in this embodiment;

[0093] Figure 4 This is a schematic diagram of the character region detection network for the enlarged license plate number provided in this embodiment;

[0094] Figure 5 This is a schematic diagram of a text recognition network for enlarged license plate numbers provided in this embodiment;

[0095] Figure 6 This is a schematic diagram of the DSC module provided in this embodiment. Detailed Implementation

[0096] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the invention.

[0097] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0098] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0099] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0100] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The invention will be further described below with reference to the accompanying drawings and embodiments.

[0101] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0102] Example 1

[0103] Please refer to Figure 1 This embodiment provides a method for license plate enlargement recognition based on overall and character region detection, the method including:

[0104] A license plate enlargement dataset and a scene text public dataset are constructed respectively, wherein the license plate enlargement dataset includes a license plate enlargement training set and a license plate enlargement test set;

[0105] Improved DBNet overall region detection model, improved CRAFT character region detection model, and improved SVTR text recognition model were constructed using the license plate enlargement training set and the public scene text dataset, respectively.

[0106] The improved DBNet license plate enlargement area detection model was used to detect the license plate enlargement area in the test set data images and obtain the overall license plate enlargement area in the image.

[0107] The improved CRAFT license plate enlargement character region detection model was used to detect all character text regions in the license plate enlargement test set data images.

[0108] The pre-defined connectivity module merges the entire area of ​​the enlarged license plate number in the image with all character text areas in the image to obtain the overall location of the merged enlarged license plate number, as well as the location of other text areas that may belong to the same text line.

[0109] By improving the SVTR text recognition model, the image regions corresponding to the overall location of the enlarged license plate number and other text regions are identified, and the text content in the image is obtained.

[0110] The preset verification module filters and merges the text content in the image, verifies the text content according to the license plate number encoding rules, and outputs the final verified license plate number.

[0111] In one specific embodiment, the method includes:

[0112] A license plate enlargement dataset and a scene text public dataset are constructed respectively, wherein the license plate enlargement dataset includes a license plate enlargement training set and a license plate enlargement test set;

[0113] Improved DBNet overall region detection model, improved CRAFT character region detection model, and improved SVTR text recognition model were constructed using the license plate enlargement training set and the public scene text dataset, respectively.

[0114] The improved DBNet license plate enlargement area detection model was used to detect the license plate enlargement area in the test set data images and obtain the overall license plate enlargement area in the image.

[0115] The improved CRAFT license plate enlargement character region detection model was used to detect all character text regions in the license plate enlargement test set data images.

[0116] The pre-defined connectivity module merges the entire area of ​​the enlarged license plate number in the image with all character text areas in the image to obtain the overall location of the merged enlarged license plate number, as well as the location of other text areas that may belong to the same text line.

[0117] By improving the SVTR text recognition model, the image regions corresponding to the overall location of the enlarged license plate number and other text regions are identified, and the text content in the image is obtained.

[0118] The preset verification module filters and merges the text content in the image, verifies the text content according to the license plate number encoding rules, and outputs the final verified license plate number.

[0119] In one specific embodiment, the scene text public dataset includes an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, and a text recognition training set and a text recognition verification set.

[0120] In one specific embodiment, please refer to Figure 2 The methods for constructing improved DBNet overall region detection models, improved CRAFT character region detection models, and improved SVTR text recognition models using license plate enlargement training sets and publicly available scene text datasets include:

[0121] The improved DBNet license plate enlargement area detection network is iteratively trained using the overall detection training set, and the model accuracy is verified using the overall detection validation set. The optimal model is saved and retrained using the enlargement training set, and finally the improved DBNet overall area detection model is obtained.

[0122] The improved CRAFT license plate enlargement character region detection network is iteratively trained using the character detection training set, and the accuracy of the model is verified using the character detection validation set. The optimal model is saved and retrained using the enlargement training set, and finally the improved CRAFT character region detection model is obtained.

[0123] The improved SVTR license plate enlargement text recognition network was iteratively trained using the text recognition training set, and the accuracy of the model was verified using the text recognition validation set. The optimal model was saved and retrained using the enlargement training set, and finally the improved SVTR text recognition model was obtained.

[0124] In one specific embodiment, please refer to Figure 3 The improved DBNet global region detection model includes a first feature extraction module, a feature fusion module, a first classification prediction module, a first binarization module, and a global region output module connected in sequence.

[0125] The first feature extraction module is used to extract features from the license plate enlargement test set data images to obtain a first primary feature map;

[0126] The feature fusion module is used to obtain several feature maps of different sizes by passing the primary feature map through four 1×1 convolutions, then upsampling from top to bottom, and horizontally connecting the feature maps of the same size generated from bottom to top; the connected feature maps are then passed through four 3×3 convolutions to eliminate the aliasing effect of upsampling, and finally multi-scale feature representations are obtained and connected to obtain the first fused feature map.

[0127] The first classification prediction module is used to input the first fused feature map into a preset first volume integral class network to obtain a target probability map and a threshold map;

[0128] The first binarization module is used to generate a first approximate binary map by using a differentiable binarization method on the target probability map and the threshold map;

[0129] The overall region output module is used to obtain the vertex coordinates of the region polygon based on the first approximate binary image, and then restore these coordinates to the original image size to obtain the overall region of the enlarged license plate number.

[0130] In one specific embodiment, please refer to Figure 3 The first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer.

[0131] The first stage layer includes 3 residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation.

[0132] The second stage layer includes three residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0133] The third stage layer includes 6 residual units. The structure of the first 4 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, and 1×1 convolution, batch normalization and hard swish activation function. The structure of the last 2 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, SE module, and 1×1 convolution, batch normalization and hard swish activation function.

[0134] The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation.

[0135] The feature fusion module is a feature pyramid network;

[0136] The first classification prediction module includes a first volume integral network, which includes a target probability map output module and a threshold map output module. The target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution.

[0137] The loss function of the improved DBNet global region detection model is the prediction probability loss. Binarization loss and target area distance loss The weighted sum, where, and Using cross-entropy loss, use Norm loss.

[0138] In one specific embodiment, please refer to Figure 4 The improved CRAFT license plate enlargement character region detection model includes a second feature extraction module, a deconvolution module, a second classification prediction module, a second binarization module, and a character region output module connected in sequence.

[0139] The second feature extraction module is used to extract features from the license plate enlargement test set data images to obtain a second primary feature map;

[0140] The deconvolution module is used to deconvolve the second primary feature map to obtain a deconvolution feature map.

[0141] The second classification prediction module is used to input the deconvolution feature map into a preset second-volume integral network to obtain the region score of the corresponding character and the connection score between characters;

[0142] The second binarization module is used to connect and expand the region scores and connection scores between characters using a fixed threshold method to generate a second approximate binary image;

[0143] The character region output module is used to obtain the vertex coordinates of the region polygon based on the second approximate binary image, and then restore these coordinates to the original image size to obtain the target character region output.

[0144] In one specific embodiment, please refer to Figure 4 The second feature extraction module is VGG16_BN, which includes a first slice layer, a second slice layer, a third slice layer, a fourth slice layer and a fifth slice layer.

[0145] The first slice layer includes 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0146] The second slice layer includes ReLU activation, max pooling, 3×3 convolution, batch normalization, and ReLU activation, 3×3 convolution, and batch normalization.

[0147] The third slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0148] The fourth slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0149] The fifth slice layer includes max pooling, 3×3 convolution, and 1×1 convolution;

[0150] The deconvolution module includes a first deconvolution module, a second deconvolution module, a third deconvolution module, and a fourth deconvolution module;

[0151] The fifth slice layer is connected to the output jump layer of the fourth slice layer, then connected to the output jump layer of the third slice layer through the first deconvolution module, then connected to the output jump layer of the second slice layer through the second deconvolution module, then connected to the output jump layer of the first slice layer through the third deconvolution module, and finally connected to the input of the fourth deconvolution module.

[0152] The second classification prediction module includes a second convolutional integral network, which includes three 3×3 convolutions and two 1×1 convolutions connected in sequence. Except for the last 1×1 convolution, the other convolutional layers are activated by ReLU.

[0153] The loss function of the improved CRAFT license plate enlargement character region detection model is the region score of the character and the connection score between characters. Norm loss weighted sum.

[0154] In one specific embodiment, the method for merging the entire area of ​​the enlarged license plate number in the image and all character text areas in the image through a preset connectivity module to obtain the merged overall location of the enlarged license plate number, as well as the location of other text areas that may belong to the same text line, includes:

[0155] Based on the overall region of the license plate enlargement number output by the improved DBNet overall region detection model, it is determined whether the vehicle has been painted with an enlarged license plate number.

[0156] The entire area of ​​the enlarged license plate number is divided into different groups using a pre-defined connectivity module. Within each group, bounding boxes with a similarity score greater than a specified threshold are considered to belong to the same text sequence. The similarity score is calculated using the following formula:

[0157]

[0158]

[0159] in, This indicates that the dimensions of different positioning frames are similar. To indicate the similarity of the ordinate values ​​of the center points of different bounding boxes, set... and For a certain positioning box The height and ordinate of the frame. and This represents another positioning box. The height and ordinate of the frame; The normalized ratio of the height difference between different positioning boxes to their mean is used as a measure of whether the text sizes of the two boxes are similar. The normalized ratio of the difference in the ordinate height of different positioning boxes to the average height of their boxes is used as a measure of whether the ordinate values ​​of the text center points of the two boxes are similar.

[0160] The entire area of ​​the enlarged license plate number in the image and all text character areas in the image are merged, specifically as follows:

[0161] First, use the bounding boxes within the entire area of ​​the license plate enlargement and all character text areas, simultaneously calculating the height, width, and center coordinates of each bounding box; then iterate through the input bounding box list, and when different bounding boxes... and of and If the similarity score is greater than a specified threshold, they are marked as belonging to the same group. The text boxes marked as belonging to the same group are merged, that is, the x-coordinates of the center points of all text boxes in the same group are arranged from left to right, and finally the top left and bottom left coordinates of the first box and the top right and bottom right coordinates of the last box are used as the four vertices of this region to output the connected bounding boxes. Based on the threshold of the similarity score, the similarity results of the character regions can be merged. The setting of different thresholds indicates that the connectivity method allows the characters in the same text sequence to have size differences and vertical fluctuation ranges, so as to fully locate the long text of the research object.

[0162] In one specific embodiment, please refer to Figure 5 The improved SVTR text recognition model includes a third feature extraction module, an encoding module, a transcription module, a decoding module, and a text content output module;

[0163] The third feature extraction module is used to extract features from the image regions corresponding to the overall positioning of the license plate enlargement number and the positioning of other text regions to obtain a third primary feature map.

[0164] Specifically, the third feature extraction module is LCNet.

[0165] LCNet is an optimization of MobileNetV1. First, it uses a 3×3 convolution and batch normalization operation with the Hard Swish activation function. Second, it has 13 layers of depthwise separable convolutions (DSC).

[0166] The first six DSC layers consist of two steps. First, a channel-wise convolution (also known as depthwise convolution, DC) is implemented using 3×3 convolutions, batch normalization, and hard swish activation to reduce model parameters. Second, a pointwise convolution (PC) is implemented using 1×1 convolutions, batch normalization, and hard swish activation to deepen the model channels. The seventh layer and subsequent DSC layers also contain these two steps, but the 3×3 convolutions in the depthwise convolution (DC) are replaced with 5×5 convolutions. In the last two DSC layers, an SE module is inserted between DC and PC.

[0167] One average pooling is performed after all the above DSCs.

[0168] The encoding module includes an Im2Seq module and a MultiHead module; the Im2Seq module is used to transpose the third primary feature map channel into a sequence, and then use two layers of convolution and normalization to downsample and obtain a patch; the Im2Seq module includes 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation.

[0169] The core design of the MultiHead module is the Global Mixing block, which is used to perceive the correlation between characters and between characters and the background. Specifically:

[0170]

[0171] in, It is a dimension. It is obtained by linear transformation of the input image region matrix. and After matrix multiplication, softmax normalization is performed to obtain the attention matrix, which is then multiplied by... Matrix multiplication is performed, followed by a linear transformation to output the result; after each linear transformation, 10% of the neurons are randomly dropped out to reduce overfitting;

[0172] The MultiHead module is used to normalize the patch feature input mixture, then perceive the correlation between characters using the Attention method. After performing identity mapping and layer normalization on the Attention method output, the features are fed into a Multilayer Perceptron (MLP) for fusion, finally obtaining the second fused feature map. The specific expression of the Attention method is as follows:

[0173]

[0174] in, It is a dimension. It is obtained by linear transformation of the input image region matrix. and After matrix multiplication, softmax normalization is performed to obtain the attention matrix, which is then multiplied by... Matrix multiplication is performed, followed by a linear transformation to output the result; after each linear transformation, 10% of the neurons are randomly dropped out to reduce overfitting;

[0175] The implementation path of the MLP is: fully connected layer and Swish activation, fully connected layer and Dropout;

[0176] The transcription module is used to perform layer normalization on the second fused feature map, and then merge the features using 3-layer convolution and batch normalization to obtain the merged feature map; wherein the 3-layer convolution and batch normalization include 1×1 convolution, batch normalization and Swish activation, 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation.

[0177] The decoding module is used to obtain the confidence score of each character prediction in the text by applying the Softmax method to the merged feature map;

[0178] The text content output module is used to mark the probability of non-target encoded characters as 0, and to use CTC decoding mapping based on the confidence of each character predicted in the text to output the text content in the image.

[0179] Specifically, the improved SVTR text recognition model uses a weighted sum of CTCLoss and CrossEntropyLoss as the network's loss function.

[0180] In one specific embodiment, the method of filtering and merging text content in an image through a preset verification module, verifying the text content according to the license plate number encoding rules, and outputting the final verified license plate number includes:

[0181] The text information is divided into different groups through a preset verification module. The text information in each group is then judged whether to be filtered, merged, and output based on the encoding rules of the license plate number. The design method for calculating the encoding rule score is detailed in the following formula:

[0182]

[0183] in, The encoding rule score for the input text information; settings It is a text string. Indicates the string length. Define as a characteristic variable The value is 1 if the text does not conform to the license plate coding rules, and 0 otherwise. The absolute value of the difference between the current text length and the target text length, plus the indicator variable for whether the current text conforms to the license plate coding rules, is used as the metric. The maximum value of this metric is used as the score of the text regarding the license plate coding rule; the threshold for the rule score is determined by actual needs, and the verified area information will be output as the final result after filtering out false characters.

[0184] Filtering the text content in the image, specifically:

[0185] The improved SVTR text recognition model outputs text content and bounding boxes in a list. The encoding rule score for each text and the height, width, and center coordinates of its corresponding bounding box are calculated. The input text list is iterated through, and the encoding rule scores of adjacent text sequence combinations are calculated. The text sequence combinations with the highest scores are merged, sorted, and then the text sequence information that meets the minimum output threshold is returned.

[0186] Based on the threshold judgment of the coding rule score, it can merge the overall regions that are not merged in the overall positioning of the merged license plate enlargement number due to excessive spacing, as well as the overall regions that may belong to the same text line. After filtering out false detection characters with too low scores, the final output is more accurate license plate enlargement number and standard motor vehicle license plate number information.

[0187] Example 2

[0188] This embodiment can be considered an improved or extended embodiment based on Embodiment 1, specifically: a method for license plate enlargement recognition based on overall and character region detection, the method comprising:

[0189] A license plate enlargement dataset and a scene text public dataset are constructed respectively, wherein the license plate enlargement dataset includes a license plate enlargement training set and a license plate enlargement test set;

[0190] Specifically, the license plate enlargement dataset includes images of the rear of vehicles in street scenes, both with and without enlarged license plate numbers. The enlarged license plate data is named by number and divided into a training set and a test set. Publicly available datasets of scene text from general scenarios are collected for pre-training the model, divided into overall detection training and validation sets, character detection training and validation sets, and text recognition training and validation sets.

[0191] Specifically, enlarged license plate images are typically obtained from road or checkpoint cameras and on-site photography. This dataset includes 1663 rear-view images of medium to large vehicles that may contain enlarged license plates from real-world scenes, including publicly available license plate image databases and manually collected images. The files are saved with the .jpg extension and vary in resolution, covering samples that are relatively difficult to detect and identify, including those obtained under conditions such as extreme weather, poor exposure, non-rigid painted surfaces, and obscured enlarged license plates. It includes 1404 positive class sample images of enlarged license plates and 259 background sample images containing only medium to large vehicles but not enlarged license plates.

[0192] Specifically, PaddleLabel is used for manual text bounding box annotation. The method involves selecting the entire area of ​​the enlarged license plate number within the image (annotating only the enlarged license plate number, ignoring the annotation of regular license plates). The text content of the image samples is labeled with the characters and numbers corresponding to the enlarged license plate number. The difficulty level of the image sample is recorded as 1, indicating that it is a relatively difficult sample to detect and recognize; if the enlarged license plate number is clearly identifiable, it is recorded as 0. The resulting label list is saved with a .txt extension, with each .jpg image corresponding to one .txt file. For example, Z01882.txt contains two lines:

[0193] 947,477,1196,477,1196,532,947,532,0, "Beijing KAY030"

[0194] 542,671,673,671,673,701,542,701,1, "Beijing AAD276"

[0195] In the above image with the representative number "Z01882", there are two marked license plate enlargement numbers. "947,477", "1196,477", "1196,532", and "947,532" respectively represent the values corresponding to the upper - left, upper - right, lower - right, and lower - left coordinates of the license plate enlargement number. "0" represents the difficulty level of recognition, and "Beijing KAY030" is the corresponding license plate number information. In the second line, "542,671,673,671,673,701,542,701,1" represents the coordinate information and the difficulty level of recognition corresponding to the license plate enlargement number "Beijing AAD276".

[0196] Specifically, the original image does not require pre - processing. After reading the name, the train_test_split method in the sklearn library in Python is used to randomly divide the training set and test set in an 8:2 ratio, and then it can be directly loaded through the cv2 library. The loaded images are uniformly converted into numpy arrays with three BGR channels.

[0197] Specifically, an open - source dataset ICDAR and a publicly available pre - trained model based on this dataset are additionally obtained. The model is uniformly migrated to the PyTorch framework, and experiments are conducted on the real - world dataset based on this.

[0198] An improved DBNet overall region detection model, an improved CRAFT character region detection model, and an improved SVTR text recognition model are respectively constructed through the license plate enlargement number training set and the scene text public dataset;

[0199] Specifically, the "pre - training - re - training" method is adopted to train the improved DBNet license plate enlargement number overall region detection network, the improved CRAFT license plate enlargement number character region detection network, and the improved SVTR license plate enlargement number text recognition network. In this embodiment, considering that the amount of real - world license plate enlargement number data is small, the publicly available scene text public dataset ICDAR is selected to pre - train the above models, and then re - training is carried out through the enlargement number dataset collected in S1, so as to obtain the final model.

[0200] Specifically, during the detection phase, the input size of the DBNet model is 1×3×640×W or 1×3×H×640, where H and W are multiples of 32; the maximum input size H and W of the CRAFT model during the detection phase is limited to 1280, and H and W are multiples of 64 and 32, respectively. Preprocessing operations are performed on the original image input to the detection network, including scaling, normalization, and channel transposition.

[0201] The uniform size of the model in the recognition stage is 1×3×48×W, where W is not limited. When the detection results are input into the recognition network, image processing is performed, including cropping the image in the detection bounding box to obtain the image in the localization box, performing high-level top-down deformation to return the transformed rectangular image, updating the width and height of the horizontal or non-horizontal image list to fit the network input size, and sorting the images by the ordinate of the center point of the rectangular region after normalization.

[0202] Specifically, in this embodiment, 80% of the license plate enlargement dataset is randomly selected as the retraining set, and 20% is used as the test set for numerical experiments. The initial learning rate is set to 0.001, the decay rate to 0.9, and the number of training epochs to 100. This embodiment is based on the GPU (Nvidia Tesla T4), PyTorch, Paddle, and CUDA frameworks.

[0203] The improved DBNet license plate enlargement area detection model was used to detect the license plate enlargement area in the test set data images and obtain the overall license plate enlargement area in the image.

[0204] Specifically, the DBNet-based license plate enlargement region detection model comprises four steps: feature extraction, feature fusion, classification prediction, and binarization. This embodiment replaces the traditional ResNet network with the MobileNetV3 network as the model's backbone. It introduces an SE module to enhance the network's local attention by strengthening the interdependence between convolutional feature channels. The specific structure of the improved DBNet network is shown below. Figure 3 .

[0205] Specifically, in combination Figure 3 The network structure used for feature data extraction has the following path structure:

[0206] [((C3×3_3_8, s=2×2, p=1×1), BN, HSwish)]

[0207] Where (C3×3_3_8, s=2×2, p=1×1) represents a convolution (denoted as C) with a kernel size of 3×3, 3 input channels, 8 output channels, a stride (denoted as s) of 1, and padding (denoted as p) of 1. BN represents batch normalization operation, and HSwish represents activation function Hard Swish.

[0208] Stage1: [((C1×1_8_8, s=1×1), BN, R), ((C3×3_8_8, s=1×1, p=1×1, g=8), BN, R), ((C1×1_8_8, s=1×1), BN)], [((C1×1_8_32, s=1×1), BN, R), ((C3×3_32_32, s=2×2, p=1×1, g=32), BN, R), ((C1×1_32_16, s=1×1), BN)], [((C1×1_16_40, s=1×1), BN, R), ((C3×3_40_40, s=1×1, p=1×1, g=40), BN, R),((C1×1_40_16, s=1×1), BN)].

[0209] Here, ((C1×1_8_8, s=1×1), BN, R) represents a convolution with a kernel size of 1×1, 8 input channels, and 8 output channels, where BN represents batch normalization and R represents the ReLU activation function. The next ((C3×3_8_8, s=1×1, p=1×1, g=8), BN, R) represents a grouped convolution with a kernel size of 3×3, 8 input channels, 8 output channels, a stride of 1, padding of 1, and 8 groups after the 1×1 convolution, batch normalization, and ReLU activation, where BN represents batch normalization and R represents the ReLU activation function. The following ((C1×1_8_8, s=1×1), BN) represents a convolution with a kernel size of 1×1, 8 input channels, 8 output channels, and a stride of 1, where BN represents batch normalization, and so on. [((C1×1_8_8, s=1×1), BN, R), ((C3×3_8_8, s=1×1, p=1×1, g=8), BN, R),((C1×1_8_8, s=1×1), BN)] indicates that this is a unit.

[0210] Stage2: [((C1×1_16_40, s=1×1), BN, R), ((C5×5_40_40, s=2×2, p=2×2, g=40), BN, R), [Adaptive, ((C1×1_40_10, s=1×1), R), ((C1×1_10_40, s=1×1), HSigmoid)], ((C1×1_40_24, s=1×1), BN)], [((C1×1_24_64, s=1×1), BN,R), ((C5×5_64_64, s=1×1, p=2×2, g=64), BN, R), [Adaptive, ((C1×1_64_16, s=1×1), R), ((C1×1_16_64, s=1×1), HSigmoid)], ((C1×1_64_24, s=1×1), BN)],[((C1×1_24_64, s=1×1), BN, R), ((C5×5_64_64, s=1×1, p=2×2, g=64), BN,R), [Adaptive, ((C1×1_64_16, s=1×1), R), ((C1×1_16_64, s=1×1),HSigmoid)], ((C1×1_64_24, s=1×1), BN)].

[0211] Here, Adaptive represents adaptive average pooling, HSigmoid represents activation function HardSigmoid, and the rest are the same as above. [Adaptive, ((C1×1_40_10, s=1×1), R), ((C1×1_10_40, s=1×1), HSigmoid)], ((C1×1_40_24, s=1×1), BN)] represents the structure of an SE module.

[0212] Stage3: [((C1×1_24_120, s=1×1), BN, HSwish), ((C3×3_120_120, s=2×2, p=1×1, g=120), BN, HSwish), ((C1×1_120_40, s=1×1), BN)], [((C1×1_40_104, s=1×1), BN, HSwish), ((C3×3_104_104, s=1×1, p=1×1, g=104), BN,HSwish), ((C1×1_104_40, s=1×1), BN)], [((C1×1_40_96, s=1×1), BN, HSwish),((C3×3_96_96, s=1×1, p=1×1, g=96), BN, HSwish), ((C1×1_96_40, s=1×1),BN)], [((C1×1_40_96, s=1×1), BN, HSwish), ((C3×3_96_96, s=1×1, p=1×1, g=96), BN, HSwish), ((C1×1_96_40, s=1×1), BN)], [((C1×1_40_240, s=1×1), BN,HSwish), ((C3×3_240_240, s=1×1, p=1×1, g=240), BN, HSwish), [Adaptive,((C1×1_240_60, s=1×1), R), ((C1×1_60_240, s=1×1), HSigmoid)], ((C1×1_240_56, s=1×1), BN)], [((C1×1_56_336, s=1×1), BN, HSwish), ((C3×3_336_336, s=1×1, p=1×1, g=336), BN, HSwish), [Adaptive), ((C1×1_336_84, s=1×1), R), ((C1×1_84_336, s=1×1), HSigmoid)], ((C1×1_336_56, s=1×1), BN)].

[0213] Here, HSwish indicates that the activation function is Hard Swish, and the rest is the same as above.

[0214] Stage4: [((C1×1_56_336, s=1×1), BN, HSwish), ((C5×5_336_336, s=2×2, p=2×2, g=336), BN, HSwish), [Adaptive, ((C1×1_336_84, s=1×1), R), ((C1×1_84_336, s=1×1), HSigmoid)], ((C1×1_336_80, s=1×1), BN)], [((C1×1_80_480, s=1×1), BN, HSwish), ((C5×5_480_480, s=1×1, p=2×2, g=480), BN,HSwish), [Adaptive, ((C1×1_480_120, s=1×1), R), ((C1×1_120_480, s=1×1),HSigmoid)], ((C1×1_480_80, s=1×1), BN)], [((C1×1_80_480, s=1×1), BN,HSwish), ((C5×5_480_480, s=1×1, p=2×2, g=480), BN, HSwish), [Adaptive),((C1×1_480_120, s=1×1), R), ((C1×1_120_480, s=1×1), HSigmoid)], ((C1×1_480_80, s=1×1), BN)], [((C1×1_80_480, s=1×1), BN, HSwish)].

[0215] Here, 1×1 convolution represents a fully connected layer.

[0216] Specifically, in combination Figure 3 The network structure used for feature fusion has the following path structure:

[0217] [(C1×1_16_256, s=1×1), (C1×1_24_256, s=1×1), (C1×1_56_256, s=1×1), (C1×1_480_256, s=1×1), (C3×3_256_64, s=1×1, p=1×1), (C3×3_256_64, s=1×1, p=1×1), (C3×3_256_64, s=1×1, p=1×1), (C3×3_256_64, s=1×1, p=1×1)].

[0218] Here, feature maps of different sizes obtained from four 1×1 convolutions are laterally concatenated with feature maps of the same size generated from the four stages in S301. The concatenated feature maps are then subjected to four 3×3 convolutions to obtain multi-scale feature representations and then concatenated.

[0219] Specifically, in combination Figure 3 The network structure used for classification prediction has the following path structure:

[0220] Probability: [((C3×3_256_64, s=1×1, p=1×1), BN, R), ((T2×2_64_64, s=2×2), BN), (T2×2_64_1, s=2×2)].

[0221] Threshold: [((C3×3_256_64, s=1×1, p=1×1), BN, R), ((T2×2_64_64, s=2×2), BN), (T2×2_64_1, s=2×2)].

[0222] The two parts have the same network structure and are used to output the probability map and threshold map, respectively.

[0223] Specifically, the model was pre-trained using the ICDAR dataset and then retrained on the license plate enlargement training set. The trained model was used to segment the entire license plate enlargement region. Test results show that, in detecting enlarged license plates of medium and large vehicles, this embodiment achieved a precision of 91.79%, a recall of 83.56%, and an F1 score of 87.48% on the test set.

[0224] The improved CRAFT license plate enlargement character region detection model was used to detect all character text regions in the license plate enlargement test set data images.

[0225] Specifically, the CRAFT-based license plate enlargement character region detection model includes four steps: feature extraction, deconvolution, classification prediction, and binarization. This embodiment employs a U-Net-like skip-layer connection structure in the deconvolution layer. The specific structure of the improved CRAFT network can be found in [link to documentation]. Figure 4 .

[0226] Specifically, in combination Figure 4 The network structure used for feature data extraction has the following path structure:

[0227] Slice1: [(C3×3_3_64, s=1×1, p=1×1), BN, R), ((C3×3_64_64, s=1×1, p=1×1), BN, R), M(kernel=2, s=2, p=0, d=1), ((C3×3_64_128, s=1×1, p=1×1),BN, R), ((C3×3_128_128, s=1×1, p=1×1), BN)].

[0228] Slice2: [R, M(kernel=2, s=2, p=0, d=1), ((C3×3_128_256, s=1×1, p=1×1), BN, R), ((C3×3_256_256, s=1×1, p=1×1), BN)].

[0229] Slice3: [R, ((C3×3_256_256, s=1×1, p=1×1), BN, R), M(kernel=2, s=2, p=0, d=1), ((C3×3_256_512, s=1×1, p=1×1), BN, R), ((C3×3_512_512, s=1×1, p=1×1), BN)].

[0230] Slice4: [R, ((C3×3_512_512, s=1×1, p=1×1), BN, R), M(kernel=2, s=2, p=0, d=1), ((C3×3_512_512, s=1×1, p=1×1), BN, R), ((C3×3_512_512, s=1×1, p=1×1), BN)].

[0231] Slice5: [M(kernel=3, s=1, p=1, d=1), (C3×3_512_1024, s=1×1, p=6×6, d=6×6), (C1×1_1024_1024, s=1×1)].

[0232] Where M(kernel=3, s=1, p=1, d=1) represents a max pooling operation with a kernel size of 3, a stride of 1, a padding ratio of 1, and an expansion factor of 1. The remaining notations have the same meaning as S3.

[0233] Combination Figure 4 The network structure, the path structure of deconvolution is:

[0234] [((C1×1_1536_512, s=1×1), BN, R), ((C3×3_512_256, s=1×1, p=1×1),BN, R), [((C1×1_768_256, s=1×1), BN, R), ((C3×3_256_128, s=1×1, p=1×1),BN, R)], [((C1×1_384_128, s=1×1), BN, R), ((C3×3_128_64, s=1×1, p=1×1),BN, R)], [((C1×1_192_64, s=1×1), BN, R), ((C3×3_64_32, s=1×1, p=1×1),BN, R)].

[0235] Here, the results obtained from the four deconvolution modules are connected to the results of Slices via a skip layer.

[0236] Combination Figure 4 The network structure used for classification prediction has the following path structure:

[0237] [((C3×3_32_32, s=1×1, p=1×1), R), ((C3×3_32_32, s=1×1, p=1×1),R), ((C3×3_32_16, s=1×1, p=1×1), R), ((C1×1_16_16, s=1×1), R)],[(C1×1_16_2, s=1×1)].

[0238] The final convolution output has 2 channels, representing the results of the region score map and the affinity score map, respectively.

[0239] Specifically, the binarization method involves manually fixing a threshold value, which is set to 0.5 in this embodiment.

[0240] Specifically, the model is pre-trained using the ICDAR dataset, eliminating the need for retraining on the license plate enlargement training set. The trained model is then used to segment image character regions. Test results show that, in detecting enlarged license plate numbers for medium and large vehicles, this embodiment achieves a precision of 78.74%, a recall of 83.16%, and an F1 score of 80.89% on the test set.

[0241] The pre-defined connectivity module merges the entire area of ​​the enlarged license plate number in the image with all character text areas in the image to obtain the overall location of the merged enlarged license plate number, as well as the location of other text areas that may belong to the same text line.

[0242] Specifically, this embodiment is based on the assumption that although there is no fixed layout for enlarged license plate numbers, the size of the characters is generally consistent and they are usually arranged horizontally. In other words, the size difference of characters with the same enlarged license plate number in the image coordinate system should be small, and the difference in the vertical coordinate value of characters with the same enlarged license plate number in the image coordinate system should be small. Therefore, this embodiment sets... and The similarity score thresholds were 0.83 and 0.70, respectively.

[0243] By improving the SVTR text recognition model, the image regions corresponding to the overall location of the enlarged license plate number and other text regions are identified, and the text content in the image is obtained.

[0244] Specifically, the SVTR-based license plate amplification text model comprises four steps: feature extraction, encoding, transcription, and decoding. This embodiment uses the LCNet network to replace the traditional ResNet network as the model's backbone. Please refer to... Figure 6 This implementation introduces a DSC module, reducing the number of model parameters and computational complexity through depthwise convolution and pointwise convolution. Furthermore, this embodiment replaces the original network's simultaneous use of Local Mixing and Global Mixing with Global Mixing, increasing global attention while reducing the computational complexity of local attention, thus better perceiving the relationships between characters. The specific structure of the improved SVTR network can be found in [link to improved SVTR network]. Figure 5 .

[0245] Combination Figure 5 The network structure used for feature data extraction has the following path structure:

[0246] [((C3×3_3_8, s=2×2, p=1×1), BN, HSwish)] [((C3×3_3_16, s=2×2, p=1×1), BN, HSwish)]

[0247] DSC1: [((C3×3_16_16, s=1×1, p=1×1, g=16), BN, HSwish), ((C1×1_16_32, s=1×1), BN, HSwish)]

[0248] DSC2: [((C3×3_32_32, s=1×1, p=1×1, g=32), BN, HSwish), ((C1×1_32_64, s=1×1), BN, HSwish)]

[0249] DSC3: [((C3×3_64_64, s=1×1, p=1×1, g=64), BN, HSwish), ((C1×1_64_64, s=1×1), BN, HSwish)]

[0250] DSC4: [((C3×3_64_64, s=1×1, p=1×1, g=64), BN, HSwish), ((C1×1_64_128, s=1×1), BN, HSwish)]

[0251] DSC5: [((C3×3_128_128,s=1×1,p=1×1, g=128), BN, HSwish), ((C1×1_128_128, s=1×1), BN, HSwish)]

[0252] DSC6: [((C3×3_128_128,s=1×1,p=1×1, g=128), BN, HSwish), ((C1×1_128_256, s=1×1), BN, HSwish)]

[0253] DSC7: [((C5×5_256_256,s=1×1,p=2×2, g=256), BN, HSwish), ((C1×1_256_256, s=1×1), BN, HSwish)]

[0254] DSC8: [((C5×5_256_256,s=1×1,p=2×2, g=256), BN, HSwish), ((C1×1_256_256, s=1×1), BN, HSwish)]

[0255] DSC9: [((C5×5_256_256,s=1×1,p=2×2,g=256), BN, HSwish),((C1×1_256_256, s=1×1), BN, HSwish)]

[0256] DSC10: [((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256, s=1×1), BN, HSwish)]

[0257] DSC11: [((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256, s=1×1), BN, HSwish)]

[0258] DSC12: [((C5×5_256_256, s=1×1, p=2×2, g=256), BN, HSwish),[Adaptive, ((C1×1_256_64, s=1×1), R), ((C1×1_64_256, s=1×1), HSigmoid)],((C1×1_256_512, s=1×1), BN, HSwish)]

[0259] DSC13: [((C5×5_512_512, s=1×1, p=2×2, g=512), BN, HSwish),[Adaptive, ((C1×1_512_128, s=1×1), R), ((C1×1_128_512, s=1×1),HSigmoid)], ((C1×1, 512, 512, s=1×1), BN, HSwish)]

[0260] [A(kernel=2, s=2, p=0)]

[0261] Here, [A(kernel=2, s=2, p=0)] represents an average pooling operation with a kernel size of 2, a stride of 1, and a padding of 1. The remaining notations have the same meaning as in S3. See the diagram for the structure of the DSC module. Figure 6 .

[0262] Combination Figure 5 The network structure, and the path structure used for the encoding part are as follows:

[0263] Embedding: [((C3×3_512_64, s=1×1, p=1×1), BN, Swish), ((C1×1_64_120, s=1×1), BN, Swish)]

[0264] Attention: [LN, ((L_120_360), Drop(0.1)), ((L_120_120), Drop(0.1))]

[0265] MLP: [LN, ((L_120_240), Swish), ((L_240_120), Drop(0.1))]

[0266] Attention: [LN, ((L_120_360), Drop(0.1)), ((L_120_120), Drop(0.1))]

[0267] MLP: [LN, ((L_120_240), Swish), ((L_240_120), Drop(0.1))]

[0268] Where Swish indicates that the activation function is Swish, LN indicates the layer normalization operation, (L_120_240) indicates a linear connection layer with 120 input channels and 240 output channels, and Drop(0.1) indicates a Dropout layer with a dropout rate of 0.1.

[0269] Combination Figure 5 The network structure, and the pathway structure for the transcription part, are as follows:

[0270] [LN, ((C1×1_120_512, s=1×1), BN, Swish)], [((C3×3_1024_64, s=1×1,p=1×1), BN, Swish), ((C1×1_64_64, s=1×1), BN, Swish)], [(L_64_6625)]

[0271] The last fully connected layer has 6625 output channels, which means that there are 6623 Chinese, English and numeric characters in the model's pre-training dictionary. The transcribed sequence is marked with blank at the beginning and end, which corresponds to transing non-textual perception into blank symbols.

[0272] The confidence score for each character in the text is obtained using Softmax. In this embodiment, the confidence score for characters other than the license plate number is set to 0.

[0273] Pre-training was performed using the ICDAR dataset, eliminating the need for retraining on the license plate enlargement training set. The trained model was then used to recognize the text box region output by S5. Test results showed that, for enlarged license plate numbers of medium and large vehicles, this embodiment achieved a license plate recognition accuracy of 81.75% on the test set.

[0274] The text content in the image is filtered and merged by a preset verification module, and the text content is verified according to the license plate number encoding rules. The final verified license plate number is then output.

[0275] Specifically, in this embodiment As a scoring factor in power calculation Value, setting The value is set to 0.8, which normalizes the encoding rule score to the [0, 1] interval and, through exponentiation, significantly highlights the differences in encoding rule scores for different text information. Finally, the result with the highest score among all localized bounding boxes in the image is output.

[0276] This embodiment is based on a dataset collected from existing real-world scenarios. Its test results show that, in the recognition of license plates and enlarged license plate numbers of medium and large vehicles, the accuracy of this invention is 23.17% higher than that of existing license plate recognition technologies, and the recognition performance is significantly improved, as shown in Table 1.

[0277] Table 1 Model Performance Evaluation (Note: HyperLPR is the current mainstream method, which can only recognize standard vehicle license plates; T / F indicates whether the module is used or not)

[0278]

[0279] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for license plate enlargement recognition based on overall and character region detection, characterized in that, The method includes: A license plate enlargement dataset and a scene text public dataset are constructed respectively, wherein the license plate enlargement dataset includes a license plate enlargement training set and a license plate enlargement test set; Improved DBNet overall region detection model, improved CRAFT character region detection model, and improved SVTR text recognition model were constructed using the license plate enlargement training set and the public scene text dataset, respectively. By improving the DBNet global region detection model, the image of the license plate enlargement test set was detected, and the global region of the license plate enlargement in the image was obtained. By improving the CRAFT character region detection model, the test set data images of enlarged license plate numbers were detected, and all character text regions in the images were obtained. The pre-defined connectivity module merges the entire area of ​​the enlarged license plate number in the image with all character text areas in the image to obtain the overall location of the merged enlarged license plate number, as well as the location of other text areas that may belong to the same text line. By improving the SVTR text recognition model, the image regions corresponding to the overall location of the enlarged license plate number and other text regions are identified, and the text content in the image is obtained. The preset verification module filters and merges the text content in the image, verifies the text content according to the license plate number encoding rules, and outputs the final verified license plate number.

2. The method for license plate enlargement recognition based on overall and character region detection according to claim 1, characterized in that, The publicly available dataset of scene texts includes an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, and a text recognition training set and a text recognition verification set.

3. The method for license plate enlargement recognition based on overall and character region detection according to claim 2, characterized in that, The methods for constructing improved DBNet overall region detection models, improved CRAFT character region detection models, and improved SVTR text recognition models using license plate enlargement training sets and publicly available scene text datasets include: The improved DBNet license plate enlargement area detection network is iteratively trained using the overall detection training set, and the model accuracy is verified using the overall detection validation set. The optimal model is saved and retrained using the enlargement training set, and finally the improved DBNet overall area detection model is obtained. The improved CRAFT license plate enlargement character region detection network is iteratively trained using the character detection training set, and the accuracy of the model is verified using the character detection validation set. The optimal model is saved and retrained using the enlargement training set, and finally the improved CRAFT character region detection model is obtained. The improved SVTR license plate enlargement text recognition network was iteratively trained using the text recognition training set, and the accuracy of the model was verified using the text recognition validation set. The optimal model was saved and retrained using the enlargement training set, and finally the improved SVTR text recognition model was obtained.

4. The method for license plate enlargement recognition based on overall and character region detection according to claim 1, characterized in that, The improved DBNet global region detection model includes a first feature extraction module, a feature fusion module, a first classification prediction module, a first binarization module, and a global region output module connected in sequence. The first feature extraction module is used to extract features from the license plate enlargement test set data images to obtain a first primary feature map; The feature fusion module is used to obtain several feature maps of different sizes by passing the primary feature map through four 1×1 convolutions, then upsampling from top to bottom, and horizontally connecting the feature maps of the same size generated from bottom to top; the connected feature maps are then passed through four 3×3 convolutions to eliminate the aliasing effect of upsampling, and finally multi-scale feature representations are obtained and connected to obtain the first fused feature map. The first classification prediction module is used to input the first fused feature map into a preset first volume integral class network to obtain a target probability map and a threshold map; The first binarization module is used to generate a first approximate binary map by using a differentiable binarization method on the target probability map and the threshold map; The overall region output module is used to obtain the vertex coordinates of the region polygon based on the first approximate binary image, and then restore these coordinates to the original image size to obtain the overall region of the enlarged license plate number.

5. The method for license plate enlargement recognition based on overall and character region detection according to claim 4, characterized in that, The first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer. The first stage layer includes 3 residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation. The second stage layer includes three residual units. The structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation; The third stage layer includes 6 residual units. The structure of the first 4 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, and 1×1 convolution, batch normalization and hard swish activation function. The structure of the last 2 residual units includes 1×1 convolution, batch normalization and hard swish activation function, 3×3 convolution, batch normalization and hard swish activation function, SE module, and 1×1 convolution, batch normalization and hard swish activation function. The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation. The feature fusion module is a feature pyramid network; The first classification prediction module includes a first volume integral network, which includes a target probability map output module and a threshold map output module. The target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution. The loss function of the improved DBNet global region detection model is the prediction probability loss L. s Binarization loss L b and target area distance loss L t The weighted sum of L, where L s and L b Using cross-entropy loss, L t Use L1 norm loss.

6. The method for license plate enlargement recognition based on overall and character region detection according to claim 1, characterized in that, The improved CRAFT character region detection model includes a second feature extraction module, a deconvolution module, a second classification prediction module, a second binarization module, and a character region output module connected in sequence. The second feature extraction module is used to extract features from the license plate enlargement test set data images to obtain a second primary feature map; The deconvolution module is used to deconvolve the second primary feature map to obtain a deconvolution feature map. The second classification prediction module is used to input the deconvolution feature map into a preset second-volume integral network to obtain the region score of the corresponding character and the connection score between characters; The second binarization module is used to connect and expand the region scores and connection scores between characters using a fixed threshold method to generate a second approximate binary image; The character region output module is used to obtain the vertex coordinates of the region polygon based on the second approximate binary image, and then restore these coordinates to the original image size to obtain the target character region output.

7. The method for license plate enlargement recognition based on overall and character region detection according to claim 6, characterized in that, The second feature extraction module is VGG16_BN, which includes a first slice layer, a second slice layer, a third slice layer, a fourth slice layer, and a fifth slice layer; The first slice layer includes 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The second slice layer includes ReLU activation, max pooling, 3×3 convolution, batch normalization, and ReLU activation, 3×3 convolution, and batch normalization. The third slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The fourth slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The fifth slice layer includes max pooling, 3×3 convolution, and 1×1 convolution; The deconvolution module includes a first deconvolution module, a second deconvolution module, a third deconvolution module, and a fourth deconvolution module; The fifth slice layer is connected to the output jump layer of the fourth slice layer, then connected to the output jump layer of the third slice layer through the first deconvolution module, then connected to the output jump layer of the second slice layer through the second deconvolution module, then connected to the output jump layer of the first slice layer through the third deconvolution module, and finally connected to the input of the fourth deconvolution module. The second classification prediction module includes a second convolutional integral network, which includes three 3×3 convolutions and two 1×1 convolutions connected in sequence. Except for the last 1×1 convolution, the other convolutional layers are activated by ReLU. The loss function of the improved CRAFT license plate enlargement character region detection model is the weighted sum of the L2 norm loss corresponding to the region score of the character and the connection score between characters.

8. The method for license plate enlargement recognition based on overall and character region detection according to claim 1, characterized in that, The method of merging the entire area of ​​the enlarged license plate number in the image and all character text areas in the image through a preset connectivity module to obtain the overall location of the merged enlarged license plate number, as well as the location of other text areas that may belong to the same text line, includes: Based on the overall region of the license plate enlargement number output by the improved DBNet overall region detection model, it is determined whether the vehicle has been painted with an enlarged license plate number. The entire area of ​​the enlarged license plate number is divided into different groups using a pre-defined connectivity module. Within each group, bounding boxes with a similarity score greater than a specified threshold are considered to belong to the same text sequence. The similarity score is calculated using the following formula: Here, Shape(u,v) represents the size similarity of different bounding boxes, and Coord(u,v) represents the similarity of the center point ordinate values ​​of different bounding boxes. h(u) and y(u) are set as the box height and ordinate of a certain bounding box u, and h(v) and y(v) represent the box height and ordinate of another bounding box v. Shape(u,v) uses the normalized ratio of the difference in box heights of different bounding boxes to their mean as a measure of whether the text sizes of the two are similar, and Coord(u,v) uses the normalized ratio of the difference in ordinate heights of different bounding boxes to their mean as a measure of whether the ordinate values ​​of the center points of the two texts are similar. The entire area of ​​the enlarged license plate number in the image and all text character areas in the image are merged, specifically as follows: First, bounding boxes are used in the entire region of the license plate enlargement and all character text regions. The height, width, and center coordinates of each bounding box are calculated. The input bounding box list is traversed. When the Shape(u,v) and Coord(u,v) of different bounding boxes u and v are both greater than the specified similarity score threshold, they are marked as the same group. The text boxes marked as the same group are merged. That is, the x-coordinates of the center points of all text boxes in the same group are arranged from left to right. Finally, the top-left and bottom-left coordinates of the first box and the top-right and bottom-right coordinates of the last box are used as the four vertices of this region to output the connected bounding boxes. The similarity result of the merged character regions is judged based on the similarity score threshold. The setting of different thresholds indicates that the connectivity method allows characters in the same text sequence to have size differences and vertical fluctuation ranges, so as to fully locate the long text of the research object.

9. The method for license plate enlargement recognition based on overall and character region detection according to claim 1, characterized in that, The improved SVTR text recognition model includes a third feature extraction module, an encoding module, a transcription module, a decoding module, and a text content output module; The third feature extraction module is used to extract features from the image regions corresponding to the overall positioning of the license plate enlargement number and the positioning of other text regions to obtain a third primary feature map. The encoding module includes an Im2Seq module and a MultiHead module; the Im2Seq module is used to transpose the channels of the third primary feature map into a sequence, and then downsample it using two layers of convolution and normalization to obtain a patch; the Im2Seq module includes 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation. The core design of the MultiHead module is the Global Mixing block, which is used to perceive the correlation between characters and between characters and the background. The MultiHead module is used to input patch features into a mixed block, perform layer normalization, then use the Attention method to perceive the correlation between characters, and then perform identity mapping and layer normalization on the output of the Attention method. The features are then fed into a multilayer perceptron (MLP) to achieve fusion, and finally the second fused feature map is obtained. The implementation path of the MLP is: fully connected layer and Swish activation, fully connected layer and Dropout; The transcription module is used to perform layer normalization on the second fused feature map, and then use 3 layers of convolution and batch normalization to merge the features to obtain the merged feature map; The decoding module is used to obtain the confidence score of each character prediction in the text by applying the Softmax method to the merged feature map; The text content output module is used to mark the probability of non-target encoded characters as 0, and to use CTC decoding mapping based on the confidence level predicted for each character in the text to output the text content in the image.

10. A method for license plate enlargement recognition based on overall and character region detection according to claim 1, characterized in that, The method of filtering and merging text content in an image through a preset verification module, verifying the text content according to the license plate number encoding rules, and outputting the final verified license plate number includes: The text information is divided into different groups through a preset verification module. The text information in each group is then judged whether to be filtered, merged, and output based on the encoding rules of the license plate number. The design method for calculating the encoding rule score is detailed in the following formula: Score(a)=max(|Len(a)-7|+I a ,|Len(a)-8|+I a ) Where Score(a) represents the encoding rule score of the input text information; let a be a text string, Len(a) represent the string length, and I a As an indicator variable, I is defined when 'a' does not conform to the license plate coding rules. a Set to 1 otherwise set to 0; The absolute value of the difference between the current text length and the target text length, plus the indicator variable of whether the current text meets the license plate coding rules, is used as a metric. Score(a) uses the maximum value of this metric as the score of the text with respect to the license plate coding rules; The threshold for the rule score is determined by actual needs, and the verified area information will be output as the final result after filtering out false characters. Filtering the text content in the image, specifically: The improved SVTR text recognition model outputs text content and bounding boxes in a list. The encoding rule score of each text and the height, width and center coordinates of the corresponding bounding box are calculated. The input text list is traversed, and the encoding rule scores of adjacent text sequence combinations are calculated. The text sequence combinations with the highest scores are merged, sorted and returned. The text sequence information that meets the minimum output threshold is returned. By judging the overall location of the merged license plate enlargement number due to excessively large merging gaps based on the scoring of the encoding rules, as well as the overall location of other text regions that may belong to the same text line that are not merged, and filtering out false detection characters with too low scores, the final output is the license plate enlargement number and standard motor vehicle license plate number information.

Images