License plate amplified number identification method based on overall and character region detection

Abstract

The invention discloses a license plate amplified number recognition method based on overall and character region detection. The method comprises the steps that an improved DBNet overall region detection model, an improved CRAFT character region detection model and an improved SVTR text recognition model are constructed through a license plate amplified number training set and a scene text public data set; the method comprises the following steps: detecting amplified number test set data through an improved DBNet and CRAFT model to obtain an amplified number overall area and all character text areas, then merging through a communication module to obtain amplified number overall positioning and positioning of other text areas possibly belonging to the same text line, and then performing identification through an improved SVTR model to obtain an amplified number overall positioning result. Obtaining text content in the image; and the text content in the image is screened, merged and verified through a preset verification module, and the final license plate number after verification is output. According to the invention, high-precision license plate amplified number identification is realized.

Description

Technical Field

[0001] The invention relates to the field of intelligent transportation, and in particular to a method for recognizing enlarged license plate numbers based on overall and character region detection. Background Art

[0002] The license plate number is the most important identity information and unique identifier of a vehicle. The construction of an urban intelligent transportation system requires accurate and rapid detection and identification of license plates. The existing detection and identification technologies for most blue license plates, yellow license plates and new energy license plates in China are relatively mature, but these methods are prone to misdetection and missed detection of medium and large vehicles such as trucks and muck trucks due to shooting angles, lighting conditions, license plate positions, license plate stains, etc. For example: the compartment of a truck is likely to hinder the detection and identification of the license plate at the rear of the vehicle by the bayonet camera at a high-position overlooking angle. According to Article 13 of the "Regulations for the Implementation of the Road Traffic Safety Law of the People's Republic of China": The body or rear of the compartment of heavy-duty and medium-duty trucks and their trailers, tractors and their trailers shall be sprayed with enlarged license plates (referred to as license plate enlargement numbers). Since the size and position of the license plate enlargement number are more obvious than the standard license plate, detecting and identifying the vehicle's license plate enlargement number is conducive to better obtaining the vehicle's identity information, thereby reducing the problem of misdetection and missed detection.

[0003] In the field of intelligent transportation, standard license plates have specified sizes, colors, significant boundaries and character styles. The existing technology locates based on these characteristics, and then obtains license plate information by identifying the segmented characters or the overall sequence in the positioning area. The current license plate detection and recognition methods are mainly based on deep learning. The mainstream open source systems include EasyPR and HyperLPR, etc. These methods have been widely used in practical problems. However, the existing technology only detects and recognizes standard license plates. Since the enlarged license plate number does not have the above-mentioned specified characteristics, such as the color of the carriage sprayed with the enlarged number is not fixed, the enlarged number character spacing is large, the style is many, and there may be other noise interference such as advertisements on the carriage, resulting in the general license plate detection and recognition method and the scene text identification method cannot accurately locate or locate incompletely. In addition, due to the small data sample size and poor scene targeting effect, how to more accurately determine the complete area of ​​the enlarged license plate number in the carriage, solve the problem of inconsistent character spacing, and how to accurately identify the license plate information for the positioning frame and eliminate false detection information are still urgent problems to be solved.

[0004] Therefore, it is necessary to make targeted improvements to the above problems existing in the prior art. The detection and recognition method based on the whole and character area of ​​the present invention can more effectively realize the license plate recognition task for the enlarged license plate number, thereby providing further technical support for the identification of medium and large vehicles in the field of smart transportation.

[0005] The Chinese invention application with application number: 202011532814.1 discloses "A method for detecting and identifying enlarged license plates". Its technical solution is: detect and locate the area where the enlarged license plate number is located to obtain a sample image of the original enlarged license plate number; recognize the characters of the enlarged license plate number based on a deep convolutional recognition network, expand the sample image of the original enlarged license plate number in the training stage to obtain a training sample set, and then build a recognition network and perform feature extraction on the actual enlarged license plate number image to obtain the final text recognition result. Summary of the invention

[0006] In order to solve the technical problem of insufficient accuracy in the prior art of enlarged license plate number recognition, the present invention provides an enlarged license plate number recognition method based on overall and character area detection. The technical solution adopted by the present invention is:

[0007] A method for recognizing enlarged license plate numbers based on overall and character region detection, the method comprising:

[0008] Constructing a license plate enlargement number dataset and a scene text public dataset respectively, wherein the license plate enlargement number dataset includes a license plate enlargement number training set and a license plate enlargement number test set;

[0009] The improved DBNet overall region detection model, the improved CRAFT character region detection model, and the improved SVTR text recognition model were constructed using the license plate enlargement training set and the scene text public dataset.

[0010] The improved DBNet enlarged license plate number overall area detection model is used to detect the enlarged license plate number test set data image, and the overall area of ​​the enlarged license plate number in the image is obtained;

[0011] The improved CRAFT license plate enlargement character region detection model is used to detect the license plate enlargement test set data images and obtain all the character text regions in the image;

[0012] The entire area of ​​the enlarged license plate number in the image and all the character text areas in the image are merged through a preset connection module to obtain the overall location of the merged enlarged license plate number and the locations of other text areas that may belong to the same text line;

[0013] By improving the SVTR text recognition model, the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning is identified to obtain the text content in the image;

[0014] The text content in the image is filtered and merged through the preset verification module, the text content is verified according to the license plate number encoding rules, and the final license plate number after verification is output.

[0015] As a preferred solution, the scene text public data set includes an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, and a text recognition training set and a text recognition verification set.

[0016] As a preferred solution, the method of constructing an improved DBNet overall region detection model, an improved CRAFT character region detection model, and an improved SVTR text recognition model respectively through the license plate enlargement training set and the scene text public data set includes:

[0017] The preset improved DBNet enlarged license plate overall area detection network is iteratively trained using the overall detection training set, the accuracy of the model is verified using the overall detection verification set, the optimal model is saved and retrained using the enlarged license plate training set, and finally the improved DBNet overall area detection model is obtained;

[0018] The preset improved CRAFT license plate enlarged number character region detection network is iteratively trained using the character detection training set, the accuracy of the model is verified using the character detection verification set, the optimal model is saved and retrained using the enlarged number training set, and finally the improved CRAFT character region detection model is obtained;

[0019] The constructed improved SVTR license plate enlarged number text recognition network is iteratively trained using the text recognition training set, and the accuracy of the model is verified using the text recognition verification set. The optimal model is saved and retrained using the enlarged number training set, and finally the improved SVTR text recognition model is obtained.

[0020] As a preferred solution, the improved DBNet overall region detection model includes a first feature extraction module, a feature fusion module, a first classification prediction module, a first binarization module and an overall region output module connected in sequence;

[0021] The first feature extraction module is used to extract features from the license plate enlargement test set data image to obtain a first primary feature map;

[0022] The feature fusion module is used to obtain several feature maps of different sizes by subjecting the primary feature map to four 1×1 convolutions, and then upsampling from top to bottom, and horizontally connecting the feature maps of the same size generated from bottom to top; the connected feature maps are then subjected to four 3×3 convolutions to eliminate the aliasing effect of upsampling, and finally a multi-scale feature representation is obtained and connected to obtain a first fused feature map;

[0023] The first classification prediction module is used to input the first fusion feature map into a preset first convolution classification network to obtain a target probability map and a threshold map;

[0024] The first binarization module is used to generate a first approximate binary image by using a differentiable binarization method on the target probability image and the threshold image;

[0025] The overall area output module is used to obtain the vertex coordinates of the regional polygon according to the first approximate binary image, and then restore these coordinates to the original image size to obtain the overall area of ​​the enlarged license plate number;

[0026] The first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer;

[0027] The first stage layer includes 3 residual units, and the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation;

[0028] The second stage layer includes 3 residual units, each of which has a structure of 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0029] The third stage layer includes 6 residual units, wherein the structures of the first 4 residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, 1×1 convolution, batch normalization and Hard Swish activation function, and the structures of the last two residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, SE module, 1×1 convolution, batch normalization and Hard Swish activation function;

[0030] The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0031] The feature fusion module is a feature pyramid network;

[0032] The first classification prediction module includes a first convolution classification network, the first convolution classification network includes a target probability map output module and a threshold map output module, the target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution;

[0033] The loss function of the improved DBNet overall region detection model is the prediction probability loss L s , Binarization loss L b And the target area distance loss L t The weighted sum of s and L b Using cross entropy loss, L t Use L1 norm loss.

[0034] As a preferred solution, the first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer and a fourth stage layer;

[0035] The first stage layer includes 3 residual units, and the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation;

[0036] The second stage layer includes 3 residual units, each of which has a structure of 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0037] The third stage layer includes 6 residual units, wherein the structures of the first 4 residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, 1×1 convolution, batch normalization and Hard Swish activation function, and the structures of the last two residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, SE module, 1×1 convolution, batch normalization and Hard Swish activation function;

[0038] The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0039] The feature fusion module is a feature pyramid network;

[0040] The first classification prediction module includes a first convolution classification network, the first convolution classification network includes a target probability map output module and a threshold map output module, the target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution;

[0041] The loss function of the improved DBNet overall region detection model is the prediction probability loss L s , Binarization loss L b And the target area distance loss L t The weighted sum of s and L b Using cross entropy loss, L t Use L1 norm loss.

[0042] As a preferred solution, the improved CRAFT license plate enlarged number character area detection model includes a second feature extraction module, a deconvolution module, a second classification prediction module, a second binarization module and a character area output module connected in sequence;

[0043] The second feature extraction module is used to extract features from the license plate enlargement test set data image to obtain a second primary feature map;

[0044] The deconvolution module is used to perform deconvolution on the second primary feature map to obtain a deconvolution feature map;

[0045] The second classification prediction module is used to input the deconvolution feature map into a preset second convolution classification network to obtain the area score of the corresponding character and the connection score between the characters;

[0046] The second binarization module is used to connect and expand the region scores of the characters and the connection scores between the characters by using a fixed threshold method to generate a second approximate binary graph;

[0047] The character region output module is used to obtain the vertex coordinates of the region polygon according to the second approximate binary image, and then restore these coordinates to the original image size to output the target character region.

[0048] As a preferred solution, the second feature extraction module is VGG16_BN, which includes a first slice layer, a second slice layer, a third slice layer, a fourth slice layer and a fifth slice layer;

[0049] The first slice layer includes 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0050] The second slice layer includes ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0051] The third slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0052] The fourth slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0053] The fifth slice layer includes maximum pooling, 3×3 convolution and 1×1 convolution;

[0054] The deconvolution module includes a first deconvolution module, a second deconvolution module, a third deconvolution module, and a fourth deconvolution module;

[0055] The fifth slice layer is connected to the output skip layer of the fourth slice layer, and then connected to the output skip layer of the third slice layer through the first deconvolution module, and then connected to the output skip layer of the second slice layer through the second deconvolution module, and then connected to the output skip layer of the first slice layer through the third deconvolution module, and finally connected to the input of the fourth deconvolution module;

[0056] The second classification prediction module includes a second convolution classification network, which includes three 3×3 convolutions and two 1×1 convolutions connected in sequence, wherein except for the last 1×1 convolution, ReLU activation is used after other convolution layers;

[0057] The loss function of the improved CRAFT license plate enlarged number character region detection model is the weighted sum of the L2 norm loss corresponding to the character region score and the connection score between characters.

[0058] As a preferred solution, the method of merging the entire area of ​​the enlarged license plate number in the image and all the character text areas in the image through a preset connection module to obtain the overall positioning of the merged enlarged license plate number and the positioning of other text areas that may belong to the same text line includes:

[0059] Determine whether the vehicle has been painted with an enlarged license plate number based on the overall area of ​​the enlarged license plate number output by the improved DBNet overall area detection model;

[0060] The entire area of ​​the enlarged license plate number is divided into different groups through the preset connectivity module, so that the positioning boxes in each group are considered to belong to the same text sequence under the condition that the similarity score is greater than the specified threshold. The similarity score calculation formula is as follows:

[0061]

[0062] Among them, Shape(u,v) represents the size similarity of different positioning boxes, Coord(u,v) represents the similarity of the vertical coordinate values ​​of the center points of different positioning boxes, h(u) and y(u) are set as the box height and vertical coordinate of a certain positioning box u, h(v) and y(v) represent the box height and vertical coordinate of another positioning box v; Shape(u,v) uses the normalized ratio of the box height difference of different positioning boxes to their mean as a measure of whether the text sizes of the two are similar, and Coord(u,v) uses the normalized ratio of the vertical coordinate height difference of different positioning boxes to their mean box height as a measure of whether the vertical coordinate values ​​of the center points of the two texts are similar;

[0063] Merge the overall area of ​​the enlarged license plate number in the image and all the character text areas in the image, specifically:

[0064] First, use the positioning boxes in the overall area of ​​the enlarged license plate number and all the character text areas, and calculate the height, width and center coordinates of each positioning box at the same time; traverse the input positioning box list, and when the Shape(u,v) and Coord(u,v) of different positioning boxes u and v simultaneously meet the specified similarity score threshold, mark them as the same group; merge the text boxes marked as the same group, that is, arrange the horizontal coordinates of the center points of all text boxes in the same group from left to right, and finally output the connected positioning boxes of the upper left and lower left coordinates of the first box and the upper right and lower right coordinates of the last box as the four vertices of this area; based on the threshold judgment of the similarity score, the similar results of the character area can be merged, and the setting of different thresholds indicates that the connectivity method allows the characters of the same text sequence to have size differences and upper and lower floating ranges, thereby fully locating the long text of the research object.

[0065] As a preferred solution, the improved SVTR text recognition model includes a third feature extraction module, an encoding module, a transcription module, a decoding module and a text content output module;

[0066] The third feature extraction module is used to extract features from the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning to obtain a third primary feature map;

[0067] The encoding module includes an Im2Seq module and a MultiHead module; the Im2Seq module is used to transpose the third primary feature map channel into a sequence, and then downsample it using 2 layers of convolution and normalization to obtain a patch, and the Im2Seq module includes 3×3 convolution, batch normalization and Swish activation, 1×1 convolution, batch normalization and Swish activation;

[0068] The core global mixing block in the MultiHead module is Global Mixing, which is used to perceive the correlation between characters and between characters and background. Specifically:

[0069]

[0070] Among them, d k is the dimension, Q, K, V are obtained by linear transformation of the input image region matrix, Q and K T After matrix multiplication, Softmax normalization is performed to obtain the Attention matrix, which is then multiplied with the V matrix and then output by linear transformation. After each linear transformation, 10% of the neurons are randomly dropped to reduce overfitting.

[0071] The MultiHead module is used to input the patch features into the hybrid block for layer normalization, and then perceive the correlation between characters through the Attention method. After the output of the Attention method is subjected to identity mapping and layer normalization, the features enter the multi-layer perceptron MLP for fusion, and finally obtain the second fused feature map. The specific expression of the Attention method is:

[0072]

[0073] Among them, d k is the dimension, Q, K, V are obtained by linear transformation of the input image region matrix, Q and K T After matrix multiplication, Softmax normalization is performed to obtain the Attention matrix, which is then multiplied with the V matrix and then output by linear transformation. After each linear transformation, 10% of the neurons are randomly dropped to reduce overfitting.

[0074] The implementation path of the MLP is: fully connected layer and Swish activation, fully connected layer and Dropout;

[0075] The transcription module is used to perform a layer normalization operation on the second fused feature map, and then use three layers of convolution and batch normalization to merge features to obtain a merged feature map; wherein the three layers of convolution and batch normalization include 1×1 convolution, batch normalization and Swish activation, 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation;

[0076] The decoding module is used to obtain the confidence of each character prediction in the text by using the Softmax method on the merged feature map;

[0077] The text content output module is used to mark the probability of non-target encoded characters as 0, use CTC decoding mapping according to the confidence of each character prediction in the text, and output the text content in the image.

[0078] As a preferred solution, the method of filtering and merging the text content in the image by a preset verification module, verifying the text content according to the license plate number encoding rule, and outputting the final license plate number after verification includes:

[0079] The text information is divided into different groups through the preset verification module, so that the text information in each group is judged whether to be filtered, merged and output according to the coding rule score of the license plate number; the design method of coding rule score calculation is detailed in the following formula:

[0080] Score(a)=max(|Len(a)-7|+I a ,|Len(a)-8|+I a )

[0081] Where Score(a) represents the coding rule score of the input text information; set a to be the text string, Len(a) to be the length of the string, and I a is an indicative variable. It is defined that when a does not meet the license plate coding rules, the value is 1, otherwise it is 0. The absolute value of the difference between the current text length and the target text length plus the indicative variable of whether the current text meets the license plate coding rules is used as a metric. Score(a) uses the maximum value of this metric as the score of the text with respect to the license plate coding rules. The threshold of the rule score is determined by actual needs. The verified area information will be output as the final result after filtering out the falsely detected characters.

[0082] Filter the text content in the image, specifically:

[0083] Use a list to save the text content and positioning box output by the improved SVTR text recognition model, calculate the encoding rule score of each text and the height, width and center coordinates of the corresponding positioning box, etc.; traverse the input text list, then calculate the encoding rule scores of adjacent text sequence combinations, merge the text sequence combinations with the maximum scores, sort and return the text sequence information that meets the minimum output threshold;

[0084] Based on the threshold judgment of the coding rule score, it is possible to merge the overall positioning of the merged enlarged license plate number due to excessive spacing, as well as the overall areas that are not merged in the positioning of other text areas that may belong to the same text line. After screening out the misdetected characters with too low scores, more accurate enlarged license plate number and standard motor vehicle license plate number information can be finally output.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] Based on the three models of DBNet, CRAFT and SVTR, the present invention constructs a DBNet-based overall area detection network, a CRAFT-based character area detection network, and a SVTR-based text recognition network according to the characteristics of the enlarged license plate. Through the connectivity method, the merging of the positioning results of different regions is achieved to solve the problem of difficult or incomplete positioning due to the large spacing between the characters of the enlarged license plate; through the verification method, other text noise interference on the carriage is further screened out.

[0087] Based on the example data, the present invention uses the output of the enlarged license plate overall area detection network as a classification result, thereby realizing the judgment of whether there is an enlarged license plate number in the image. The text recognition network adopts perspective transformation to solve the problem of high-position overhead deformation that may exist in the image in general monitoring scenes. Compared with the general license plate detection and recognition method and the enlarged license plate number detection and recognition method, the present invention proposes a method of cascading the overall and character area detection networks, which can ultimately output the positioning results of the enlarged license plate number and the standard motor vehicle license plate at the same time.

[0088] The method designed by the present invention is cascaded, and can be applied to other specific scenarios by replacing different modules therein, such as the overall detection network, the connectivity module and the verification module. Therefore, the method designed by the present invention has better accuracy, robustness and portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic flow chart of a method for recognizing an enlarged license plate number based on overall and character region detection provided in this embodiment;

[0090] Figure 2 A training flow chart of a method for recognizing enlarged license plate numbers based on overall and character region detection provided in this embodiment;

[0091] Figure 3 A schematic diagram of the overall area detection network for the enlarged license plate number provided in this embodiment;

[0092] Figure 4 A schematic diagram of a network for detecting the character area of ​​the enlarged license plate provided in this embodiment;

[0093] Figure 5 A schematic diagram of a text recognition network for the enlarged license plate provided in this embodiment;

[0094] Figure 6 This is a schematic diagram of a DSC module provided in this embodiment. DETAILED DESCRIPTION

[0095] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention;

[0096] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0097] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0098] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0099] In addition, in the description of this application, unless otherwise specified, "multiple" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0100] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0101] Example 1

[0102] Please refer to Figure 1 This embodiment provides a method for recognizing a magnified license plate number based on overall and character region detection, the method comprising:

[0103] Constructing a license plate enlargement number dataset and a scene text public dataset respectively, wherein the license plate enlargement number dataset includes a license plate enlargement number training set and a license plate enlargement number test set;

[0104] The improved DBNet overall region detection model, the improved CRAFT character region detection model, and the improved SVTR text recognition model were constructed using the license plate enlargement training set and the scene text public dataset.

[0105] The improved DBNet enlarged license plate number overall area detection model is used to detect the enlarged license plate number test set data image, and the overall area of ​​the enlarged license plate number in the image is obtained;

[0106] The improved CRAFT license plate enlargement character region detection model is used to detect the license plate enlargement test set data images and obtain all the character text regions in the image;

[0107] The entire area of ​​the enlarged license plate number in the image and all the character text areas in the image are merged through a preset connection module to obtain the overall location of the merged enlarged license plate number and the locations of other text areas that may belong to the same text line;

[0108] By improving the SVTR text recognition model, the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning is identified to obtain the text content in the image;

[0109] The text content in the image is filtered and merged through the preset verification module, the text content is verified according to the license plate number encoding rules, and the final license plate number after verification is output.

[0110] In a specific embodiment, the method comprises:

[0111] Constructing a license plate enlargement number dataset and a scene text public dataset respectively, wherein the license plate enlargement number dataset includes a license plate enlargement number training set and a license plate enlargement number test set;

[0112] The improved DBNet overall region detection model, the improved CRAFT character region detection model, and the improved SVTR text recognition model were constructed using the license plate enlargement training set and the scene text public dataset.

[0113] The improved DBNet enlarged license plate number overall area detection model is used to detect the enlarged license plate number test set data image, and the overall area of ​​the enlarged license plate number in the image is obtained;

[0114] The improved CRAFT license plate enlargement character region detection model is used to detect the license plate enlargement test set data images and obtain all the character text regions in the image;

[0115] The entire area of ​​the enlarged license plate number in the image and all the character text areas in the image are merged through a preset connection module to obtain the overall location of the merged enlarged license plate number and the locations of other text areas that may belong to the same text line;

[0116] By improving the SVTR text recognition model, the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning is identified to obtain the text content in the image;

[0117] The text content in the image is filtered and merged through the preset verification module, the text content is verified according to the license plate number encoding rules, and the final license plate number after verification is output.

[0118] In a specific embodiment, the scene text public data set includes an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, a text recognition training set and a text recognition verification set.

[0119] In a specific embodiment, please refer to Figure 2 , the method of constructing an improved DBNet overall region detection model, an improved CRAFT character region detection model, and an improved SVTR text recognition model respectively through the license plate enlargement training set and the scene text public data set includes:

[0120] The preset improved DBNet enlarged license plate overall area detection network is iteratively trained using the overall detection training set, the accuracy of the model is verified using the overall detection verification set, the optimal model is saved and retrained using the enlarged license plate training set, and finally the improved DBNet overall area detection model is obtained;

[0121] The preset improved CRAFT license plate enlarged number character region detection network is iteratively trained using the character detection training set, the accuracy of the model is verified using the character detection verification set, the optimal model is saved and retrained using the enlarged number training set, and finally the improved CRAFT character region detection model is obtained;

[0122] The constructed improved SVTR license plate enlarged number text recognition network is iteratively trained using the text recognition training set, and the accuracy of the model is verified using the text recognition verification set. The optimal model is saved and retrained using the enlarged number training set, and finally the improved SVTR text recognition model is obtained.

[0123] In a specific embodiment, please refer to Figure 3 The improved DBNet overall region detection model includes a first feature extraction module, a feature fusion module, a first classification prediction module, a first binarization module and an overall region output module connected in sequence;

[0124] The first feature extraction module is used to extract features from the license plate enlargement test set data image to obtain a first primary feature map;

[0125] The feature fusion module is used to obtain several feature maps of different sizes by subjecting the primary feature map to four 1×1 convolutions, and then upsampling from top to bottom, and horizontally connecting the feature maps of the same size generated from bottom to top; the connected feature maps are then subjected to four 3×3 convolutions to eliminate the aliasing effect of upsampling, and finally a multi-scale feature representation is obtained and connected to obtain a first fused feature map;

[0126] The first classification prediction module is used to input the first fusion feature map into a preset first convolution classification network to obtain a target probability map and a threshold map;

[0127] The first binarization module is used to generate a first approximate binary image by using a differentiable binarization method on the target probability image and the threshold image;

[0128] The overall area output module is used to obtain the vertex coordinates of the regional polygon according to the first approximate binary image, and then restore these coordinates to the original image size to obtain the overall area of ​​the enlarged license plate number;

[0129] In a specific embodiment, please refer to Figure 3 , the first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer;

[0130] The first stage layer includes 3 residual units, and the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation;

[0131] The second stage layer includes 3 residual units, each of which has a structure of 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0132] The third stage layer includes 6 residual units, wherein the structures of the first 4 residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, 1×1 convolution, batch normalization and Hard Swish activation function, and the structures of the last two residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, SE module, 1×1 convolution, batch normalization and Hard Swish activation function;

[0133] The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation;

[0134] The feature fusion module is a feature pyramid network;

[0135] The first classification prediction module includes a first convolution classification network, the first convolution classification network includes a target probability map output module and a threshold map output module, the target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution;

[0136] The loss function of the improved DBNet overall region detection model is the prediction probability loss L s , Binarization loss L b And the target area distance loss L t The weighted sum of s and L b Using cross entropy loss, L t Use L1 norm loss.

[0137] In a specific embodiment, please refer to Figure 4 The improved CRAFT license plate enlarged number character area detection model includes a second feature extraction module, a deconvolution module, a second classification prediction module, a second binarization module and a character area output module connected in sequence;

[0138] The second feature extraction module is used to extract features from the license plate enlargement test set data image to obtain a second primary feature map;

[0139] The deconvolution module is used to perform deconvolution on the second primary feature map to obtain a deconvolution feature map;

[0140] The second classification prediction module is used to input the deconvolution feature map into a preset second convolution classification network to obtain the area score of the corresponding character and the connection score between the characters;

[0141] The second binarization module is used to connect and expand the region scores of the characters and the connection scores between the characters by using a fixed threshold method to generate a second approximate binary graph;

[0142] The character region output module is used to obtain the vertex coordinates of the region polygon according to the second approximate binary image, and then restore these coordinates to the original image size to output the target character region.

[0143] In a specific embodiment, please refer to Figure 4 , the second feature extraction module is VGG16_BN, which includes a first slice layer, a second slice layer, a third slice layer, a fourth slice layer and a fifth slice layer;

[0144] The first slice layer includes 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0145] The second slice layer includes ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0146] The third slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0147] The fourth slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization;

[0148] The fifth slice layer includes maximum pooling, 3×3 convolution and 1×1 convolution;

[0149] The deconvolution module includes a first deconvolution module, a second deconvolution module, a third deconvolution module, and a fourth deconvolution module;

[0150] The fifth slice layer is connected to the output skip layer of the fourth slice layer, and then connected to the output skip layer of the third slice layer through the first deconvolution module, and then connected to the output skip layer of the second slice layer through the second deconvolution module, and then connected to the output skip layer of the first slice layer through the third deconvolution module, and finally connected to the input of the fourth deconvolution module;

[0151] The second classification prediction module includes a second convolution classification network, which includes three 3×3 convolutions and two 1×1 convolutions connected in sequence, wherein except for the last 1×1 convolution, ReLU activation is used after other convolution layers;

[0152] The loss function of the improved CRAFT license plate enlarged number character region detection model is the weighted sum of the L2 norm loss corresponding to the character region score and the connection score between characters.

[0153] In a specific embodiment, the method of merging the entire area of ​​the enlarged license plate number in the image and all the character text areas in the image through a preset connection module to obtain the overall positioning of the merged enlarged license plate number and the positioning of other text areas that may belong to the same text line includes:

[0154] Determine whether the vehicle has been painted with an enlarged license plate number based on the overall area of ​​the enlarged license plate number output by the improved DBNet overall area detection model;

[0155] The entire area of ​​the enlarged license plate number is divided into different groups through the preset connectivity module, so that the positioning boxes in each group are considered to belong to the same text sequence under the condition that the similarity score is greater than the specified threshold. The similarity score calculation formula is as follows:

[0156]

[0157] Among them, Shape(u,v) represents the size similarity of different positioning boxes, Coord(u,v) represents the similarity of the vertical coordinate values ​​of the center points of different positioning boxes, h(u) and y(u) are set as the box height and vertical coordinate of a certain positioning box u, h(v) and y(v) represent the box height and vertical coordinate of another positioning box v; Shape(u,v) uses the normalized ratio of the box height difference of different positioning boxes to their mean as a measure of whether the text sizes of the two are similar, and Coord(u,v) uses the normalized ratio of the vertical coordinate height difference of different positioning boxes to their mean box height as a measure of whether the vertical coordinate values ​​of the center points of the two texts are similar;

[0158] Merge the overall area of ​​the enlarged license plate number in the image and all the character text areas in the image, specifically:

[0159] First, use the positioning boxes in the overall area of ​​the enlarged license plate number and all the character text areas, and calculate the height, width and center coordinates of each positioning box at the same time; traverse the input positioning box list, and when the Shape(u,v) and Coord(u,v) of different positioning boxes u and v simultaneously meet the specified similarity score threshold, mark them as the same group; merge the text boxes marked as the same group, that is, arrange the horizontal coordinates of the center points of all text boxes in the same group from left to right, and finally output the connected positioning boxes of the upper left and lower left coordinates of the first box and the upper right and lower right coordinates of the last box as the four vertices of this area; based on the threshold judgment of the similarity score, the similar results of the character area can be merged, and the setting of different thresholds indicates that the connectivity method allows the characters of the same text sequence to have size differences and upper and lower floating ranges, thereby fully locating the long text of the research object.

[0160] In a specific embodiment, please refer to Figure 5 , the improved SVTR text recognition model includes a third feature extraction module, an encoding module, a transcription module, a decoding module and a text content output module;

[0161] The third feature extraction module is used to extract features from the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning to obtain a third primary feature map;

[0162] Specifically, the third feature extraction module is LCNet.

[0163] LCNet is an optimization of MobileNetV1. First, there is a 3×3 convolution and batch normalization operation with the activation function Hard Swish. Second, there are 13 layers of depth-wise separable convolution (DSC).

[0164] The first 6 DSCs consist of two steps. In the first step, 3×3 convolution, batch normalization, and Hard Swish activation are used to implement channel-by-channel convolution (also known as depthwise convolution, DC) to reduce model parameters. In the second step, 1×1 convolution, batch normalization, and Hard Swish activation are used to implement point-by-point convolution (PC) to deepen the model channel. The 7th and subsequent DSCs also contain the above two steps, but at this time the 3×3 convolution in the depthwise convolution (DC) is replaced by a 5×5 convolution. In the last 2 layers of DSC, the SE module is connected between DC and PC.

[0165] One pass of average pooling is used after all the above DSCs.

[0166] The encoding module includes an Im2Seq module and a MultiHead module; the Im2Seq module is used to transpose the third primary feature map channel into a sequence, and then downsample it using 2 layers of convolution and normalization to obtain a patch, and the Im2Seq module includes 3×3 convolution, batch normalization and Swish activation, 1×1 convolution, batch normalization and Swish activation;

[0167] The core global mixing block in the MultiHead module is Global Mixing, which is used to perceive the correlation between characters and between characters and background. Specifically:

[0168]

[0169] Among them, d k is the dimension, Q, K, V are obtained by linear transformation of the input image region matrix, Q and K T After matrix multiplication, Softmax normalization is performed to obtain the Attention matrix, which is then multiplied with the V matrix and then output by linear transformation. After each linear transformation, 10% of the neurons are randomly dropped to reduce overfitting.

[0170] The MultiHead module is used to input the patch features into the hybrid block for layer normalization, and then perceive the correlation between characters through the Attention method. After the output of the Attention method is subjected to identity mapping and layer normalization, the features enter the multi-layer perceptron MLP for fusion, and finally obtain the second fused feature map. The specific expression of the Attention method is:

[0171]

[0172] Among them, d k is the dimension, Q, K, V are obtained by linear transformation of the input image region matrix, Q and K TAfter matrix multiplication, Softmax normalization is performed to obtain the Attention matrix, which is then multiplied with the V matrix and then output by linear transformation. After each linear transformation, 10% of the neurons are randomly dropped to reduce overfitting.

[0173] The implementation path of the MLP is: fully connected layer and Swish activation, fully connected layer and Dropout;

[0174] The transcription module is used to perform a layer normalization operation on the second fused feature map, and then use three layers of convolution and batch normalization to merge features to obtain a merged feature map; wherein the three layers of convolution and batch normalization include 1×1 convolution, batch normalization and Swish activation, 3×3 convolution, batch normalization and Swish activation, and 1×1 convolution, batch normalization and Swish activation;

[0175] The decoding module is used to obtain the confidence of each character prediction in the text by using the Softmax method on the merged feature map;

[0176] The text content output module is used to mark the probability of non-target encoded characters as 0, use CTC decoding mapping according to the confidence of each character prediction in the text, and output the text content in the image;

[0177] Specifically, the improved SVTR text recognition model uses the weighted sum of CTCLoss and CrossEntropyLoss as the loss function of the network.

[0178] In a specific embodiment, the method of filtering and merging the text content in the image by a preset verification module, verifying the text content according to the license plate number encoding rule, and outputting the final license plate number after verification includes:

[0179] The text information is divided into different groups through the preset verification module, so that the text information in each group is judged whether to be filtered, merged and output according to the coding rule score of the license plate number; the design method of coding rule score calculation is detailed in the following formula:

[0180] Score(a)=max(|Len(a)-7|+I a ,|Len(a)-8|+I a )

[0181] Where Score(a) represents the coding rule score of the input text information; set a to be the text string, Len(a) to be the length of the string, and I eis an indicative variable. It is defined that when a does not meet the license plate coding rules, its value is 1, otherwise it is 0; the absolute value of the difference between the current text length and the target text length plus the indicative variable of whether the current text meets the license plate coding rules is used as a metric, and Score(a) uses the maximum value of this metric as the score of the text with respect to the license plate coding rules; the threshold of the rule score is determined by actual needs, and the verified area information will be output as the final result after filtering out the falsely detected characters.

[0182] Filter the text content in the image, specifically:

[0183] Use a list to save the text content and positioning box output by the improved SVTR text recognition model, calculate the encoding rule score of each text and the height, width and center coordinates of the corresponding positioning box, etc. Traverse the input text list, then calculate the encoding rule scores of adjacent text sequence combinations, merge the text sequence combinations with the maximum scores, sort and return the text sequence information that meets the minimum output threshold;

[0184] Based on the threshold judgment of the coding rule score, it is possible to merge the overall positioning of the merged enlarged license plate number due to excessive spacing, as well as the overall areas that are not merged in the positioning of other text areas that may belong to the same text line. After screening out the misdetected characters with too low scores, more accurate enlarged license plate number and standard motor vehicle license plate number information can be finally output.

[0185] Example 2

[0186] This embodiment can be regarded as an improved or extended embodiment based on the embodiment 1, specifically: a method for recognizing enlarged license plate numbers based on overall and character area detection, the method comprising:

[0187] Constructing a license plate enlargement number dataset and a scene text public dataset respectively, wherein the license plate enlargement number dataset includes a license plate enlargement number training set and a license plate enlargement number test set;

[0188] Specifically, the enlarged license plate number dataset includes images of the rear of a car with and without enlarged license plates in street scenes. The enlarged license plate number data are named by numbers and divided into an enlarged license plate number training set and an enlarged license plate number test set. Public scene text datasets in general scenarios are collected for pre-training models, which are divided into an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, and a text recognition training set and a text recognition verification set.

[0189] Specifically, license plate magnifier images are usually obtained from road or checkpoint cameras and on-site shooting channels. This dataset includes a total of 1,663 rear-view images of medium and large-sized vehicles that may contain license plate magnifiers in real scenarios, such as publicly available license plate image databases and manually collected images. The files are saved with the.jpg suffix, and the resolution sizes vary. It covers samples that are relatively difficult to detect and recognize obtained in environments including extreme weather, poor exposure, non-rigid spraying surfaces, and damaged magnifiers. Among them, there are 1,404 positive sample images containing license plate magnifiers, and 259 background sample images that only contain medium and large-sized vehicles but do not contain license plate magnifiers.

[0190] Specifically, PaddleLabel is used for manual text bounding box annotation. The specific method is to select the position of the entire magnifier area in the image (only annotate the license plate magnifier and ignore the annotation of the general license plate); the text content of the image sample is annotated as the corresponding characters and numbers of the license plate magnifier; the difficulty level of image sample recognition is recorded as 1, indicating that it is a sample that is relatively difficult to detect and recognize; if the magnifier is clearly distinguishable, it is recorded as 0. The obtained label list is saved with the.txt suffix, and each.jpg image corresponds to a.txt. For example, the content of Z01882.txt contains two lines:

[0191] 947,477,1196,477,1196,532,947,532,0,"京KAY030"

[0192] 542,671,673,671,673,701,542,701,1,"京AAD276"

[0193] The above represents that there are two annotated license plate magnifiers in the image numbered "Z01882". "947,477", "1196,477", "1196,532", and "947,532" respectively represent the values corresponding to the upper left, upper right, lower right, and lower left coordinates of the license plate magnifier. "0" represents the difficulty level of recognition, and "京KAY030" is the corresponding license plate number information. In the second line, "542,671,673,671,673,701,542,701,1" represents the coordinate information and recognition difficulty level corresponding to the license plate magnifier "京AAD276".

[0194] Specifically, the original images do not require preprocessing. After reading in the names, the train_test_split method in the sklearn library in Python is used for random splitting of the training set and test set at a ratio of 8:2, and then they can be directly loaded through the cv2 library. The loaded images are uniformly converted into BGR three-channel numpy arrays.

[0195] Specifically, we obtained the open source dataset ICDAR and the public pre-trained models based on this dataset. We migrated the models to the PyTorch framework and conducted experiments on real datasets based on it.

[0196] The improved DBNet overall region detection model, the improved CRAFT character region detection model, and the improved SVTR text recognition model were constructed using the license plate enlargement training set and the scene text public dataset.

[0197] Specifically, the "pre-training-retraining" method is used to train the improved DBNet enlarged license plate overall area detection network, the improved CRAFT enlarged license plate character area detection network and the improved SVTR enlarged license plate text recognition network. In this embodiment, considering the small amount of real-scene enlarged license plate data, the public scene text public data set ICDAR is selected to pre-train the above model, and then retrained with the enlarged license plate data set collected in S1 to obtain the final model.

[0198] Specifically, the input size of the DBNet model in the detection phase is 1×3×640×W or 1×3×H×640, where H and W are multiples of 32; the maximum limit of H and W of the CRAFT model input size in the detection phase is 1280, and H and W are multiples of 64 and 32 respectively. Preprocessing operations are performed on the original image when it is input into the detection network, including scaling, normalization, and channel transposition.

[0199] The uniform size of the model in the recognition stage is 1×3×48×W, with no limit on W. Image processing is performed when the detected results are input into the recognition network, including cropping the image in the detection boundary box to obtain the image in the positioning box, returning the transformed rectangular image from a high-level top-down deformation, updating the width and height of the horizontal or non-horizontal image list to adapt to the network input size, and sorting by the vertical coordinate of the center point of the rectangular area after normalization.

[0200] Specifically, this embodiment randomly selects 80% of the license plate enlargement data set as the retraining set and 20% as the test set for numerical experiments. The initial learning rate is set to 0.001, the decay rate is set to 0.9, and the number of training rounds is set to 100. This embodiment is based on GPU (Nvidia Tesla T4), PyTorch, Paddle and CUDA framework.

[0201] The improved DBNet enlarged license plate number overall area detection model is used to detect the enlarged license plate number test set data images, and the overall area of ​​the enlarged license plate number in the image is obtained;

[0202] Specifically, the overall area detection model of the enlarged license plate number based on DBNet includes four steps: feature extraction, feature fusion, classification prediction and binarization. This embodiment uses the MobileNetV3 network to replace the traditional ResNet network as the backbone of the model. It introduces the SE module to improve the local attention of the network by strengthening the interdependence between convolutional feature channels. The specific structure of the improved DBNet network is shown in Figure 3 .

[0203] Specifically, combined Figure 3 The network structure of the feature data extraction path structure is:

[0204] [((C3×3_3_8,s=2×2,p=1×1),BN,HSwish)]

[0205] Among them, (C3×3_3_8,s=2×2,p=1×1) indicates a convolution (denoted by C) with a kernel size of 3×3, 3 input channels, 8 output channels, a stride (denoted by s) of 1, and a padding (denoted by p) of 1. BN indicates a batch normalization operation, and HSwish indicates that the activation function is Hard Swish.

[0206] Stage1:[((C1×1_8_8,s=1×1),BN,R),((C3×3_8_8,s=1×1,p=1×1,g=8),BN,R), ((C1×1_8_8,s=1×1),BN)],[((C1×1_8_32,s=1×1),BN,R),((C3×3_32_32,s=2×2 ,p=1×1,g=32),BN,R),((C1×1_32_16,s=1×1),BN)],[((C1×1_16_40,s=1×1),B N,R),((C3×3_40_40,s=1×1,p=1×1,g=40),BN,R),((C1×1_40_16,s=1×1),BN)].

[0207] Among them, ((C1×1_8_8,s=1×1),BN,R) represents a convolution with a kernel size of 1×1, 8 input channels, and 8 output channels, BN represents a batch normalization operation, and R represents the activation function is ReLU; the following ((C3×3_8_8,s=1×1,p=1×1,g=8),BN,R) represents a grouped convolution with a kernel size of 3×3, 8 input channels, 8 output channels, a stride of 1, a padding of 1, and 8 groupings after the above 1×1 convolution, batch normalization, and ReLU activation, BN represents a batch normalization operation, and R represents the activation function is ReLU; the following ((C1×1_8_8,s=1×1),BN) represents a convolution with a kernel size of 1×1, 8 input channels, 8 output channels, and a stride of 1, and BN represents a batch normalization operation; and so on. [((C1×1_8_8,s=1×1),BN,R),((C3×3_8_8,s=1×1,p=1×1,g=8),BN,R),((C1×1_8_8,s=1×1),BN)] indicates that this is a unit.

[0208] Stage2:[((C1×1_16_40,s=1×1),BN,R),((C5×5_40_40,s=2×2,p=2×2,g=40),BN,R),[Adaptive,((C1×1_40_10,s=1×1),R),((C1×1_10_40 ,s=1×1),HSigmoid)],((C1×1_40_24,s=1×1),BN)],[((C1×1_24_64,s=1×1),BN,R),((C5×5_64_64,s=1×1,p=2×2,g=64),BN,R),[Adaptive ,((C1×1_64_16,s=1×1),R),((C1×1_16_64,s=1×1),HSigmoid)],((C1×1_64_24,s=1×1),BN)],[((C1×1_24_64,s=1×1),BN,R),((C5×5_64 _64,s=1×1,p=2×2,g=64),BN,R),[Adaptive,((C1×1_64_16,s=1×1),R),((C1×1_16_64,s=1×1),HSigmoid)],((C1×1_64_24,s=1×1),BN)].

[0209] Among them, Adaptive represents the adaptive average pooling operation, HSigmoid represents the activation function is HardSigmoid, and the rest is the same as above. [Adaptive,((C1×1_40_10,s=1×1),R),((C1×1_10_40,s=1×1),HSigmoid)],((C1×1_40_24,s=1×1),BN)] represents the structure of a SE module.

[0210] Stage3:[((C1×1_24_120,s=1×1),BN,HSwish),((C3×3_120_120,s=2×2,p=1×1,g=120),BN,HSwish),((C1×1_12 0_40,s=1×1),BN)],[((C1×1_40_104,s=1×1),BN,HSwish),((C3×3_104_104,s=1×1,p=1×1,g=104),BN,HSwish) ,((C1×1_104_40,s=1×1),BN)],[((C1×1_40_96,s=1×1),BN,HSwish),((C3×3_96_96,s=1×1,p=1×1,g=96),BN,H Swish),((C1×1_96_40,s=1×1),BN)],[((C1×1_40_96,s=1×1),BN,HSwish),((C3×3_96_96,s=1×1,p=1×1,g=96), BN,HSwish),((C1×1_96_40,s=1×1),BN)],[((C1×1_40_240,s=1×1),BN,HSwish),((C3×3_240_240,s=1×1,p=1× 1,g=240),BN,HSwish),[Adaptive,((C1×1_240_60,s=1×1),R),((C1×1_60_240,s=1×1),HSigmoid)],((C1×1_2 40_56,s=1×1),BN)],[((C1×1_56_336,s=1×1),BN,HSwish),((C3×3_336_336,s=1×1,p=1×1,g=336),BN,HSwish ),[Adaptive),((C1×1_336_84,s=1×1),R),((C1×1_84_336,s=1×1),HSigmoid)],((C1×1_336_56,s=1×1),BN)].

[0211] Among them, HSwish indicates that the activation function is Hard Swish, and the rest is the same as above.

[0212] Stage4:[((C1×1_56_336,s=1×1),BN,HSwish),((C5×5_336_336,s=2×2,p=2×2,g= 336),BN,HSwish),[Adaptive,((C1×1_336_84,s=1×1),R),((C1×1_84_336,s=1×1) ,HSigmoid)],((C1×1_336_80,s=1×1),BN)],[((C1×1_80_480,s=1×1),BN,HSwish ),((C5×5_480_480,s=1×1,p=2×2,g=480),BN,HSwish),[Adaptive,((C1×1_480_12 0,s=1×1),R),((C1×1_120_480,s=1×1),HSigmoid)],((C1×1_480_80,s=1×1),BN) ],[((C1×1_80_480,s=1×1),BN,HSwish),((C5×5_480_480,s=1×1,p=2×2,g=480),B N,HSwish),[Adaptive),((C1×1_480_120,s=1×1),R),((C1×1_120_480,s=1×1),H Sigmoid)], ((C1×1_480_80,s=1×1),BN)],[((C1×1_80_480,s=1×1),BN,HSwish)].

[0213] Among them, 1×1 convolution represents a fully connected layer.

[0214] Specifically, combined Figure 3 The network structure of the feature fusion path structure is:

[0215] [(C1×1_16_256,s=1×1),(C1×1_24_256,s=1×1),(C1×1_56_256,s=1×1),(C1×1_480_256,s=1×1),(C3×3_256_6 4, s=1×1, p=1×1), (C3×3_256_64, s=1×1, p=1×1), (C3×3_256_64, s=1×1, p=1×1), (C3×3_256_64, s=1×1, p=1×1)].

[0216] Here, the feature maps of different sizes obtained by the four 1×1 convolutions are horizontally connected to the feature maps of the same size generated by the four stages in S301. The connected feature maps are then passed through four 3×3 convolutions to obtain multi-scale feature representations and connected.

[0217] Specifically, combined Figure 3 The network structure used for classification prediction is:

[0218] Probability: [((C3×3_256_64,s=1×1,p=1×1),BN,R),((T2×2_64_64,s=2×2),BN),(T2×2_64_1,s=2×2)].

[0219] Threshold: [((C3×3_256_64,s=1×1,p=1×1),BN,R),((T2×2_64_64,s=2×2),BN),(T2×2_64_1,s=2×2)].

[0220] The network structures of the two parts are the same, and they are used to output probability map and threshold map respectively.

[0221] Specifically, the ICDAR dataset is used for pre-training, and the license plate enlargement training set is retrained. The trained model is used to segment the entire area of ​​the license plate enlargement. The test results show that in the detection of license plate enlargements of medium and large vehicles, the precision rate of this embodiment on the test set is 91.79%, the recall rate is 83.56%, and the F1 score is 87.48%.

[0222] The improved CRAFT license plate enlargement character region detection model is used to detect the license plate enlargement test set data images and obtain all the character text regions in the image;

[0223] Specifically, the CRAFT-based license plate enlarged number character area detection model includes four steps: feature extraction, deconvolution, classification prediction and binarization. This embodiment uses a skip layer connection structure similar to U-Net in the deconvolution layer. The specific structure of the improved CRAFT network is shown in Figure 4 .

[0224] Specifically, combined Figure 4 The network structure of the feature data extraction path structure is:

[0225] Slice1:[(C3×3_3_64,s=1×1,p=1×1),BN,R),((C3×3_64_64,s=1×1,p=1×1),BN,R),M(kernel=2 ,s=2,p=0,d=1),((C3×3_64_128,s=1×1,p=1×1),BN,R),((C3×3_128_128,s=1×1,p=1×1),BN)].

[0226] Slice2: [R,M(kernel=2,s=2,p=0,d=1),((C3×3_128_256,s=1×1,p=1×1),BN,R),((C3×3_256_256,s=1×1,p=1×1),BN)].

[0227] Slice3:[R,((C3×3_256_256,s=1×1,p=1×1),BN,R),M(kernel=2,s=2,p=0,d=1) ,((C3×3_256_512,s=1×1,p=1×1),BN,R),((C3×3_512_512,s=1×1,p=1×1),BN)].

[0228] Slice4:[R,((C3×3_512_512,s=1×1,p=1×1),BN,R),M(kernel=2,s=2,p=0,d=1) ,((C3×3_512_512,s=1×1,p=1×1),BN,R),((C3×3_512_512,s=1×1,p=1×1),BN)].

[0229] Slice5: [M(kernel=3,s=1,p=1,d=1),(C3×3_512_1024,s=1×1,p=6×6,d=6×6),(C1×1_1024_1024,s=1×1)].

[0230] Among them, M (kernel = 3, s = 1, p = 1, d = 1) represents the maximum pooling operation with a kernel size of 3, a step size of 1, a padding of 1, and a dilation factor of 1. The meanings of the other symbols are the same as S3.

[0231] Combination Figure 4 The network structure of the deconvolution path is:

[0232] [((C1×1_1536_512,s=1×1),BN,R),((C3×3_512_256,s=1×1,p=1×1),BN,R) ,[((C1×1_768_256,s=1×1),BN,R),((C3×3_256_128,s=1×1,p=1×1),BN,R) ],[((C1×1_384_128,s=1×1),BN,R),((C3×3_128_64,s=1×1,p=1×1),BN,R) ], [((C1×1_192_64,s=1×1),BN,R),((C3×3_64_32,s=1×1,p=1×1),BN,R)].

[0233] Here, the results obtained from the four deconvolution modules are skip-connected with the results from Slices.

[0234] Combination Figure 4 The network structure used for classification prediction is:

[0235] [((C3×3_32_32,s=1×1,p=1×1),R),((C3×3_32_32,s=1×1,p=1×1),R),((C3×3_ 32_16,s=1×1,p=1×1),R),((C1×1_16_16,s=1×1),R)],[(C1×1_16_2,s=1×1)].

[0236] The last convolution output channel is 2, representing the results of region score map and affinity score map respectively.

[0237] Specifically, the binarization method is to artificially fix the threshold, which is set to 0.5 in this embodiment.

[0238] Specifically, the ICDAR dataset is used for pre-training, and there is no need to retrain the license plate enlargement training set. The trained model is used to segment the image character area. The test results show that in the detection of license plate enlargements of medium and large vehicles, the precision rate of this embodiment on the test set is 78.74%, the recall rate is 83.16%, and the F1 score is 80.89%.

[0239] The entire area of ​​the enlarged license plate number in the image and all the character text areas in the image are merged through a preset connection module to obtain the overall location of the merged enlarged license plate number and the locations of other text areas that may belong to the same text line;

[0240] Specifically, this embodiment is based on the assumption of such a realistic situation, that is, although the enlarged license plate number has no fixed layout style, the size of the characters sprayed is generally consistent, and they are generally arranged horizontally. In other words, the size difference of the characters of the same enlarged license plate number in the image coordinate system should be small, and the vertical coordinate value difference of the characters of the same enlarged license plate number in the image coordinate system should be small. Therefore, this embodiment sets the Shape (u, v) and Coord (u, v) similarity score thresholds to 0.83 and 0.70 respectively.

[0241] By improving the SVTR text recognition model, the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning is identified to obtain the text content in the image;

[0242] Specifically, the SVTR-based license plate enlargement text model includes four steps: feature extraction, encoding, transcription, and decoding. This embodiment uses the LCNet network to replace the traditional ResNet network as the backbone of the model. Please refer to Figure 6 , which introduces the DSC module, reduces the model parameters and computational complexity through deep convolution and point-by-point convolution; further, this embodiment uses Global Mixing to replace the structure of using both Local Mixing and Global Mixing in the original network, increasing global attention while reducing the computational complexity of local attention, so as to better perceive the relationship between characters. The specific structure of the improved SVTR network can be found in Figure 5 .

[0243] Combination Figure 5 The network structure of the feature data extraction path structure is:

[0244] [((C3×3_3_8,s=2×2,p=1×1),BN,HSwish)][((C3×3_3_16,s=2×2,p=1×1),BN,HSwish)]

[0245] DSC1:[((C3×3_16_16,s=1×1,p=1×1,g=16),BN,HSwish),((C1×1_16_32,s=1×1),BN,HSwish)]

[0246] DSC2:[((C3×3_32_32,s=1×1,p=1×1,g=32),BN,HSwish),((C1×1_32_64,s=1×1),BN,HSwish)]

[0247] DSC3:[((C3×3_64_64,s=1×1,p=1×1,g=64),BN,HSwish),((C1×1_64_64,s=1×1),BN,HSwish)]

[0248] DSC4:[((C3×3_64_64,s=1×1,p=1×1,g=64),BN,HSwish),((C1×1_64_128,s=1×1),BN,HSwish)]

[0249] DSC5:[((C3×3_128_128,s=1×1,p=1×1,g=128),BN,HSwish),((C1×1_128_128,s=1×1),BN,HSwish)]

[0250] DSC6:[((C3×3_128_128,s=1×1,p=1×1,g=128),BN,HSwish),((C1×1_128_256,s=1×1),BN,HSwish)]

[0251] DSC7:[((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256,s=1×1),BN,HSwish)]

[0252] DSC8:[((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256,s=1×1),BN,HSwish)]

[0253] DSC9:[((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256,s=1×1),BN,HSwish)]

[0254] DSC10:[((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256,s=1×1),BN,HSwish)]

[0255] DSC11:[((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),((C1×1_256_256,s=1×1),BN,HSwish)]

[0256] DSC12:[((C5×5_256_256,s=1×1,p=2×2,g=256),BN,HSwish),[Adaptive,((C1×1_256_64 ,s=1×1),R),((C1×1_64_256,s=1×1),HSigmoid)],((C1×1_256_512,s=1×1),BN,HSwish)]

[0257] DSC13:[((C5×5_512_512,s=1×1,p=2×2,g=512),BN,HSwish),[Adaptive,((C1×1_512_128 ,s=1×1),R),((C1×1_128_512,s=1×1),HSigmoid)],((C1×1,512,512,s=1×1),BN,HSwish)]

[0258] [A(kernel=2,s=2,p=0)]

[0259] Among them, [A(kernel=2,s=2,p=0)] represents the average pooling operation with kernel size of 2, stride of 1, and padding of 1. The meanings of other symbols are the same as S3. Figure 6 .

[0260] Combination Figure 5 The network structure used for encoding is:

[0261] Embedding:[((C3×3_512_64,s=1×1,p=1×1),BN,Swish),((C1×1_64_120,s=1×1),BN,Swish)]

[0262] Attention:[LN,((L_120_360),Drop(0.1)),((L_120_120),Drop(0.1))]

[0263] MLP:[LN,((L_120_240),Swish),((L_240_120),Drop(0.1))]

[0264] Attention:[LN,((L_120_360),Drop(0.1)),((L_120_120),Drop(0.1))]

[0265] MLP:[LN,((L_120_240),Swish),((L_240_120),Drop(0.1))]

[0266] Among them, Swish means the activation function is Swish, LN means the layer normalization operation, (L_120_240) means the linear connection layer with 120 input channels and 240 output channels, and Drop(0.1) means the Dropout layer with a dropout rate of 0.1.

[0267] Combination Figure 5 The network structure used for the transcription part is:

[0268] [LN,((C1×1_120_512,s=1×1),BN,Swish)],[((C3×3_1024_64,s=1×1,p=1×1),BN,Swish),((C1×1_64_64,s=1×1),BN,Swish)],[(L_64_6625)]

[0269] The last fully connected layer has 6625 output channels, which means that there are 6623 Chinese, English and numeric characters in the pre-trained dictionary of the model. The transcription sequence begins and ends with blank marks, which corresponds to transcribing non-text perceptions into blank symbols.

[0270] Softmax is used to obtain the confidence of each character prediction in the text. In this embodiment, the confidence of non-license plate number characters is set to 0.

[0271] The ICDAR dataset is used for pre-training, and there is no need to retrain the enlarged license plate number training set. The trained model is used to recognize the text box area output by S5. The test results show that in the recognition of enlarged license plates of medium and large vehicles, the license plate recognition accuracy of this embodiment on the test set is 81.75%.

[0272] The text content in the image is filtered and merged through the preset verification module, the text content is verified according to the license plate number encoding rules, and the final license plate number after verification is output;

[0273] Specifically, in this embodiment, Score(a) is used as the power calculation scoring factor ρ Score(a) The value of ρ is set to 0.8, which can normalize the coding rule score to the interval [0,1] and significantly highlight the difference in coding rule scores of different text information through power operation. Finally, the highest score result among all positioning boxes in the image is output.

[0274] This embodiment is based on a data set collected from existing real scenes. The test results show that in the recognition of license plates and enlarged license plate numbers of medium and large vehicles, the accuracy of the present invention is improved by 23.17% compared with the existing license plate recognition technology, and the recognition performance is significantly improved, as shown in Table 1.

[0275] Table 1 Model performance evaluation (Note: HyperLPR is the existing mainstream method and can only recognize standard motor vehicle license plates; T / F means using / not using this module)

[0276]

[0277] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for recognizing enlarged license plate numbers based on overall and character region detection, characterized in that: The method comprises: Constructing a license plate enlargement number dataset and a scene text public dataset respectively, wherein the license plate enlargement number dataset includes a license plate enlargement number training set and a license plate enlargement number test set; The improved DBNet overall region detection model, the improved CRAFT character region detection model, and the improved SVTR text recognition model were constructed using the license plate enlargement training set and the scene text public dataset. The improved DBNet enlarged license plate number overall area detection model is used to detect the enlarged license plate number test set data image, and the overall area of ​​the enlarged license plate number in the image is obtained; The improved CRAFT license plate enlargement character region detection model is used to detect the license plate enlargement test set data images and obtain all the character text regions in the image; The entire area of ​​the enlarged license plate number in the image and all the character text areas in the image are merged through a preset connection module to obtain the overall location of the merged enlarged license plate number and the locations of other text areas that may belong to the same text line; By improving the SVTR text recognition model, the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning is identified to obtain the text content in the image; The text content in the image is filtered and merged through the preset verification module, the text content is verified according to the license plate number encoding rules, and the final license plate number after verification is output.

2. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 1, characterized in that: The scene text public data set includes an overall detection training set and an overall detection verification set, a character detection training set and a character detection verification set, and a text recognition training set and a text recognition verification set.

3. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 2 is characterized in that: The method of constructing an improved DBNet overall region detection model, an improved CRAFT character region detection model, and an improved SVTR text recognition model respectively by using the license plate enlargement training set and the scene text public data set includes: The preset improved DBNet enlarged license plate overall area detection network is iteratively trained using the overall detection training set, the accuracy of the model is verified using the overall detection verification set, the optimal model is saved and retrained using the enlarged license plate training set, and finally the improved DBNet overall area detection model is obtained; The preset improved CRAFT license plate enlarged number character region detection network is iteratively trained using the character detection training set, the accuracy of the model is verified using the character detection verification set, the optimal model is saved and retrained using the enlarged number training set, and finally the improved CRAFT character region detection model is obtained; The constructed improved SVTR license plate enlarged number text recognition network is iteratively trained using the text recognition training set, and the model accuracy is verified using the text recognition verification set. The optimal model is saved and retrained using the enlarged number training set, and finally the improved SVTR text recognition model is obtained.

4. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 1, characterized in that: The improved DBNet overall region detection model comprises a first feature extraction module, a feature fusion module, a first classification prediction module, a first binarization module and an overall region output module connected in sequence; The first feature extraction module is used to extract features from the license plate enlargement test set data image to obtain a first primary feature map; The feature fusion module is used to obtain several feature maps of different sizes by subjecting the primary feature map to four 1×1 convolutions, and then upsampling from top to bottom, and horizontally connecting the feature maps of the same size generated from bottom to top; the connected feature maps are then subjected to four 3×3 convolutions to eliminate the aliasing effect of upsampling, and finally a multi-scale feature representation is obtained and connected to obtain a first fused feature map; The first classification prediction module is used to input the first fusion feature map into a preset first convolution classification network to obtain a target probability map and a threshold map; The first binarization module is used to generate a first approximate binary image by using a differentiable binarization method on the target probability image and the threshold image; The overall area output module is used to obtain the vertex coordinates of the regional polygon according to the first approximate binary image, and then restore these coordinates to the original image size to obtain the overall area of ​​the enlarged license plate number.

5. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 4 is characterized in that: The first feature extraction module is MobileNetV3, which includes a 3×3 convolution, a batch normalization operation layer, a Hard Swish activation function, a first stage layer, a second stage layer, a third stage layer, and a fourth stage layer; The first stage layer includes 3 residual units, and the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, and 1×1 convolution, batch normalization and ReLU activation; The second stage layer includes 3 residual units, each of which has a structure of 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation; The third stage layer includes 6 residual units, wherein the structures of the first 4 residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, 1×1 convolution, batch normalization and Hard Swish activation function, and the structures of the last two residual units include 1×1 convolution, batch normalization and Hard Swish activation function, 3×3 convolution, batch normalization and Hard Swish activation function, SE module, 1×1 convolution, batch normalization and Hard Swish activation function; The fourth stage layer includes 3 residual units, 1×1 convolution, batch normalization and Hard Swish activation function, wherein the structure of each residual unit includes 1×1 convolution, batch normalization and ReLU activation, 5×5 convolution, batch normalization and ReLU activation, SE module, 1×1 convolution, batch normalization and ReLU activation; The feature fusion module is a feature pyramid network; The first classification prediction module includes a first convolution classification network, the first convolution classification network includes a target probability map output module and a threshold map output module, the target probability map output module and the threshold map output module have the same structure, both including 3×3 convolution, batch normalization and ReLU activation, 2×2 deconvolution, batch normalization and 2×2 deconvolution; The loss function of the improved DBNet overall region detection model is the prediction probability loss L s , Binarization loss L b And the target area distance loss L t The weighted sum of s and L b Using cross entropy loss, L t Use L1 norm loss.

6. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 1, characterized in that: The improved CRAFT license plate enlarged number character region detection model comprises a second feature extraction module, a deconvolution module, a second classification prediction module, a second binarization module and a character region output module which are connected in sequence; The second feature extraction module is used to extract features from the license plate enlargement test set data image to obtain a second primary feature map; The deconvolution module is used to perform deconvolution on the second primary feature map to obtain a deconvolution feature map; The second classification prediction module is used to input the deconvolution feature map into a preset second convolution classification network to obtain the area score of the corresponding character and the connection score between the characters; The second binarization module is used to connect and expand the region scores of the characters and the connection scores between the characters by using a fixed threshold method to generate a second approximate binary graph; The character region output module is used to obtain the vertex coordinates of the region polygon according to the second approximate binary image, and then restore these coordinates to the original image size to output the target character region.

7. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 6, characterized in that: The second feature extraction module is VGG16_BN, which includes a first slice layer, a second slice layer, a third slice layer, a fourth slice layer and a fifth slice layer; The first slice layer includes 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The second slice layer includes ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The third slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The fourth slice layer includes ReLU activation, 3×3 convolution, batch normalization and ReLU activation, max pooling, 3×3 convolution, batch normalization and ReLU activation, 3×3 convolution and batch normalization; The fifth slice layer includes maximum pooling, 3×3 convolution and 1×1 convolution; The deconvolution module includes a first deconvolution module, a second deconvolution module, a third deconvolution module, and a fourth deconvolution module; The fifth slice layer is connected to the output skip layer of the fourth slice layer, and then connected to the output skip layer of the third slice layer through the first deconvolution module, and then connected to the output skip layer of the second slice layer through the second deconvolution module, and then connected to the output skip layer of the first slice layer through the third deconvolution module, and finally connected to the input of the fourth deconvolution module; The second classification prediction module includes a second convolution classification network, which includes three 3×3 convolutions and two 1×1 convolutions connected in sequence, wherein except for the last 1×1 convolution, ReLU activation is used after other convolution layers; The loss function of the improved CRAFT license plate enlarged number character region detection model is the weighted sum of the L2 norm loss corresponding to the character region score and the connection score between characters.

8. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 1, characterized in that: The method of merging the entire area of ​​the enlarged license plate number in the image and all the character text areas in the image through a preset connection module to obtain the overall positioning of the merged enlarged license plate number and the positioning of other text areas that may belong to the same text line includes: Determine whether the vehicle has been painted with an enlarged license plate number based on the overall area of ​​the enlarged license plate number output by the improved DBNet overall area detection model; The entire area of ​​the enlarged license plate number is divided into different groups through the preset connectivity module, so that the positioning boxes in each group are considered to belong to the same text sequence under the condition that the similarity score is greater than the specified threshold. The similarity score calculation formula is as follows: Among them, Shape(u,v) represents the size similarity of different positioning boxes, Coord(u,v) represents the similarity of the vertical coordinate values ​​of the center points of different positioning boxes, h(u) and y(u) are set as the box height and vertical coordinate of a certain positioning box u, h(v) and y(v) represent the box height and vertical coordinate of another positioning box v; Shape(u,v) uses the normalized ratio of the box height difference of different positioning boxes to their mean as a measure of whether the text sizes of the two are similar, and Coord(u,v) uses the normalized ratio of the vertical coordinate height difference of different positioning boxes to their mean box height as a measure of whether the vertical coordinate values ​​of the center points of the two texts are similar; Merge the overall area of ​​the enlarged license plate number in the image and all the character text areas in the image, specifically: First, use the positioning boxes in the overall area of ​​the enlarged license plate number and all the character text areas, and calculate the height, width and center coordinates of each positioning box at the same time; traverse the input positioning box list, and when the Shape(u,v) and Coord(u,v) of different positioning boxes u and v simultaneously meet the specified similarity score threshold, mark them as the same group; merge the text boxes marked as the same group, that is, arrange the horizontal coordinates of the center points of all text boxes in the same group from left to right, and finally output the connected positioning boxes of the upper left and lower left coordinates of the first box and the upper right and lower right coordinates of the last box as the four vertices of this area; judge the similarity results of merging character areas based on the threshold of the similarity score, and the setting of different thresholds indicates that the connectivity method allows the characters of the same text sequence to have size differences and upper and lower floating ranges, thereby fully locating the long text of the research object.

9. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 1, characterized in that: The improved SVTR text recognition model includes a third feature extraction module, an encoding module, a transcription module, a decoding module and a text content output module; The third feature extraction module is used to extract features from the image area corresponding to the overall positioning of the enlarged license plate number and other text area positioning to obtain a third primary feature map; The encoding module includes an Im2Seq module and a MultiHead module; the Im2Seq module is used to transpose the third primary feature map channel into a sequence, and then downsample it using 2 layers of convolution and normalization to obtain a patch, and the Im2Seq module includes 3×3 convolution, batch normalization and Swish activation, 1×1 convolution, batch normalization and Swish activation; The core global mixing block in the MultiHead module is Global Mixing, which is used to perceive the correlation between characters and between characters and background. Specifically: Among them, d k is the dimension, Q, K, V are obtained by linear transformation of the input image region matrix, Q and K T After matrix multiplication, Softmax normalization is performed to obtain the Attention matrix, which is then multiplied with the V matrix and then output by linear transformation. After each linear transformation, 10% of the neurons are randomly dropped to reduce overfitting. The MultiHead module is used to input the patch features into the hybrid block for layer normalization, and then perceive the correlation between characters through the Attention method. After the output of the Attention method is subjected to identity mapping and layer normalization, the features enter the multi-layer perceptron MLP for fusion, and finally obtain the second fused feature map. The specific expression of the Attention method is: Among them, d k is the dimension, Q, K, V are obtained by linear transformation of the input image region matrix, Q and K T After matrix multiplication, Softmax normalization is performed to obtain the Attention matrix, which is then multiplied with the V matrix and then output by linear transformation. After each linear transformation, 10% of the neurons are randomly dropped to reduce overfitting. The implementation path of the MLP is: fully connected layer and Swish activation, fully connected layer and Dropout; The transcription module is used to perform a layer normalization operation on the second fused feature map, and then use 3 layers of convolution and batch normalization to merge features to obtain a merged feature map; The decoding module is used to obtain the confidence of each character prediction in the text by using the Softmax method on the merged feature map; The text content output module is used to mark the probability of non-target encoded characters as 0, use CTC decoding mapping according to the confidence of each character prediction in the text, and output the text content in the image.

10. The method for recognizing enlarged license plate numbers based on overall and character region detection according to claim 1, characterized in that: The method of filtering and merging the text content in the image by a preset verification module, verifying the text content according to the license plate number encoding rule, and outputting the final license plate number after verification includes: The text information is divided into different groups through the preset verification module, so that the text information in each group is judged whether to be filtered, merged and output according to the coding rule score of the license plate number; the design method of coding rule score calculation is detailed in the following formula: Score(a)=max(|Len(a)-7|+I a ,|Len(a)-8|+I a ) Where Score(a) represents the coding rule score of the input text information; set a to be the text string, Len(a) to be the length of the string, and I a is an indicative variable. It is defined that when a does not meet the license plate coding rules, the value is 1, otherwise it is 0. The absolute value of the difference between the current text length and the target text length plus the indicative variable of whether the current text meets the license plate coding rules is used as a metric. Score(a) uses the maximum value of this metric as the score of the text with respect to the license plate coding rules. The threshold of the rule score is determined by actual needs. The verified area information will be output as the final result after filtering out the falsely detected characters. Filter the text content in the image, specifically: Use a list to save the text content and positioning box output by the improved SVTR text recognition model, calculate the encoding rule score of each text and the height, width and center coordinates of the corresponding positioning box, etc.; traverse the input text list, then calculate the encoding rule scores of adjacent text sequence combinations, merge the text sequence combinations with the maximum scores, sort and return the text sequence information that meets the minimum output threshold; By judging the overall positioning of the enlarged license plate number after the merger due to excessive merging spacing based on the threshold of the coding rule score, and the overall area that is not merged in the positioning of other text areas that may belong to the same text line, the misdetected characters with too low scores are screened out, and finally the enlarged license plate number and standard motor vehicle license plate information are output.

Images