License Plate Recognition Method in Unconstrained Scenarios Based on Spatial Attention Features

Abstract

The present invention discloses a license plate recognition method in an unrestricted scenario based on spatial attention features. The license plate area in the complete license plate image is extracted through the YOLOv5 target detection network, and then the license plate area is sent to the spatial transformation network to correct the license plate characters; the corrected features are then sent to the spatial attention feature extraction module to extract the spatial attention features of the area to obtain the final features; finally, the final features are input into a trained character recognition network for character recognition to obtain character information of seven license plates, and then the license plate characters are identified. The present invention calculates the spatial attention features of the license plate area, so that the network is more focused on the character information of the license plate area, avoiding the influence of complex background on the success rate of license plate recognition. The present invention adopts a non-segmentation method in the character recognition process, and does not require a character segmentation process for the license plate character area, thereby avoiding license plate character recognition errors caused by character segmentation errors.

Description

Technical Field

[0001] The present invention relates to the technical field of license plate recognition, and in particular to a license plate recognition method in an unrestricted scenario based on spatial attention features. Background Art

[0002] With the explosive growth of private cars in recent years, automatic license plate recognition (ALLR) technology has become increasingly widely used in various scenarios, such as highway toll booths and parking lots. Researchers have proposed numerous high-performance ALR methods for these restricted scenarios. However, research on improving LPR accuracy in various unrestricted scenarios remains a significant area of application.

[0003] Generally speaking, license plate images can be divided into two categories: one is license plate images in fixed scenes, such as those at highway toll booths, unmanned parking lots, or residential gates. License plate images in the above scenes are almost always taken in locations with sufficient light, moderate distance, and fixed angles; the other is license plate images in unconstrained scenes, such as license plate images taken by traffic police (they take license plate photos using mobile devices or smartphones), or license plate images in the "Skynet" monitoring system. Most of these license plate images are taken under unfavorable conditions such as darkness, underexposure, rain, and snow. Since these unfavorable conditions often cause license plate images to be blurred, distorted, or tilted, research on license plate recognition technology in unconstrained scenes is of practical significance.

[0004] The entire process of automatic license plate recognition is usually divided into two main tasks: license plate detection, which aims to locate the license plate in the image; license plate recognition, which aims to identify the character string in the license plate.

[0005] License plate region detection is an essential task, aiming to locate the license plate region in an image. Thanks to deep learning-based object detectors, license plate region detection has made significant progress. These object detection methods can be primarily categorized into two types based on their design principles: one-stage object detection algorithms, representative of which include SSD, YOLO, and FCOS; and two-stage object detection algorithms, including R-CNN, Fast R-CNN, and Faster R-CNN.

[0006] Current license plate recognition algorithms can be divided into two categories: one is the segmentation-based recognition method, and the other is the non-segmentation method. The segmentation-based license plate recognition method usually segments the characters in the license plate area and then recognizes them through optical character recognition. However, various situations in unconstrained scenes: such as overexposure, rainy and snowy weather and other complex backgrounds may lead to incorrect license plate character segmentation. At the same time, the probability of successful segmentation determines the success rate of license plate recognition based on the segmentation method. Therefore, the segmentation-based license plate recognition method has great limitations. The non-segmentation license plate character recognition method can recognize all characters in one step, avoiding the adverse effects of license plate character segmentation errors. However, this method has a greatly reduced recognition accuracy in unrestricted scenes due to unfavorable conditions such as exposure, shooting distance that is too far or too close, and skewed license plates. Summary of the Invention

[0007] In view of the deficiencies of the existing technology, the technical problem to be solved by the present invention is to provide a license plate recognition method in an unrestricted scenario based on spatial attention features.

[0008] The technical solution of the present invention to solve the above technical problem is to provide a license plate recognition method in an unrestricted scene based on spatial attention features, characterized in that the method comprises the following steps:

[0009] Step 1: Process the complete license plate image in the CCPD license plate dataset to obtain the true value of the license plate area and the true value of the license plate characters gn i and license plate area images; then the complete license plate images and license plate area images are divided into training set, validation set and test set respectively;

[0010] Step 2: Use the complete license plate image training set to train the YOLOv5 target detection network to obtain a trained YOLOv5 target detection network; then input the complete license plate image test set into the trained YOLOv5 target detection network, and predict the coordinate values of the four vertices of the license plate area of the image in the complete license plate image test set as the predicted value of the license plate area;

[0011] Step 3: The license plate frame cut out by the predicted value of the license plate area is sent to the spatial transformation network to correct the license plate characters and obtain the corrected features;

[0012] Step 4: Use the license plate area image training set to train the spatial attention feature extraction module to obtain a trained spatial attention feature extraction module; then input the corrected features obtained in step 3 into the trained spatial attention feature extraction module, first pass through three identical convolution layers to obtain a basic feature map F; then perform global maximum pooling and global average pooling on the basic feature map F to obtain two feature maps; then superimpose the two feature maps based on the channel dimension, and then reduce the feature map to one dimension through a convolution layer, and then pass through the sigmoid activation function to obtain the spatial attention feature M s ; Then the spatial attention feature M s Do matrix Hadamard product with the basic feature map F to get the final feature;

[0013] Step 5: Use the license plate area image training set to train the character recognition network to obtain a trained character recognition network; then input the final features obtained in step 4 into the trained character recognition network for character recognition, obtain the character information of the seven license plates, and then identify the license plate characters.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] (1) The present invention extracts the license plate area in the complete license plate image through the YOLOv5 target detection network, and then sends the license plate area to the spatial transformation network to correct the license plate characters; then the corrected features are sent to the spatial attention feature extraction module to extract the spatial attention features of the area to obtain the final features; finally, the final features are input into the trained character recognition network for character recognition, and the character information of the seven license plates is obtained, thereby identifying the license plate characters.

[0016] (2) By calculating the spatial attention features of the license plate area, the present invention enables the network to ignore the background information between the license plate characters and focus more on the characters in the license plate area, thus preventing the complex background from affecting the success rate of license plate recognition. Therefore, using a recognition network with spatial attention can more accurately recognize license plate characters in various complex scenarios, improving the accuracy of character recognition.

[0017] (3) The present invention adopts a non-segmentation method in the character recognition process, which does not require a character segmentation process for the license plate character area, thereby avoiding license plate character recognition errors caused by character segmentation errors.

[0018] (4) The present invention embeds a spatial transformation network in the two processes of license plate region segmentation and license plate character recognition to correct deformed license plate characters and improve the success rate of license plate character recognition.

[0019] (5) The deep learning model constructed by the present invention can automatically update parameters without the need for other prior information, and can learn license plate feature information in complex scenarios, and can quickly and accurately perform license plate recognition in an environment built by a GPU.

[0020] (6) The present invention constructs a complete license plate recognition framework, which fully realizes the complete process from license plate area detection to license plate character recognition. It can recognize license plates in images under adverse conditions such as overexposure, tilted shooting angle, bad weather, etc., and its accuracy is better than other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the overall network structure of the present invention;

[0022] Figure 2 Schematic diagram of the network of the spatial attention feature extraction module of the present invention;

[0023] Figure 3 A network diagram of a character recognition network according to the present invention;

[0024] Figure 4 Schematic diagram of a complete license plate image according to embodiment 1 of the present invention;

[0025] Figure 5 A schematic diagram of the name of a complete license plate image according to embodiment 1 of the present invention;

[0026] Figure 6 Schematic diagram of a license plate area image according to Example 1 of the present invention;

[0027] Figure 7 This is an example diagram of the completion of license plate area detection in Example 1 of the present invention;

[0028] Figure 8 This is a diagram of the license plate area detection and character recognition results obtained in Example 1 of the present invention. DETAILED DESCRIPTION

[0029] The specific embodiments of the present invention are given below. The specific embodiments are only used to further illustrate the present invention and do not limit the scope of protection of the claims of this application.

[0030] The present invention provides a license plate recognition method in an unrestricted scene based on spatial attention features (hereinafter referred to as the method), characterized in that the method comprises the following steps:

[0031] Step 1: Process the complete license plate image in the CCPD license plate dataset to obtain the true value of the license plate area and the true value of the license plate characters gn i and license plate area images; then the complete license plate images and license plate area images are divided into training set, validation set and test set respectively;

[0032] Preferably, step 1 specifically comprises: taking the coordinate values of the four vertices of the license plate area contained in the name of the complete license plate image as the true value of the license plate area; taking the license plate character information contained in the name of the complete license plate image as the true value gn of the license plate character i ; The license plate area in each complete license plate image is cropped according to the coordinate values of the four vertices of the license plate area contained in the name of the complete license plate image to obtain the license plate area image; then the complete license plate images are divided into a complete license plate image training set, a complete license plate image verification set and a complete license plate image test set, and the license plate area images are divided into a license plate area image training set, a license plate area image verification set and a license plate area image test set.

[0033] Preferably, in step 1, during the process of obtaining the license plate area image, the cropped image is randomly scaled for image enhancement to simulate the deformation of the license plate in an unrestricted scenario, thereby increasing the data volume and improving the robustness of the model. Random scaling is to stretch or shrink the corresponding license plate area by 1 / 10 of its length and width.

[0034] Step 2: Use the complete license plate image training set to train the YOLOv5 target detection network to obtain the trained YOLOv5 target detection network as the license plate area detection network; then use the complete license plate image test set as the input of the trained YOLOv5 target detection network, input it into the trained YOLOv5 target detection network, and predict the coordinate values of the four vertices of the license plate area of the image in the complete license plate image test set as the predicted value of the license plate area;

[0035] Preferably, in step 2, when the intersection of union (IOU) between the predicted value of the license plate area and the true value of the license plate area of the corresponding image obtained in step 1 is greater than 60%, it is determined that the license plate area detection is correct and is used in the subsequent calculation of the accuracy.

[0036] Preferably, in step 2, the trained YOLOv5 target detection network is tested on a complete license plate image verification set to calculate the accuracy, and the parameters for license plate detection in the YOLOv5 target detection network are modified according to the accuracy of the verification set until the accuracy reaches at least 95%, and the training is completed.

[0037] Preferably, in step 2, the intersection-over-union ratio is calculated by the coordinates (x1, y1) of the upper left corner point and the coordinates (x2, y2) of the lower right corner point in the predicted value of the license plate area and the coordinates (a1, b1) of the upper left corner point and the coordinates (a2, b2) of the lower right corner point in the true value of the license plate area of the same image obtained in step 1. The specific formula is shown in formula (1):

[0038]

[0039] Preferably, in step 2, the training process of the YOLOv5 target detection network is specifically: the complete license plate image training set is used as the input of the YOLOv5 target detection network, input into the YOLOv5 target detection network for training, and the output result is the predicted value; the coordinate values of the four vertices of the license plate area contained in the name of the image in the complete license plate image training set are used as the true value; the smaller the difference between the predicted value and the true value, the more accurate the license plate area prediction, until the set target is reached, the training is completed, and a trained YOLOv5 target detection network is obtained; the target is set to iterate 300 rounds or the license plate area detection accuracy in the complete license plate image verification set reaches at least 95%.

[0040] Preferably, in step 2, in order to achieve more accurate license plate area prediction and make it more suitable for license plate detection, the YOLOv5 target detection network can be improved to obtain an improved YOLOv5 target detection network; the specific improvement method is: in order to make the preset anchor frame size suitable for detecting license plates, kmeans clustering calculation is performed on the license plates in the complete license plate image training set to obtain an adaptive anchor frame, and then an improved YOLOv5 target detection network is obtained.

[0041] Preferably, in step 2, the specific steps of kmeans clustering are as follows:

[0042] (2.1) Randomly select k vertex coordinates of the license plate area from the complete license plate image training set as the cluster center C = {c1, c2, ..., c k}, k also represents the number of initial anchor boxes of the YOLOv5 target detection network;

[0043] (2.2) Calculate each image x in the complete license plate image training set separately i The distance to the k cluster centers, and then the image x i Divide into the category corresponding to the cluster center with the smallest distance; i represents the i-th image in the complete license plate image training set;

[0044] (2.3) Recalculate the center of the category based on the reclassified image in step (2.2) Finally, k cluster center points are calculated as the sizes of k preset anchor boxes; 1≤j≤k, n is the number of images in each category.

[0045] Step 3: The license plate frame cropped by the predicted value of the license plate area is sent to the Spatial Transformer Networks to correct the license plate characters to avoid recognition errors caused by the physical deformation of the license plate and obtain the corrected features; the input is a feature vector of H*W*C (H represents the high dimension of the image, W represents the wide dimension of the image, and C represents the channel dimension of the image). After passing through the Spatial Transformer Network, the output dimension remains unchanged;

[0046] Preferably, in step 3, the correction of the license plate characters in the license plate frame using the spatial transformation network includes three processes: parameter prediction, coordinate mapping, and output pixel. First, the input of the parameter prediction part is the license plate frame cropped by the predicted value of the license plate area, and after the convolution operation, the transformation matrix θ∈R is obtained. 2×3 ; Then the coordinate mapping part uses the transformation matrix θ to obtain the mapping relationship between the output image and the corresponding position of the input image; finally, the interpolation algorithm is used to calculate the value of each pixel in the output image, and the value of each pixel constitutes the corrected feature.

[0047] Step 4: Use the license plate area image training set to train the spatial attention feature extraction module to obtain a trained spatial attention feature extraction module; then input the corrected features obtained in step 3 into the trained spatial attention feature extraction module, first pass through three identical convolution layers to obtain a basic feature map F; then perform global maximum pooling and global average pooling on the basic feature map F to obtain two feature maps; then superimpose the two feature maps based on the channel dimension, and then reduce the feature map to one dimension through a convolution layer, and then pass through the sigmoid activation function to obtain the spatial attention feature M s ; Then the spatial attention feature M s Do matrix Hadamard product with the basic feature map F to get the final feature;

[0048] Preferably, in step 4, in the three identical convolutional layers, the number of convolution kernels in each convolutional layer is 48, the convolution kernel size is 3*3, the step size is 1, padding=2, and activation is performed by the ReLU activation function.

[0049] Preferably, in step 4, in a convolution layer, the number of convolution kernels is 1, the convolution kernel size is 7*7, the step size is 1, and padding=6.

[0050] Preferably, step 4 is specifically as follows: the rectified features of size H×W×C are processed by three layers of convolution to obtain a basic feature map F of size H×W×48; then a global maximum pooling and a global average pooling based on the channel dimension are performed respectively to obtain two feature maps of size H×W×1; then the two feature maps are spliced based on the channel to obtain a feature map of H×W×2, and then the feature map is reduced to one dimension by a convolution layer with a convolution kernel size of 7*7. At this time, the feature map size is H×W×1; and then the spatial attention feature M is obtained by the sigmoid activation function. s ; Then the spatial attention feature M s Do a matrix Hadamard product with the basic feature map F to get the final feature; the above process is expressed by formula (2):

[0051]

[0052] Step 5: Use the license plate area image training set to train the character recognition network to obtain a trained character recognition network; then input the final features obtained in step 4 into the trained character recognition network for character recognition, obtain the character information of the seven license plates, and then identify the license plate characters.

[0053] Preferably, in steps 3 to 5, the spatial transformation network, the spatial attention feature extraction module and the character recognition network constitute a character recognition network based on spatial attention features;

[0054] The training process of the character recognition network based on spatial attention features is as follows: the license plate area image training set is input into the character recognition network based on spatial attention features for training. Through steps 3 to 5, the output result is seven license plate characters as the predicted value pn of the license plate characters. i ; Then the predicted value pn of the license plate character i The true value gn of the license plate character obtained in step 1 i Substitute into the loss function of formula (2) and get the loss function value L cls (pn,gn); through the loss function value L cls (pn, gn) continuously optimizes the character recognition network based on spatial attention features. The smaller the loss function value, the more accurate the license plate character recognition is, until the set goal is reached. The training is completed and a trained character recognition network based on spatial attention features is obtained. The set goal is to iterate 50 rounds or the license plate character recognition accuracy in the license plate area image verification set reaches at least 98%. If and only if the predicted value is the same as the true value gn of the license plate character of the same image obtained in step 1 i Only when the characters are exactly the same as each other can it be determined that the characters in the license plate area are correctly recognized;

[0055]

[0056] In formula (3), N represents the number of training samples in the license plate area; gn i Represents the true value of the seven license plate characters; pn i Represents the predicted value of the seven license plate characters, pn i Also includes nc i floating point numbers, each representing the probability of belonging to a particular character.

[0057] Preferably, in step 5, the trained character recognition network based on spatial attention features is tested on a verification set of license plate area images to calculate the accuracy, and the parameters for license plate recognition in the character recognition network based on spatial attention features are adjusted according to the accuracy of the verification set until the accuracy reaches at least 98%, and the training is completed.

[0058] Preferably, in step 5, the character recognition network is designed for seven-digit license plates and includes seven parallel character classifiers, each character classifier corresponding to a character at a fixed position on the license plate, thus being able to predict seven characters;

[0059] Each character classifier contains 3 convolutional layers, 1 maximum pooling layer, 1 extension layer and 2 fully connected layers;

[0060] Each convolution layer contains 24 convolution kernels, each convolution kernel size is 3*3, the stride is 1, and padding = 2; the pooling kernel size of the maximum pooling layer is 2*2, and the stride is 1;

[0061] The number of neurons in the first fully connected layer of each character classifier is 100; the number of neurons in the second fully connected layer of the first character classifier is 34, the number of neurons in the second fully connected layer of the second character classifier is 25, and the number of neurons in the second fully connected layer of the third to seventh character classifiers is 35.

[0062] Step 5 is as follows: the final feature obtained in step 4 is processed by two layers of convolution to obtain a feature map of H×W×24, the feature is subjected to maximum pooling processing to obtain a feature map of H / 2×W / 2×24, the pooled feature is subjected to one more convolution processing to obtain a feature map of H / 2×W / 2×24; the H / 2×W / 2×24 feature map is stretched into a one-dimensional vector through the extension layer, and then passed through the first fully connected layer and the second fully connected layer, and finally the character with the highest probability at that position is predicted through the softmax function; seven classifiers predict a set of seven characters to identify all the characters in the entire license plate.

[0063] Example 1

[0064] In step 1, half of the images in the CCPD-base and CCPD-blur sub-datasets are used as the training set, the remaining half of the images in the CCPD-base sub-dataset and the unused 10% of the images in CCPD-blur are used as the validation set, and finally the remaining images in the CCPD-blur sub-dataset and other sub-datasets: CCPD-fn, CCPD-rotate, CCPD-tilt, CCPD-weather, and CCPD-Challenge are used as the test set;

[0065] Figure 5 In the title bar, the file name (that is, the image name) is displayed Figure 5 The portion enclosed by the rectangular box in the image is 01-1_3-263&456_407&514-407&510_268&514_263&460_402&456-0_0_10_23_32_28_33-166-2.jpg. The numbers in the image name mean the following: 01: The proportion of the license plate to the entire interface. 1_3: The horizontal and vertical angles of the license plate. 263&456_407&514: The coordinates of the upper-left and lower-right corners of the license plate annotation box. 407&510_268&514_263&460_402&456: The coordinates of the four vertices of the license plate, in the order of lower right, lower left, upper left, and upper right. 0_0_10_23_32_28_33: These seven digits represent the license plate character information, with the first digit representing the province, the second digit representing the city, and the remaining portion representing the license plate number. 166: Brightness; a larger value indicates a higher brightness. 2: Blur; a smaller value indicates a more blurred image.

[0066] Figure 4 In the figure, the size of the complete license plate image is 720 (width) * 1160 (height) * 3 (channels); Figure 6 In the image, the license plate area size is 40 (height) * 116 (width) * 3 (channels).

[0067] In step 2, the initial anchor box size of the YOLOv5 target detection network is updated with the parameters of the anchor boxes [96, 25, 248, 66, 149, 35], [190, 41, 206, 84, 209, 50], and [260, 119, 260, 55, 316, 72] generated by kmeans clustering based on the complete license plate image training set, and its detection target is set to 'license plate area'. In this embodiment, k = 9, and the number of initial anchor boxes of the YOLOv5 target detection network is 9. If and only if the IOU is greater than 60%, it is determined that the license plate area detection is correct, such as Figure 7 shown.

[0068] In step 3, the input of the parameter prediction part is the license plate frame of size 40*116*3 cut out by the predicted value of the license plate area, and the affine transformation parameter θ∈R is predicted by the positioning network. 2×3 Then, the license plate feature image is spatially transformed using the transformation parameters through the parameterized sampling grid to obtain the corrected features. The size of the corrected features is still 40*116*3.

[0069] In step 4, the rectified features of size 40*116*3 are processed through three layers of convolution operation to further extract features. The basic feature map F∈R C×H×W The size is 40*116*48; then the basic feature map F is subjected to a channel-based global maximum pooling (Max pooling) and a global average pooling (Aver pooling) to obtain two feature maps of size 40×116×1; then these two feature maps are spliced based on the channel to obtain a feature map of 40×116×2, and then a convolution layer with a convolution kernel size of 7*7 is used to reduce the dimension of the feature map to one dimension. At this time, the feature map size is 40×116×1; then the spatial attention feature M is obtained after the sigmoid activation function. s ; Then the spatial attention feature M s Perform matrix Hadamard product with the basic feature map F to obtain the final feature of size 40×116×48;

[0070] In step 5, the final feature obtained in step 4 is processed by two layers of convolution to obtain a 40×116×24 feature map, which is then subjected to maximum pooling to obtain a 20×58×24 feature map. The pooled feature is then subjected to one more convolution to obtain a 20×58×24 feature map. The 20×58×24 feature map is stretched into a one-dimensional vector through the extension layer, and then passed through the first and second fully connected layers, and finally the softmax function is used to predict the character with the highest probability at that position. Seven classifiers predict a set of seven characters to identify all the characters in the entire license plate (such as Figure 8 shown).

[0071] Other license plate area detection methods were used to compare with the method of the present invention, and the accuracy results of license plate area detection are shown in Table 1. Other license plate recognition methods were used to compare with the method of the present invention, and the accuracy results of successful character recognition are shown in Table 2.

[0072] Table 1

[0073]

[0074]

[0075] Table 2

[0076]

[0077] In Tables 1 and 2, Base, Db, Fn, Rotate, Tilt, Weather, and Challenge are the names of CCPD neutron data, respectively.

[0078] As can be seen from Table 1, the cascade classifier has difficulty accurately locating the license plate, and therefore performs poorly when the license plate detection IOU is set to be greater than 60%. It also performs poorly when processing tilted and rotated license plates. Based on the detection accuracy of 77.3% on the CCPD-F n sub-dataset (where the license plate is relatively close or far from the camera shooting position), it is inferred that YOLO9000 has poor detection performance for objects that are too large or too small.

[0079] Table 2 shows that RPnet and TE2E outperform Faster-RCNN and YOLO9000, thanks to the unified optimization of the license plate detection and recognition frameworks. In summary, the license plate recognition framework adopted by this paper has higher character recognition accuracy in most sub-datasets due to the incorporation of a spatial attention mechanism.

[0080] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A license plate recognition method in an unrestricted scene based on spatial attention features, characterized in that: The method comprises the following steps: Step 1: Process the complete license plate image in the CCPD license plate dataset to obtain the true value of the license plate area and the true value of the license plate characters. and license plate area images; then the complete license plate images and license plate area images are divided into training set, validation set and test set respectively; Step 2: Use the complete license plate image training set to train the YOLOv5 target detection network to obtain a trained YOLOv5 target detection network; then input the complete license plate image test set into the trained YOLOv5 target detection network, and predict the coordinate values of the four vertices of the license plate area of the image in the complete license plate image test set as the predicted value of the license plate area; In step 2, in order to achieve more accurate license plate area prediction, the YOLOv5 target detection network is improved to obtain an improved YOLOv5 target detection network. The specific improvement method is: in order to make the preset anchor frame size suitable for detecting license plates, kmeans clustering calculation is performed on the license plates in the complete license plate image training set to obtain an adaptive anchor frame, and then the improved YOLOv5 target detection network is obtained. The specific steps of kmeans clustering are as follows: (2.1) Randomly select k vertex coordinates of the license plate area from the complete license plate image training set as cluster centers , k also represents the number of initial anchor boxes of the YOLOv5 target detection network; (2.2) Calculate each image in the complete license plate image training set separately The distance to the k cluster centers, and then the image Divide into the category corresponding to the cluster center with the smallest distance; i represents the i-th image in the complete license plate image training set; (2.3) Based on the reclassified image in step (2.2), recalculate the center of the category ;Finally, calculate the k cluster center points as the sizes of k preset anchor boxes; 1≤j≤k, n is the number of images in each category; Step 3: The license plate frame cut out by the predicted value of the license plate area is sent to the spatial transformation network to correct the license plate characters and obtain the corrected features; Step 4: Use the license plate area image training set to train the spatial attention feature extraction module to obtain a trained spatial attention feature extraction module; then input the corrected features obtained in step 3 into the trained spatial attention feature extraction module, first pass through three identical convolution layers to obtain a basic feature map F; then perform global maximum pooling and global average pooling on the basic feature map F to obtain two feature maps; then superimpose the two feature maps based on the channel dimension, and then reduce the feature map to one dimension through a convolution layer, and then pass through the sigmoid activation function to obtain the spatial attention feature. ; Then the spatial attention feature Do matrix Hadamard product with the basic feature map F to get the final feature; Step 5: Use the license plate area image training set to train the character recognition network to obtain a trained character recognition network; then input the final features obtained in step 4 into the trained character recognition network for character recognition, obtain the character information of the seven license plates, and then identify the license plate characters.

2. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1 is characterized in that: Step 1 is specifically: taking the coordinate values of the four vertices of the license plate area contained in the name of the complete license plate image as the true value of the license plate area; taking the license plate character information contained in the name of the complete license plate image as the true value of the license plate character ; The license plate area in each complete license plate image is cropped according to the coordinate values of the four vertices of the license plate area contained in the name of the complete license plate image to obtain a license plate area image; then the complete license plate images are divided into a complete license plate image training set, a complete license plate image verification set and a complete license plate image test set, and the license plate area images are divided into a license plate area image training set, a license plate area image verification set and a license plate area image test set.

3. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In step 1, in the process of obtaining the license plate area image, the cropped image is randomly scaled to perform image enhancement; random scaling is to stretch or reduce the length and width of the corresponding license plate area by 1 / 10.

4. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In step 2, when the intersection-over-union ratio between the predicted value of the license plate area and the true value of the license plate area of the same image obtained in step 1 exceeds 60%, it is determined that the license plate area detection is correct and is used in the subsequent calculation of the accuracy.

5. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In step 2, the training process of the YOLOv5 target detection network is as follows: the complete license plate image training set is input into the YOLOv5 target detection network for training, and the output result is the predicted value; the coordinate values of the four vertices of the license plate area contained in the name of the image in the complete license plate image training set are used as the true value; the smaller the difference between the predicted value and the true value, the more accurate the license plate area prediction, until the set target is reached, the training is completed, and a trained YOLOv5 target detection network is obtained; the set target is 300 rounds of iteration or the license plate area detection accuracy in the complete license plate image verification set reaches at least 95%.

6. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In step 3, the spatial transformation network is used to correct the license plate characters in the license plate frame, which includes three processes: parameter prediction, coordinate mapping, and output pixel. First, the input of the parameter prediction part is the license plate frame cropped by the predicted value of the license plate area, and the transformation matrix is obtained after the convolution operation. ; Then the coordinate mapping part uses the transformation matrix , and obtain the mapping relationship between the output image and the corresponding position of the input image; finally, the interpolation algorithm is used to calculate the value of each pixel in the output image, and the value of each pixel constitutes the corrected feature.

7. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In step 4, in the three identical convolutional layers, the number of convolution kernels in each convolution layer is 48, the convolution kernel size is 3*3, the stride is 1, the padding is 2, and the activation is performed by the ReLU activation function; In a convolutional layer, the number of convolution kernels is 1, the convolution kernel size is 7*7, the step size is 1, and padding=6; Step 4 is as follows: After the rectified features of size H×W×C are processed by three layers of convolution, a basic feature map F of size H×W×48 is obtained; then global maximum pooling and global average pooling are performed respectively to obtain two feature maps of size H×W×1; The two feature maps are then concatenated based on the channel to obtain a H×W×2 feature map. The feature map is then reduced to one dimension through a convolution layer with a convolution kernel size of 7*7. The feature map size is now H×W×1. The spatial attention feature is then obtained through a sigmoid activation function. ; Then the spatial attention feature Do matrix Hadamard product with the basic feature map F to get the final feature; The above process is expressed by formula (1): （1）。 8. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In steps 3 to 5, the spatial transformation network, spatial attention feature extraction module and character recognition network form a character recognition network based on spatial attention features; The training process of the character recognition network based on spatial attention features is as follows: the license plate area image training set is input into the character recognition network based on spatial attention features for training. Through steps 3 to 5, the output result is seven license plate characters as the predicted values of the license plate characters. ; Then the predicted value of the license plate character The true value of the license plate character obtained in step 1 Substitute into the loss function of formula (2) and get the loss function value ;Through the loss function value Continuously optimize the character recognition network based on spatial attention features. The smaller the loss function value, the more accurate the license plate character recognition is, until the set goal is reached. The training is completed and a trained character recognition network based on spatial attention features is obtained. The set goal is to iterate 50 rounds or the license plate character recognition accuracy in the license plate area image verification set reaches at least 98%. If and only if the predicted value is the same as the true value of the license plate character of the same image obtained in step 1 Only when the characters are exactly the same as each other can it be determined that the characters in the license plate area are correctly recognized; （3） In formula (3), N represents the number of training samples in the license plate area; Represents the true value of the seven license plate characters; represents the predicted value of seven license plate characters, Also includes floating point numbers, each representing the probability of belonging to a particular character.

9. The license plate recognition method in an unrestricted scene based on spatial attention features according to claim 1, characterized in that: In step 5, the character recognition network contains seven parallel character classifiers, each of which predicts a character at a fixed position on the license plate, so it can predict seven characters; Each character classifier contains 3 convolutional layers, 1 maximum pooling layer, 1 extension layer and 2 fully connected layers; Each convolution layer contains 24 convolution kernels, each convolution kernel size is 3*3, the stride is 1, and the padding is 2; the pooling kernel size of the maximum pooling layer is 2*2, and the stride is 1; The number of neurons in the first fully connected layer of each character classifier is 100; the number of neurons in the second fully connected layer of the first character classifier is 34, the number of neurons in the second fully connected layer of the second character classifier is 25, and the number of neurons in the second fully connected layer of the third to seventh character classifiers is 35. Step 5 is as follows: the final feature obtained in step 4 is processed by two layers of convolution to obtain a feature map of H×W×24, the feature is subjected to maximum pooling processing to obtain a feature map of H / 2×W / 2×24, the pooled feature is subjected to one more convolution processing to obtain a feature map of H / 2×W / 2×24; the H / 2×W / 2×24 feature map is stretched into a one-dimensional vector through the extension layer, and then passed through the first fully connected layer and the second fully connected layer, and finally the character with the highest probability at that position is predicted through the softmax function; seven classifiers predict a set of seven characters to identify all the characters in the entire license plate.

Images