Character non-segmentation mode license plate recognition method based on deep learning fusion network

Abstract

The application discloses a character non-segmentation mode license plate recognition method based on a deep learning fusion network, and comprises the following steps: detecting a license plate region of a high-speed vehicle image and constructing a highway license plate image set; based on a vehicle license plate image feature vector output by three layers of CTC loss function and ResNet50-LPR convolutional neural network block1, block3 and block5, a fusion model FResNet50 is constructed through serial fusion; based on an attention coding mode of license plate recognition, an attention coding module is added after the FResNet50 fusion model, and the construction of a FResNet50-Attention deep learning fusion network model is completed; the FResNet50-Attention convolutional neural network model constructed is trained by using the highway license plate image set, and finally the recognition of a vehicle license plate is completed. The recognition performance of the application is better than that of a single ResNet50-LPR convolutional neural network and a FResNet50 convolutional neural network fusion model under a traditional fusion mode, and the recognition accuracy of the whole license plate reaches 93.224%.

Description

TECHNICAL FIELD

[0001] The application relates to the field of intelligent traffic and wisdom highway, and can be applied to various scenes such as a traffic law enforcement system, a parking management system, a traffic guidance system, a highway inspection system, a vehicle scheduling system and a wisdom truck toll collection system in a highway scene. BACKGROUND

[0002] The license plate recognition technology is an application of a computer vision image algorithm in vehicle license plate recognition. It is a technology capable of detecting vehicles on a monitored road surface and automatically extracting vehicle license plate information (including Chinese characters, English letters, Arabic numerals and colors) for processing. The license plate recognition is one of important components in a modern intelligent traffic system and is widely applied. The license plate recognition is based on digital image processing, pattern recognition, computer vision and other technologies, analyzes vehicle images or video sequences captured by a camera, obtains a unique license plate number of each vehicle, and thus completes the recognition process. Through some subsequent processing means, the license plate recognition can realize parking lot toll collection management, traffic flow control index measurement, vehicle positioning, automobile theft prevention, highway overspeed automatic management, non-stop toll collection and other functions. The license plate recognition has practical significance for maintaining traffic safety and urban security, preventing traffic congestion and realizing traffic automation management.

[0003] With the rapid development of computer technology, the vehicle number plate detection and recognition technology based on deep learning has developed rapidly. Compared with traditional machine learning algorithms, deep learning does not need to manually extract features, and thus has good universality and adaptability and has been widely applied. However, due to the complex image environment of the vehicle number plate and the single recognition mode, the existing license plate recognition model has poor effect under the open-air scene characteristics of some highways, especially under special weather conditions such as rain, snow and fog, the light changes significantly, and the image noise is more, which greatly reduces the accuracy of vehicle number plate recognition. Therefore, based on the highway vehicle image under abnormal weather conditions, a character non-segmentation mode license plate recognition method based on a deep learning fusion network is proposed, and a FResNet50-Attention deep learning fusion network model is used as the core structure of the model, which further improves the recognition accuracy of the vehicle number plate in the character non-segmentation mode, and provides an effective new idea for license plate recognition. SUMMARY

[0004] The application aims to complete the recognition and classification of the vehicle number plate in the character non-segmentation mode by using the deep learning fusion network method, and provides a character non-segmentation mode license plate recognition method based on a deep learning fusion network.

[0005] The technical scheme adopted by the present application is: a character non-segmentation mode license plate recognition method based on a deep learning fusion network.

[0006] 1) Based on the high-speed vehicle image under abnormal weather conditions, the license plate region is detected and a highway license plate image set is constructed;

[0007] 2) Based on the CTC loss function, a character non-segmentation mode license plate recognition framework is constructed;

[0008] 3) A ResNet50-LPR convolutional neural network is constructed to extract features of vehicle license plate character images;

[0009] 4) Based on the vehicle license plate image feature vectors output by the three layers of ResNet50-LPR convolutional neural network block1, block3 and block5, a deep learning fusion network model FResNet50 is constructed by concatenating and fusing;

[0010] 5) Based on the attention encoding mode of license plate recognition, an attention encoding (Attention) module is added after the FResNet50 fusion model to complete the construction of the FResNet50-Attention deep learning fusion network model;

[0011] 6) The FResNet50-Attention convolutional neural network model constructed is trained using the highway license plate image set, and finally the vehicle license plate is recognized.

[0012] Further, the method of detecting the license plate region and constructing the highway license plate image set in step 1) is: based on the YOLOv3 abnormal weather condition highway vehicle holographic detection method, the constructed vehicle data set is detected to obtain vehicle holographic detection results under various weather conditions. The detection content includes vehicle window, vehicle face and license plate, then the obtained license plate image is screened and sorted, and the license plate label is corrected. Finally, 13733 highway license plate data sets under abnormal weather conditions are constructed, including 2144 rain day license plate images, 2505 snow day license plate images, 2151 fog day license plate images, 2256 night license plate images and 4677 normal weather condition license plate images.

[0013] Further, the method of constructing a character non-segmentation mode license plate recognition framework based on a CTC loss function in step 2) is: for a given sequence X = [x1, x2,..., x T ] and label data Y = [y1, y2,..., y UThe input sequence X is segmented into several time steps, and an output can be obtained for each time step. The recognition of all characters of the license plate can be completed by combining multiple time steps. Meanwhile, a blank character epsilon is introduced to represent the character interval and separate the repeated characters. At this time, the CTC algorithm can realize character alignment by removing repeated characters and blank characters epsilon, and solve the problem that X and Y are difficult to align due to the difference in length. Since there are repeated characters and blank characters epsilon, there are multiple possibilities for mapping from X to Y. In order to obtain the label data Y, it is finally necessary to maximize the posterior probability P(Y|X) of mapping from X to Y, that is, to maximize the sum of the probability of each time step from X to Y, and the posterior probability expression is:

[0014]

[0015] Accordingly, the specific expression of the CTC loss function can be formed as:

[0016] L CTC =max(∑ (X，Y)∈D -log P(Y|X))

[0017] Further, the method for constructing the ResNet50-LPR convolutional neural network in step 3) to extract vehicle license plate character image features is as follows: first, build a ResNet50 convolutional neural network structure. This network structure can be summarized as 5 Block modules, each of which is composed of a plurality of convolutional layers and pooling layers, and is additionally attached with a residual structure for branch connection; second, normalize the image to 224x224x3 and input it into the convolutional neural network; and finally, extract the feature maps output by Block1, Block3 and Block5 to obtain three feature map matrices with sizes of 64x56x56, 512x28x28 and 2048x1x1, respectively.

[0018] Further, the method for constructing the deep learning fusion network model FResNet50 based on the vehicle license plate image feature vectors output by the three layers of block1, block3 and block5 of the ResNet50-LPR convolutional neural network in step 4) is as follows: the feature maps extracted by Block1, Block3 and Block5 are serially fused based on the ResNet50-LPR convolutional neural network structure. First, reshape the feature vectors extracted by the three modules into features with sizes of 4x1x16, 32x1x16 and 128x1x16, respectively; then, combine the low-level feature information 4x1x16, the high-level feature information 32x1x16 and the high-level semantic information 128x1x16 to obtain a feature with a size of 164x1x16, which prepares for subsequent encoding and recognition operations.

[0019] Further, the step 5) attention encoding mode based on license plate recognition, an attention encoding (Attention) module is added after the FResNet50 fusion model, and the method for constructing the FResNet50-Attention deep learning fusion network model is: firstly, the feature map is globally pooled to obtain a feature of Sx1x1; secondly, a 1x1 convolution is used to convert the Sx1x1 feature into a S / Kx1x1 feature, which is activated by a Relu, and then a 1x1 convolution is used to convert the S / Kx1x1 feature into a Sx1x1 feature; then, a Sigmoid function is used to change the value in the Sx1x1 feature to between 0 and 1, that is, the weight Weight for the feature weighting is obtained; finally, the Weight is added to the L S-dimensional vectors output by the convolutional neural network to obtain the weighted L S-dimensional vectors. On this basis, a classifier is added, including a Dropout layer, a 1x1 convolution layer, a BatchNorm layer and a log_softmax classification function. The feature vector after attention encoding is input into the classifier, and 16 66-dimensional feature vectors representing the probability values of the 16 positions on the 66 characters are obtained, wherein the maximum probability in each vector is the character corresponding to the vector, so that 16 characters are obtained. The training result of the CTC loss function can remove the blank characters and repeated characters to obtain the final seven characters of the license plate on the expressway under abnormal weather conditions.

[0020] Further, in the step 6), the method for training the FResNet50-Attention convolutional neural network model and finally completing the recognition of the vehicle license plate is: inputting the normalized license plate image into the FResNet50-Attention convolutional neural network model, iteratively training, adjusting the weights of each layer of the network, and finally using the obtained network weight forward propagation to complete the license plate recognition in the character non-segmentation mode.

[0021] Compared with the prior art, the beneficial effects of the present application are that the recognition performance is better than that of a single ResNet50-LPR convolutional neural network, a ResNeXt-LPR-LONG convolutional neural network model based on a CTC loss function, and a FResNet50 convolutional neural network fusion model in a traditional fusion manner, and the recognition accuracy of the whole license plate (a total of 7 characters) reaches 93.224%. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A clear license plate image example for use.

[0023] Figure 2 An alignment strategy of the CTC algorithm.

[0024] Figure 3 The feature information and semantic information fusion model FResNet50 network model structure is based on.

[0025] Figure 4 The Attention encoding method is based on.

[0026] Figure 5 The deep learning fusion FResNet50-Attention network model structure is based on a non-segmentation mode license plate recognition. DETAILED DESCRIPTION

[0027] The application will be further illustrated below in combination with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the application and not to limit the scope of the application.

[0028] As shown in the figure, a character non-segmentation mode license plate recognition method based on a deep learning fusion network comprises the following steps:

[0029] First step: based on the high-speed whole vehicle image under abnormal weather conditions, the license plate region is detected and a highway license plate image set is constructed. First, taking the whole vehicle image actually collected on the highway as the benchmark, a highway whole vehicle dataset under abnormal weather conditions is constructed, and there are a total of 13988 vehicle images, of which 2372 are foggy vehicle images, 2645 are snowy vehicle images, 2248 are night vehicle images, 2112 are rainy vehicle images, and 4611 are normal weather vehicle images. Second, the YOLOv3 target detection model is used to detect the whole vehicle image collected on the highway under abnormal weather conditions to obtain the holographic detection results of the vehicle under various weather conditions, and the detection content includes the vehicle window, the vehicle face and the license plate. Then, the obtained license plate image is screened and sorted, and the license plate label is corrected to construct a highway license plate dataset under abnormal weather conditions, which contains 13733 license plate images, of which 2144 are rainy license plate images, 2505 are snowy license plate images, 2151 are foggy license plate images, 2256 are night license plate images, and 4677 are license plate images under normal weather conditions. Finally, the license plate data is sorted and the data label is confirmed, and the license plate is divided into license plates that can determine the license plate number and license plates that cannot determine the license plate number, of which 10316 can determine the license plate number and 3417 cannot determine the license plate number. For images that can determine the license plate number, although they have problems such as blur, rain and snow obstruction, low contrast, etc., they can clearly determine the license plate number information, and are retained and prepared for input into the neural network for training; for images that cannot determine the license plate number, their resolution is too low or too blurred, and they cannot determine the complete license plate number information, especially the Chinese character part of the license plate information, so they cannot determine the label, and this part of the license plate should not be used for subsequent experiments and should be deleted. The clear license plate images available for use are as follows: Figure 1as shown.

[0030] Second step: based on the CTC loss function, a character non-segmentation mode license plate recognition framework is constructed. In license plate recognition, the license plate character segmentation task has always been the focus and difficulty. Due to the difficulty in aligning the license plate position and characters, it is difficult for the model to converge. CTC (Connectionist temporal classification) is a method that can avoid direct alignment of input and output, which is very suitable for the task of highway license plate recognition under abnormal weather conditions. Assuming that there is a sequence X = [x1, x2,..., x T ] and label data Y = [y1, y2,..., y U ], the lengths of X and Y vary and are not equal, and X and Y are difficult to align. Finding a mapping between X and Y is an algorithm for classifying time series data called Temporal Classification. For a given input sequence X, CTC gives all possible outputs and their probabilities. The input sequence X is divided into several time steps, and each time step can get an output. For character recognition problems, due to the existence of character intervals and the possibility of repeated characters, the number of characters and time steps are not one-to-one corresponding, and the CTC algorithm introduces a blank character ε to represent the character interval and separate the repeated characters. At this time, the CTC algorithm only needs to remove the repeated characters and the blank character ε to finally realize character alignment, as shown in Figure 2

[0031] Due to the existence of repeated characters and blank characters ε, there are multiple possibilities for mapping from X to Y. If you want to get the label data Y, you need to maximize the posterior probability P(Y|X) of X mapping to Y. Assuming that the output of each time step is independent of each other, the probability of getting Y from X is the sum of the probability of each time step, that is,

[0032]

[0033] As a differentiable loss function, the CTC loss function calculates the negative logarithmic probability of all correct Y sequence paths, that is,

[0034] L CTC = max(Σ (X，Y)∈D -log P(Y|X))

[0035] ​Third step: Constructing ResNet50-LPR convolutional neural network to extract features of vehicle license plate character image. First, the ResNet50 convolutional neural network structure is built, which can be summarized as 5 Block modules, each module has a number of convolutional layers and pooling layers, and each is attached with a residual structure for branch connection. Second, the image is normalized to 224x224x3 and input into the convolutional neural network, and finally the feature maps output by Block1, Block3 and Block5 are extracted, the feature map output by Block1 is 64x56x56, the feature map output by Block3 is 512x28x28, and the feature vector output by Block5 is 2048x1x1. Three feature maps can be extracted from each vehicle license plate image.

[0036] Fourth step: Based on the vehicle license plate image feature vectors output by ResNet50-LPR convolutional neural network block1, block3 and block5, a deep learning fusion network model FResNet50 is constructed. According to the characteristics that the local position information is more obvious in the low-level convolution and the semantic information is more rich in the high-level convolution, the features containing obvious local position information in Block1 are taken as low-level feature information, the features containing local position information in Block3 are taken as high-level feature information, and the features containing rich semantic information in Block5 are taken as high-level semantic information. Therefore, the features extracted from Block1, Block3 and Block5 are serially fused based on the ResNet50-LPR convolutional neural network structure. The FResNet50 network model structure is as follows Figure 3As shown, a license plate image with a size of 3x244x244 is input into the ResNet50 network, the feature map output by Block1 has a size of 64x56x56, the feature map is subjected to a max-pooling operation and is changed into a feature vector with a size of 64x1x1, and then the feature vector is reshaped into a feature with a size of 4x1x16, and this feature is the low-level feature information of the license plate image; the feature map output by Block3 has a size of 512x28x28, the feature map is subjected to a max-pooling operation and is changed into a feature vector with a size of 512x1x1, and then the feature vector is reshaped into a feature with a size of 32x1x16, and this feature is the high-level feature information of the license plate image; finally, the feature vector output by Block5 has a size of 2048x1x1, the feature vector is reshaped into a feature with a size of 128x1x16, and this feature is the high-level semantic information of the license plate image, the low-level feature information 4x1x16, the high-level feature information 32x1x16 and the high-level semantic information 128x1x16 are combined to obtain a feature with a size of 164x1x16, and this feature integrates the local position information and the semantic information of the image, and further enhances the performance of the network.

[0037] Step 5: Based on the attention encoding mode of license plate recognition, an attention encoding (Attention) module is added after the FResNet50 fusion model to complete the construction of the FResNet50-Attention deep learning fusion network model. The attention mechanism is inspired by the attention mechanism of human vision. Human vision obtains the target area that needs to be focused on by quickly scanning the global image, that is, the so-called attention focus, and then invests more attention resources in this area to obtain more detailed information of the target that needs to be focused on, and suppresses other useless information, greatly improving the efficiency and accuracy of visual information processing. The core goal of the attention mechanism in deep learning is to select more critical information for the current task target from numerous information. Based on this, the application uses the attention encoding method to process after the fusion features. This method is inspired by the way of weighting the feature map channels in the SEBlock of the SE Net network, as shown in Figure 4As shown, for the L S-dimensional vectors output by the convolutional neural network, one dimension is added to convert it into the form of a feature map, taking S as the number of channels, L and 1 as the length and width of the feature map, to obtain an SxLx1 feature map. In the attention encoding module, first, the feature map is globally pooled to obtain an Sx1x1 feature; second, a 1x1 convolution is used to convert the Sx1x1 feature into an S / Kx1x1 feature, which is activated by a Relu function, and then a 1x1 convolution is used to convert the S / Kx1x1 feature into an Sx1x1 feature; then, the values in the Sx1x1 feature are converted to between 0 and 1 by using a Sigmoid function, i.e., a weight Weight for feature weighting is obtained; finally, the Weight is added to the L S-dimensional vectors output by the convolutional neural network to obtain the weighted L S-dimensional vectors.

[0038] A classifier is added after the attention encoding module, including a Dropout layer, a 1x1 convolutional layer, a BatchNorm layer and a log_softmax classification function, and the structure of the FResNet50-Attention network model after adding the classifier is as shown in Figure 5 The 164 dimensions in the 164x1x16 feature are converted to 66 dimensions by using the classifier, which is the same as the number of characters (adding a blank character class), the 66x1x16 feature is converted into 16 66-dimensional feature vectors, and log_softmax is used for classification to obtain the probability values of 16 vectors on 66 character classes, wherein the maximum probability in each vector is the character corresponding to the vector, thereby obtaining 16 characters. The CTC loss function is used to train the license plate recognition model, and the training result is decoded by removing the blank characters and repeated characters to obtain the final seven character results of the highway license plate under abnormal weather conditions.

[0039] Step 6: Use the highway license plate image set to train the FResNet50-Attention convolutional neural network model constructed, and finally complete the recognition of the vehicle license plate. The license plate image normalized to 3x244x244 is input into the FResNet50-Attention convolutional neural network model, and iterative training is performed to adjust the weights of each layer of the network. The training results show that after 40 epochs, the training loss function of the model no longer changes, representing that the model has basically converged, and finally the obtained network weight is used for forward propagation to complete the license plate recognition in the character non-segmentation mode.

[0040] The application provides a deep learning fusion network model (FResNet50-Attention) for character non-segmentation mode license plate recognition.

[0041] Table 1: Comparison of license plate recognition accuracy of FResNet50-Attention model with FResNet50 and ResNet50-Attention

[0042] Accuracy FResNet50-Attention FResNet50 ResNet50-Attention Training set 100.000％ 100.000％ 100.000％ Validation set 92.955％ 92.694％ 92.759％ Test set 93.224％ 93.158％ 92.697％ Time / s 0.00799 0.00791 0.00727

[0043] Table 2: Comparison of license plate recognition accuracy of FResNet50-Attention model with ResNeXt-LPR-LONG and ResNeXt-LPCR

[0044] FResNet50-Attention ResNeXt-LPR-LONG ResNeXt-LPCR Training set accuracy 100.000％ 100.000％ 89.299％ Validation set accuracy 92.955％ 92.172％ 88.421％ Test set accuracy 93.224％ 92.368％ 88.911％ Time / s 0.00799 0.01004 0.0687s

[0045] It should be noted that, for those skilled in the art, without departing from the principles of the application, a number of improvements and refinements can be made, which should also be considered within the scope of protection of the application. The components not explicitly described in the embodiments can be implemented using existing technology.

Claims

1. A method for character non-segmentation pattern license plate recognition based on deep learning fusion networks, characterized in that, Includes the following steps: 1) Based on high-speed vehicle images under abnormal weather conditions, detect license plate areas and construct a high-speed license plate image set; 2) Based on the CTC loss function, a character non-segmentation pattern license plate recognition framework is constructed; 3) Construct a ResNet50-LPR convolutional neural network to extract features from vehicle license plate character images; 4) The feature vectors of vehicle license plate images output from the three layers of the ResNet50-LPR convolutional neural network (block1, block3, and block5) are concatenated and fused to construct a deep learning fusion network model FResNet50. 5) Based on the attention encoding method for license plate recognition, an attention encoding module is added after the FResNet50 fusion model to complete the construction of the FResNet50-Attention deep learning fusion network model; 6) The constructed FResNet50-Attention convolutional neural network model is trained using a set of highway license plate images to finally complete the recognition of vehicle license plates; In step 3), the method for constructing a ResNet50-LPR convolutional neural network to extract features from vehicle license plate character images is as follows: The ResNet50-LPR convolutional neural network consists of 5 Block modules and two global average pooling layers. Each Block module consists of convolutional layers and pooling layers. The image is normalized to 224×224×3 and input into the convolutional neural network. After passing through 5 Block modules, the feature map size output by Block1 is 64×56×56, the feature map size output by Block3 is 512×28×28, and the feature vector size output by Block5 is 2048×1×1. Three feature maps of different sizes are extracted from each vehicle license plate image. In step 5), the attention encoding method based on license plate recognition adds an attention encoding module after the FResNet50 fusion model to construct the FResNet50-Attention deep learning fusion network model. The method is as follows: The attention encoding method borrows the SEBlock method in the SENet network to weight the feature map channels. For the L S-dimensional vectors output by the convolutional neural network, an additional dimension is added to transform them into feature maps. S is used as the number of channels, and L and 1 are used as the length and width of the feature map, resulting in an S×L×1 feature map. In the attention encoding module, the feature map is first globally pooled to obtain S×1×1 features. Then, a 1×1 convolution is used to transform the S×1×1 features into S / K×1×1 features. After ReLU activation, a 1×1 convolution is used to transform the S / K×1×1 features into S×1×1 features. Then, the Sigmoid function is used to change the values ​​in the S×1×1 features to between 0 and 1, which gives the weights for the features. Finally, the weights are superimposed on the L S-dimensional vectors output by the convolutional neural network to obtain the weighted L S-dimensional vectors.

2. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 1, characterized in that, In step 1), the method for detecting the license plate region and constructing a highway license plate image set is as follows: the constructed vehicle dataset is detected by the highway vehicle holographic detection method under abnormal weather conditions based on YOLOv3 to obtain vehicle holographic detection results under various weather conditions; the detection content includes vehicle windows, vehicle faces and license plates, and then the obtained license plate images are screened and sorted, and the license plate labels are corrected.

3. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 1, characterized in that, In step 2), the method for constructing a character non-segmentation pattern license plate recognition framework based on the CTC loss function is as follows: for a given license plate image containing a sequence of characters... and tag data The input sequence X is divided into multiple time steps, and each time step produces an output. The multiple time steps are combined to complete the recognition of all characters of the license plate.

4. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 3, characterized in that, Introducing whitespace characters This is used to indicate character spacing and to separate repeated characters. In this case, the CTC algorithm removes duplicate characters and whitespace characters. To achieve character alignment and solve the problem of difficulty in alignment due to unequal lengths between X and Y coordinates; this is achieved by addressing the issues of repeated characters and whitespace characters. Given the existence of X, there are multiple possibilities for mapping from X to Y. To obtain the labeled data Y, we ultimately need to find the posterior probability of mapping X to Y. The maximum is achieved by maximizing the sum of the probabilities of Y obtained from X at each time step; based on this, a specific expression for the CTC loss function is formed, and the model framework is built according to this loss function.

5. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 1, characterized in that, In step 4), the method for constructing the deep learning fusion network model FResNet50 by concatenating and fusing the feature vectors of the vehicle license plate image output from the three layers of the ResNet50-LPR convolutional neural network (block1, block3, and block5) is as follows: the backbone structure of the ResNet50-LPR convolutional neural network is the same as that of the ResNet50-LPR convolutional neural network, and the feature maps extracted from Block1, Block3, and Block5 are concatenated and fused on the basis of this network structure.

6. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 5, characterized in that, A license plate image of size 3×244×244 is input into a ResNet50 network. Block 1 outputs a feature map of size 64×56×56. This feature map is then max-pooled to become a feature vector of size 64×1×1. The feature vector is then reshaped to 4×1×16, which represents the low-level feature information of the license plate image. Block 3 outputs a feature map of size 512×28×28. This feature map is also max-pooled to become a feature vector of size 512×1×1. The feature vector is then reshaped to 4×1×16. The feature vector is reshaped to 32×1×16, which is the high-level feature information of the license plate image. The final feature vector output by Block5 is 2048×1×1. This feature vector is reshaped to 128×1×16, which is the high-level semantic information of the license plate image. The three features, the low-level feature information 4×1×16, the high-level feature information 32×1×16, and the high-level semantic information 128×1×16, are merged to obtain a feature of size 164×1×16. Subsequent classification operations are then performed based on this feature.

7. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 1, characterized in that, Following the attention encoding module, a classifier is added to complete license plate recognition for non-segmented character patterns. This classifier consists of a Dropout layer, a 1×1 convolutional layer, a BatchNorm layer, a log_softmax classification function, and a final CTC algorithm. It transforms the 164 dimensions of the 164×1×16 features into 66 dimensions, the same as the number of characters, and converts the 66×1×16 features into 16 66-dimensional feature vectors. Log_softmax is used for classification to obtain the probability values ​​of the 16 vectors for the 66 character classes. The character with the highest probability in each vector is the character corresponding to that vector, thus obtaining 16 characters. The CTC loss function is used to train the license plate recognition model, and the training results are decoded by removing whitespace and duplicate characters to obtain the final seven-character result of the highway license plate under abnormal weather conditions.

8. The method for character non-segmentation pattern license plate recognition based on deep learning fusion network according to claim 1, characterized in that, In step 6), the method for training the FResNet50-Attention convolutional neural network model to finally complete the recognition of vehicle license plates is as follows: input the standardized license plate image into the FResNet50-Attention convolutional neural network model, perform iterative training, adjust the weights of each layer of the network, and finally use the obtained network weights to propagate forward to complete the license plate recognition in the character non-segmentation mode.

Images