A method and system for recognizing vehicle license plates on roads under haze weather conditions

Abstract

The present invention discloses a method and system for vehicle license plate recognition under haze weather conditions; the recognition system includes an image dehazing module, a license plate detection module, and a license plate recognition module. The image dehazing module includes a multiple prior knowledge neural network model, an encoder, and a decoder. The encoder includes a convolutional layer, a two-dimensional graph inference module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolutional stacking layer, and a global pooling layer. The license plate detection module is used to detect the position of the license plate in the image. The license plate recognition module is used to recognize the license plate image block and obtain the license plate number text information. The present invention extracts feature codes from the vehicle image through convolution, then uses the two-dimensional graph inference module to perform graph projection on the feature codes, and then performs graph inference and graph reprojection processing on the nodes of the obtained spatial graph and channel graph respectively, eliminating the non-uniform haze in the image and fully retaining the texture and color of the image, improving the accuracy of license plate recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of license plate recognition, and particularly relates to a method for recognizing license plates of road vehicles under haze weather conditions. Background Art

[0002] In the complex environment of multi-source vehicle information, license plate recognition is crucial in road traffic safety supervision. With the development of computer vision technology and the increasing maturity of sensor hardware, license plate detection technology under good weather conditions has basically met the actual use requirements. However, vehicle images captured in bad weather such as haze have a large amount of noise, and problems such as low contrast and high whitening degree pose certain challenges to the accuracy and robustness of vehicle recognition. Although many different network architectures have been explored for vehicle recognition technology based on deep learning, there are still problems of limited utilization of context information, difficulty in removing non-uniform haze, and restoring image details and colors. Therefore, the present invention effectively eliminates non-uniform haze and solves the problem of license plate recognition of road vehicles under haze weather conditions by using the interaction of different regions and different dimensions with similar structures captured by two-dimensional graph reasoning. Summary of the Invention

[0003] In order to solve the problem of limited utilization of context information in license plate recognition under haze weather, the present invention proposes a method for recognizing license plates of road vehicles under haze weather conditions, and its specific technical solutions are as follows.

[0004] In a first aspect, the present invention provides a method for recognizing license plates of road vehicles under haze weather conditions, including the following steps:

[0005] Step 1: Image dehazing processing.

[0006] Step 1-1: Detect the area containing vehicle information in the image to be processed to obtain a vehicle image block.

[0007] Step 1-2: Convolve the vehicle image block to obtain a feature encoding X.

[0008] Step 1-3: Perform graph projection, graph reasoning, and graph reprojection processing on the feature encoding X in sequence. In graph projection, the feature encoding X is projected onto a spatial graph and a channel graph to obtain node features V of the spatial graph and the channel graph i . Graph reasoning interacts different node features to obtain features Graph reprojection reprojects the spatial graph nodes and the channel graph nodes back to the original pixel space respectively to obtain reprojection features F corresponding to the spatial graph and the channel graph i .

[0009] Step 1-4: Combine the reprojection features F corresponding to the spatial graph and the channel graph iPooling, residual connection, normalization, activation function, multi-layer convolution stacking, and global pooling are performed in sequence, followed by decoding to obtain the dehazed image.

[0010] Step 2: Detect the image obtained in Step 1 to obtain the license plate position, and get the license plate image block.

[0011] Step 3: Identify the license plate image block obtained in Step 2 to obtain the license plate information.

[0012] Preferably, the image processed in Step 1 is obtained by frame processing of the video captured by a camera on the road.

[0013] Preferably, the specific process of obtaining the vehicle image block in Step 1-1 is as follows: Aggregate the prior knowledge of multiple traffic scene sites, transform it into non-linear inequality constraints, and construct a multi-prior knowledge neural network model; input the image to be processed into the multi-prior knowledge neural network model, obtain the area where the vehicle reasonably appears, and get the vehicle image block.

[0014] Preferably, in Step 1-3, project the feature encoding X to the node feature V of the spatial graph and the channel graph i The expression is:

[0015]

[0016] According to the node feature V i Construct the connectivity matrix A i as follows:

[0017]

[0018] where a i (p,q) is the element in the p-th row and q-th column of the connectivity matrix A i indicating the connectivity from the feature V of the p-th node i (p) to the node feature V of the q-th i (q); N represents the total number of graph nodes; θ(·) and θ'(·) represent two linear embeddings with different parameters.

[0019] Preferably, in Step 1-3, the feature obtained by the interaction of different node features The expression is as follows:

[0020]

[0021] where σ(·) represents the activation function, and W i belongs to the weight of the graph convolution.

[0022] Preferably, the specific process of Step 2 is as follows:

[0023] Step 2-1: Use the corner points of the image obtained in Step 1 as feature points; use optical flow calculation based on the Lucas-Kanade constraint method to predict the positions of the feature points in the next frame of the image; describe the motion vectors and generate an optical flow map; screen out valid feature points with speed as the threshold, eliminate noise feature points, obtain the position information and external contour of the vehicle, and determine the range interval where the license plate exists. Step 2-1: Convert the image to the GRAY color space; perform median filtering and edge detection on the image, and erode interference features; perform Hough transform on the range interval where the license plate exists determined in Step 2-1 to obtain line features, delimit the license plate range and crop the image to obtain a license plate image block.

[0024] Step 2-1: Convert the image to the GRAY color space; perform median filtering and edge detection on the image, and erode interference features; perform Hough transform on the range interval where the license plate exists determined in Step 2-1 to obtain line features, delimit the license plate range and crop the image to obtain a license plate image block.

[0025] Preferably, the specific process of Step 3 is as follows:

[0026] Input the license plate image block obtained in Step 2 into a spatial transformer to obtain a flat license plate image; input the flat license plate image into a license plate inference model to identify the license plate information.

[0027] Preferably, the training process of the inference model is as follows: Use a backbone convolutional neural network to extract the convolutional features of the license plate, and then use the cross-entropy loss of the attention classifier to train the spatial transformer, the backbone convolutional neural network, and the attention branch. Use the connectionist temporal classifier to train the connection time branch, fuse various expressions of text features, and obtain a trained license plate inference model; perform pruning optimization on the license plate inference model. The finally obtained license plate inference model only retains the connectionist temporal classifier to identify the flat license plate image.

[0028] In a second aspect, the present invention provides an identification system for implementing the foregoing road vehicle license plate recognition method, which includes an image dehazing module, a license plate detection module, and a license plate recognition module. The image dehazing module includes a multi-prior knowledge neural network model, an encoder, and a decoder. The encoder includes a convolutional layer, a two-dimensional graph inference module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolutional stacking layer, and a global pooling layer. The license plate detection module is used to detect the position of the license plate in the image processed by the image dehazing module and frame out the image block containing the license plate number. The license plate recognition module is used to recognize the image block detected by the license plate detection module to obtain the license plate number text information.

[0029] The specific beneficial effects of the present invention are as follows:

[0030] 1. The present invention extracts feature codes from vehicle images captured by cameras on traffic roads through convolution, then uses a two-dimensional graph reasoning module to perform graph projection on the feature codes, and then performs graph reasoning and graph reprojection processing on the nodes of the obtained spatial graph and channel graph respectively, solving the problem of image blurring caused by haze environment. Compared with traditional road vehicle license plate recognition methods, it can recognize more blurred complex scenes and effectively cope with harsh weather such as haze.

[0031] 2. The present invention introduces a graph reasoning module to eliminate the pollution of haze to images, simplifies the pixel similarity of the non-local mean filter into more compact node similarity through a graph data structure, and improves the calculation efficiency.

[0032] 3. The present invention establishes and captures the interactions of different regions and different dimensions with similar structures through a spatial reasoning module and a channel reasoning module, eliminates non-uniform haze, well preserves the texture and color of the image, and solves the problem of limited utilization of context information in license plate recognition in haze weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic flowchart of the present invention;

[0034] Figure 2 is a schematic diagram of the image dehazing module in the present invention;

[0035] Figure 3 is a schematic framework diagram of the two-dimensional graph reasoning module in the image dehazing module of the present invention;

[0036] Figure 4 is a schematic diagram of the working process of the license plate detection module in the present invention;

[0037] Figure 5 is a schematic framework diagram of the license plate recognition module in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] In order to make the objectives, technical solutions, and technical effects of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings of the specification and embodiments.

[0039] As Figure 1 shown, a road vehicle license plate recognition method in a haze weather condition provided by an embodiment of the present invention uses an identification system including an image dehazing module, a license plate detection module, and a license plate recognition module.

[0040] The image dehazing module is used to remove the blurred state caused by fog in the image, making the image clear. The image dehazing module includes a multi-prior knowledge neural network model, an encoder, and a decoder. The encoder includes a convolutional layer, a two-dimensional graph inference module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolutional stack, and a global pooling.

[0041] The license plate detection module is used to detect the position of the license plate in the image processed by the image dehazing module and frame out the image block containing the license plate number.

[0042] The license plate recognition module is used to recognize the image block detected by the license plate detection module and obtain the license plate number text information.

[0043] The license plate recognition method for road vehicles includes the following steps:

[0044] Step S100: Communicate with the camera, obtain the image at the target checkpoint and perform frame processing, and then input the frame-level image into the image dehazing module;

[0045] Step S101: As Figure 2 shown, explore and utilize the prior knowledge of the traffic scene on-site, organically aggregate various prior knowledge, and obtain the image that may contain vehicle information in the input image; the specific process is as follows: explore and utilize the prior knowledge of the traffic scene on-site, after organically aggregating various prior knowledge, transform it into non-linear inequality constraints, and construct a multi-prior knowledge neural network model; input the captured image into the multi-prior knowledge neural network model, obtain the area where vehicles reasonably appear, and obtain the image that may contain vehicle information.

[0046] Step S102: Input the image containing vehicle information obtained in Step S101 into the convolutional layer of the encoder to obtain the feature encoding X. The dimensions of the feature encoding X obtained from all images are the same.

[0047] Step S103: As Figure 3 shown, use two-dimensional graph inference to perform non-local filtering on the spatial dimension and the channel dimension of the encoding matrix of the image information, so as to obtain new pixel features. The specific process is as follows:

[0048] (1) Graph projection: Project the feature encoding X onto the node features V i of the spatial graph and the channel graph, and its formula is expressed as follows:

[0049]

[0050] where the subscript i represents the serial number of the graph inference module used in different structures; i = 1 corresponds to the spatial graph; i = 2 corresponds to the channel graph; V i represents the graph node feature; Represents a spatial graph or channel graph structure, and the subscript proj represents the graph inference structure used in different dimensions, B i Represents the corresponding graph projection matrix, and × represents matrix multiplication.

[0051] Based on the dependence of nodes and the differences between nodes, a connectivity matrix A is constructed through the adjacency matrix between each pair of nodes i , and its formula is expressed as follows:

[0052]

[0053] where a i (p,q) is the element in the p-th row and q-th column of the connectivity matrix A i , representing the connectivity from the feature V i (p) of the p-th node to the node feature V i (q) of the q-th node; N represents the total number of graph nodes; θ(·) and θ'(·) represent two linear embeddings with different parameters.

[0054] (2) Graph inference: Input the graph node features V of the spatial graph and the channel graph i into the graph convolutional neural network respectively to learn the connections between nodes and infer the relationships by propagating information across nodes to simulate the long-term dependencies between nodes. Its formula is expressed as follows:

[0055]

[0056] where represents the feature obtained by the interaction of each node, σ(·) represents the activation function, and W i belongs to the weight of the graph convolution. Since the graph convolutional neural network has the receptive fields of all nodes in the graph, it can capture the global context of the input and simulate long-term dependencies.

[0057] (3) Graph reprojection: Reproject the spatial graph nodes and channel graph nodes in the graph space back to the original pixel space respectively. The reprojection features F corresponding to the spatial graph and the channel graph i , and its formula is expressed as follows:

[0058]

[0059] where i = 1, 2; F1 and F2 represent the reprojection features corresponding to the spatial graph and the channel graph respectively; represents the graph reprojection; D i represents the corresponding graph projection matrix.

[0060] Step S104: The reprojection features F corresponding to the spatial graph and the channel graph iA pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolutional stacking layer, and a global pooling layer that are connected in sequence in the common input encoder; the obtained features are then input into the decoder to obtain a defogged image. Its formula is expressed as follows:

[0061]

[0062] Among them, represents the finally generated defogged image, represents the remaining layers of the encoder except the convolutional layer and the two-dimensional graph inference module and the decoder network.

[0063] Step S200: As Figure 4 shown, the license plate detection module uses optical flow calculation based on the Lucas-Kanade constraint method for a series of defogged images to obtain the position information and external contour of the vehicle, and determines the range interval where the license plate may exist. The specific process is as follows: Obtain consecutive defogged images; Detect image corner points as feature points; Use optical flow calculation based on the Lucas-Kanade constraint method to predict the positions of the feature points in the next frame; Describe the motion vectors and generate an optical flow map; Screen valid feature points with speed as the threshold, eliminate noise feature points, obtain the position information and external contour of the vehicle, and determine the range interval where the license plate may exist.

[0064] Step S201: Combine the actual license plate size, perform Hough transform on the range interval where the license plate may exist, further determine the license plate position and crop the original image to obtain a license plate photo. The specific process is as follows: Convert the color space of the original image to the GRAY color space; Perform median filtering and edge detection on the image, erode interference features, and initially separate the license plate area; Use Hough transform to obtain line features, delimit the license plate range and crop the original image.

[0065] Step S300: As Figure 5 shown, the license plate recognition module inputs the training set containing skewed license plates into a spatial transformer to obtain a straight license plate image, then uses a backbone convolutional neural network to extract the convolutional features of the license plate, and then uses the cross-entropy loss of the attention classifier to train the spatial transformer, the backbone network, and the attention branch, and uses the connectionist temporal classifier to train the connection time branch, and fuses the expressions of multiple text features to obtain a trained license plate model;

[0066] Step S301: Obtain the weight parameters after training the training model, perform pruning optimization on the training model and then convert it into an inference model. Input the license plate photo obtained by the license plate recognition module into the inference model, only retain the connectionist temporal classifier for text prediction, and quickly output the final recognition result.

[0067] Compared with traditional license plate recognition methods for road vehicles, adding an image dehazing module can recognize more blurred complex scenes and achieve end-to-end license plate recognition in haze weather. At the same time, introducing a graph reasoning module simplifies the pixel similarity of the non-local mean filter into a more compact node similarity, improving the computational efficiency. And by using the spatial reasoning module and the channel reasoning module to establish and capture the interactions of different regions and different dimensions with similar structures, non-uniform haze is eliminated, the texture and color of the image are well preserved, and the problem of limited utilization of context information in license plate recognition in haze weather is solved, having broad application prospects.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing vehicle license plates on roads under smoggy weather conditions, characterized in that: It includes the following steps: Step 1, image defogging processing; Step 1-1. Detect the area containing vehicle information in the processed image to obtain vehicle image blocks. The specific process of obtaining vehicle image blocks is as follows: Aggregate the prior knowledge of multiple traffic scene sites, transform it into non-linear inequality constraints, and construct a multi-prior knowledge neural network model; Input the processed image into the multi-prior knowledge neural network model to obtain the area where vehicles reasonably appear, and obtain vehicle image blocks; Step 1-2: Convolve the vehicle image blocks to obtain the feature encoding X; Step 1-3: Perform graph projection, graph reasoning, and graph reprojection processing on the feature encoding X in sequence; in graph projection, project the feature encoding X onto the spatial graph and the channel graph to obtain the node features V of the spatial graph and the channel graph respectively i ; Graph reasoning performs interaction on different node features to obtain features Graph reprojection reprojects the spatial graph nodes and the channel graph nodes back to the original pixel space respectively to obtain the reprojection features F corresponding to the spatial graph and the channel graph i ; Steps 1-4: Input the corresponding reprojection features F of the spatial map and the channel map i Add into the successively connected pooling layer, residual connection layer, normalization layer, activation function, multi-layer convolutional stacking layer, and global pooling layer; The obtained features are then input into the decoder to obtain the dehazed image Step 2: Detect the image obtained in Step 1 to obtain the license plate position and get the license plate image block; Step 3, identify the license plate image blocks obtained in Step 2 to obtain license plate information.

2. The method for recognizing vehicle license plates on roads under smoggy weather conditions according to claim 1, characterized in that: The image processed in Step 1 is obtained by frame-by-frame processing of the images collected by the cameras on the road.

3. The method for recognizing vehicle license plates on roads under smoggy weather conditions according to claim 1, characterized in that: In steps 1-3, project the feature encoding X onto the node features V of the spatial graph and the channel graph i The expression of which is: Among them, represents a spatial graph or a channel graph structure; According to node feature V i Construct the connectivity matrix A i as follows: Among them, a i (p, q) is the connectivity matrix A i The element in the p-th row and q-th column, representing the connectivity from the feature V i (p) of the p-th node to the feature V i (q) of the q-th node; N represents the total number of graph nodes; θ(·) and θ'(·) represent two linear embeddings with different parameters.

4. The method for recognizing vehicle license plates on roads under smoggy weather conditions according to claim 3, characterized in that: In Steps 1-3, the features obtained by the interaction of different node features are expressed as follows: where, σ(·) represents the activation function, and W i belongs to the weights of graph convolution.

5. The method for recognizing vehicle license plates on roads under smoggy weather conditions according to claim 1, characterized in that: The specific process of Step 2 is as follows: Step 2-1. Use the corner points of the image obtained in Step 1 as feature points; use optical flow calculation based on the Lucas-Kanade constraint method to predict the positions of the feature points in the next frame of the image; describe the motion vectors and generate an optical flow map; Use speed as a threshold to screen out valid feature points, eliminate noise feature points, obtain the position information and external contour of the vehicle, and determine the range interval where the license plate exists; Step 2-2: Convert the image to the GRAY color space; perform median filtering and edge detection on the image, and erode the interference features; Perform the Hough transform on the range interval where the license plate exists determined in Step 2-1 to obtain line features, delimit the license plate range and crop the image to obtain license plate image blocks.

6. The method for recognizing vehicle license plates on roads under smoggy weather conditions according to claim 1, characterized in that: The specific process of Step 3 is as follows: Input the license plate image blocks obtained in Step 2 into the spatial transformer to obtain a flat license plate image; Input the flat license plate image into the license plate inference model to identify the license plate information.

7. The method for recognizing vehicle license plates on roads under smoggy weather conditions according to claim 1, characterized in that: The training process of the inference model is as follows: Use the backbone convolutional neural network to extract the convolutional features of the license plate, and then use the cross-entropy loss of the attention classifier to train the spatial transformer, the backbone convolutional neural network and the attention branch, use the connectionist temporal classifier to train the connection time branch, fuse the expressions of multiple text features, and obtain the trained license plate inference model; Perform pruning and optimization on the license plate inference model; The finally obtained license plate inference model only retains the connectionist temporal classifier to identify the flat license plate image.

8. A vehicle license plate recognition system for roads, characterized in that: The road vehicle license plate recognition method according to claim 1; The road vehicle license plate recognition system includes an image defogging module, a license plate detection module and a license plate recognition module; The image defogging module includes a multi-prior knowledge neural network model, an encoder and a decoder; The encoder includes a convolutional layer, a two-dimensional graph inference module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolutional stacking layer and a global pooling layer; The license plate detection module is used to detect the position of the license plate in the image processed by the image defogging module and frame out the image blocks containing the license plate number; The license plate recognition module is used to identify the image blocks detected by the license plate detection module to obtain the license plate number text information.

Images