Road vehicle license plate recognition method and system under haze weather condition

Abstract

The invention discloses a road vehicle license plate recognition method and system under a haze weather condition. The recognition system comprises an image defogging module, a license plate detection module and a license plate recognition module. The image defogging module comprises a multi-priori knowledge neural network model, an encoder and a decoder. The encoder comprises a convolution layer, a two-dimensional graph reasoning module, a pooling layer, a residual connection layer, a standardization layer, an activation function, a multi-layer convolution accumulation layer and a global pooling layer. The license plate detection module is used for detecting the position of a license plate in the image. The license plate recognition module is used for recognizing the license plate image blocks and obtaining license plate number text information. According to the method, feature codes are extracted from vehicle images through convolution, graph projection is carried out on the feature codes by utilizing a two-dimensional graph reasoning module, graph reasoning and graph re-projection processing are respectively carried out on nodes of an obtained space graph and a channel graph, non-uniform haze in the images is eliminated, and textures and colors of the images are fully reserved; the accuracy of license plate recognition is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of license plate recognition, and in particular relates to a method for recognizing license plates of road vehicles under haze weather conditions. Background Art

[0002] In a complex environment with multiple sources of vehicle information, license plate recognition is the key to road traffic safety supervision. With the development of computer vision technology and the increasing maturity of sensor hardware, license plate detection technology in good weather has basically met the needs of practical use. However, vehicle images captured in bad weather such as haze have a lot of noise, low contrast, high whitening and other problems, which bring certain challenges to the accuracy and robustness of vehicle recognition. Although many different network architectures have been explored for vehicle recognition technology based on deep learning, there is still a problem of limited use of contextual information, making it difficult to remove non-uniform haze and restore the details and colors of the image. Therefore, the present invention effectively eliminates non-uniform haze by using two-dimensional graph reasoning to capture the interactions of different regions and different dimensions with similar structures, and solves the problem of road vehicle license plate recognition under haze weather conditions. Summary of the invention

[0003] In order to solve the problem of limited use of context information in license plate recognition under haze weather, the present invention proposes a method for road vehicle license plate recognition under haze weather conditions, and its specific technical solution is as follows.

[0004] In a first aspect, the present invention provides a method for recognizing a license plate of a road vehicle under haze weather conditions, comprising the following steps:

[0005] Step 1: Image dehazing processing.

[0006] Step 1-1: Detect the area containing vehicle information in the processed image to obtain the vehicle image block.

[0007] Step 1-2: Convolve the vehicle image block to obtain the feature code X.

[0008] Step 1-3: Perform graph projection, graph reasoning, and graph reprojection on the feature code X in sequence. In graph projection, the feature code X is projected onto the spatial graph and the channel graph to obtain the node features V of the spatial graph and the channel graph respectively. i Graph reasoning interacts with different node features to obtain features Graph reprojection reprojects the spatial graph nodes and channel graph nodes to the original pixel space, and obtains the reprojection features F corresponding to the spatial graph and channel graph respectively. i .

[0009] Step 1-4: Reproject the features F corresponding to the spatial map and channel map iPooling, residual connection, standardization, activation function, multi-layer convolution stacking, global pooling and decoding are performed in sequence to obtain the defogged image.

[0010] Step 2: Detect the image obtained in step 1 The license plate position in is obtained to obtain the license plate image block.

[0011] Step 3: Recognize the license plate image block obtained in step 2 to obtain the license plate information.

[0012] Preferably, the image processed in step 1 is obtained by collecting images from a camera on the road and performing frame processing.

[0013] Preferably, the specific process of obtaining the vehicle image block in step 1-1 is: aggregating the on-site prior knowledge of various traffic scenes, converting it into nonlinear inequality constraints, and constructing a multiple prior knowledge neural network model; inputting the processed image into the multiple prior knowledge neural network model, obtaining the area where the vehicle reasonably appears, and obtaining the vehicle image block.

[0014] Preferably, in steps 1-3, the feature code X is projected onto the node features V of the spatial graph and the channel graph. i The expression is:

[0015]

[0016] According to the node feature V i Construct the connectivity matrix A i as follows:

[0017]

[0018] Among them, a i (p,q) is the connectivity matrix A i The element in the pth row and qth column represents the feature V from the pth node i (p) to the qth node feature V i (q); N represents the total number of graph nodes; θ(·) and θ'(·) represent two linear embeddings with different parameters.

[0019] As a preference, in steps 1-3, the features obtained by interaction of different node features are The expression is as follows:

[0020]

[0021] Among them, σ(·) represents the activation function, W i Weights belonging to the graph convolution.

[0022] As a preference, the specific process of step 2 is:

[0023] Step 2-1: The image obtained in step 1 The corner points are taken as feature points; the position of the feature points in the next frame of the image is predicted by optical flow calculation based on the Lucas-Koenig constraint method; the motion vector is described and an optical flow map is generated; the speed is used as the threshold to filter out valid feature points, eliminate noise feature points, obtain the vehicle's location information and shape contour, and determine the range of the license plate.

[0024] Step 2-1: Place the image Convert to GRAY color space; perform median filtering and edge detection on the image to erode interference features; perform Hover transform on the range of the license plate determined in step 2-1 to obtain line features, define the license plate range and crop the image to obtain the license plate image block.

[0025] As a preference, the specific process of step three is:

[0026] The license plate image block obtained in step 2 is input into the spatial transformer to obtain a flat license plate image; the flat license plate image is input into the license plate inference model to recognize the license plate information.

[0027] As a preferred method, the training process of the inference model is as follows: using the backbone convolutional neural network to extract the convolutional features of the license plate, then using the cross entropy loss of the attention classifier to train the spatial transformer, the backbone convolutional neural network and the attention branch, using the connection time classifier to train the connection time branch, integrating the expression of multiple text features, and obtaining the trained license plate inference model; the license plate inference model is pruned and optimized. The license plate inference model finally obtained only retains the connection time classifier to recognize the flat license plate image.

[0028] In a second aspect, the present invention provides a recognition system for executing the aforementioned road vehicle license plate recognition method, which includes an image defogging module, a license plate detection module and a license plate recognition module. The image defogging module includes a multiple prior knowledge neural network model, an encoder and a decoder. The encoder includes a convolution layer, a two-dimensional graph reasoning module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolution stacking layer and a global pooling layer. The license plate detection module is used to detect the position of the license plate in the image processed by the image defogging module, and select the image block containing the license plate number. The license plate recognition module is used to recognize the image block detected by the license plate detection module and obtain the text information of the license plate number.

[0029] The specific beneficial effects of the present invention are as follows:

[0030] 1. The present invention extracts feature codes from vehicle images captured by cameras on traffic roads through convolution, and then uses a two-dimensional graph reasoning module to perform graph projection on the feature codes. Then, the nodes of the obtained spatial graph and channel graph are respectively subjected to graph reasoning and graph reprojection processing to solve the image blur problem caused by haze environment. Compared with the traditional road vehicle license plate recognition method, it can recognize more blurred and complex scenes, and effectively cope with severe weather such as haze.

[0031] 2. The present invention introduces a graph reasoning module to eliminate the pollution of haze on the image, and simplifies the pixel similarity of the non-local mean filter into a more compact node similarity through the graph data structure, thereby improving the calculation efficiency.

[0032] 3. The present invention establishes and captures the interactions of different regions and different dimensions with similar structures through the spatial reasoning module and the channel reasoning module, eliminates non-uniform haze, well preserves the texture and color of the image, and solves the problem of limited use of contextual information in license plate recognition under haze weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 is a schematic diagram of an image defogging module in the present invention;

[0035] Figure 3 It is a schematic diagram of the framework of the two-dimensional graph reasoning module in the image defogging module of the present invention;

[0036] Figure 4 It is a schematic diagram of the working process of the license plate detection module in the present invention;

[0037] Figure 5 It is a schematic diagram of the framework of the license plate recognition module in the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments of the specification.

[0039] like Figure 1 As shown, an embodiment of the present invention provides a method for recognizing license plates of road vehicles under haze weather conditions, and the recognition system used includes an image defogging module, a license plate detection module and a license plate recognition module.

[0040] The image defogging module is used to remove the blur caused by fog in the image, making the image clear. The image defogging module includes a multiple prior knowledge neural network model, an encoder and a decoder. The encoder includes a convolution layer, a two-dimensional graph reasoning module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolution stacking and a global pooling.

[0041] The license plate detection module is used to detect the position of the license plate in the image processed by the image defogging module and select the image block containing the license plate number.

[0042] The license plate recognition module is used to recognize the image blocks detected by the license plate detection module and obtain the license plate number text information.

[0043] The road vehicle license plate recognition method comprises the following steps:

[0044] Step S100: Communicate with the camera, obtain the image at the target camera port and perform frame processing, and then input the frame-level image into the image defogging module;

[0045] Step S101: Figure 2 As shown, the prior knowledge of the traffic scene is explored and utilized, various prior knowledge is organically aggregated, and an image that may contain vehicle information in the input image is obtained; the specific process is: the prior knowledge of the traffic scene is explored and utilized, various prior knowledge is organically aggregated, and then converted into nonlinear inequality constraints, and a multiple prior knowledge neural network model is constructed; the captured image is input into the multiple prior knowledge neural network model, the area where the vehicle reasonably appears is obtained, and an image that may contain vehicle information is obtained.

[0046] Step S102: Input the image containing vehicle information obtained in step S101 into the convolution layer of the encoder to obtain feature code X. The dimension of feature code X obtained for all images is the same.

[0047] Step S103: Figure 3 As shown in the figure, the encoding matrix of the image information is non-locally filtered in the spatial dimension and channel dimension using two-dimensional graph reasoning to obtain new pixel features. The specific process is as follows:

[0048] (1) Graph projection: Project the feature code X to the node features V of the spatial graph and channel graph i , the formula is as follows:

[0049]

[0050] Wherein, the subscript i represents the serial number of the graph reasoning module used in different structures; i = 1 corresponds to the spatial graph; i = 2 corresponds to the channel graph; V i Represents graph node features; represents a spatial graph or channel graph structure, and the subscript proj represents the graph reasoning structure used in different dimensions. i represents the corresponding graphics projection matrix and × denotes matrix multiplication.

[0051] Based on the dependencies between nodes and differences between nodes, the connectivity matrix A is constructed by the adjacency matrix between each pair of nodes. i , the formula is as follows:

[0052]

[0053] Among them, a i (p,q) is the connectivity matrix A i The element in the pth row and qth column represents the feature V from the pth node i (p) to the qth node feature V i (q); N represents the total number of graph nodes; θ(·) and θ'(·) represent two linear embeddings with different parameters.

[0054] (2) Graph Reasoning: The graph node features V of the spatial graph and channel graph are transformed into i , respectively input into the graph convolutional neural network Learn the connections between nodes and reason about the relationships by propagating information across nodes to simulate the long-term dependencies between nodes. The formula is as follows:

[0055]

[0056] in, represents the features obtained by the interaction of each node, σ(·) represents the activation function, and W i The weights of the graph convolution. Since the graph convolutional neural network It has the receptive field of all nodes in the graph, so it can capture the global context of the input and simulate long-term dependencies.

[0057] (3) Graph reprojection: The spatial graph nodes and channel graph nodes in the graph space are reprojected to the original pixel space, and the reprojection features F corresponding to the spatial graph and channel graph are obtained. i , the formula is as follows:

[0058]

[0059] Where i = 1, 2; F1 and F2 represent the reprojection features corresponding to the spatial map and channel map; Represents image reprojection; D i Represents the corresponding graph projection matrix.

[0060] Step S104: Reproject the spatial graph and the channel graph corresponding to the reprojection feature F iThe pooling layer, residual connection layer, normalization layer, activation function, multi-layer convolution stacking layer, and global pooling layer connected in sequence in the encoder are input together; the obtained features are then input into the decoder to obtain the dehazed image The formula is as follows:

[0061]

[0062] in, represents the final dehazed image, Represents the remaining layers of the encoder except the convolutional layers and the 2D graph reasoning module and the decoder network.

[0063] Step S200: Figure 4 As shown, the license plate detection module uses optical flow calculation based on the Lucas-Koenig constraint method on the series of defogging images to obtain the vehicle's position information and shape outline, and determine the range interval where the license plate may exist in the image. The specific process is: obtain continuous images after defogging; detect image corners as feature points; use optical flow calculation based on the Lucas-Koenig constraint method to predict the position of feature points in the next frame; describe motion vectors and generate optical flow maps; use speed as a threshold to filter valid feature points, eliminate noise feature points, obtain the vehicle's position information and shape outline, and determine the range interval where the license plate may exist.

[0064] Step S201: Based on the actual license plate size, perform Hover transform on the range where the license plate may exist, further determine the license plate position and crop the original image to obtain a license plate photo. The specific process is: convert the color space of the original image to the GRAY color space; perform median filtering and edge detection on the image, erode interference features, and preliminarily separate the license plate area; use Hover transform to obtain line features, delineate the license plate range and crop the original image.

[0065] Step S300: Figure 5 As shown in the figure, the license plate recognition module inputs the training set containing crooked license plates into the spatial transformer to obtain a straight license plate image, and then uses the backbone convolutional neural network to extract the convolution features of the license plate. After that, the cross entropy loss of the attention classifier is used to train the spatial transformer, the backbone network and the attention branch, and the connection time classifier is used to train the connection time branch. The expression of multiple text features is integrated to obtain the trained license plate model;

[0066] Step S301: Obtain the weight parameters of the training model after training, and convert the training model into an inference model after trimming and optimizing. Input the license plate photo obtained by the license plate recognition module into the inference model, retain only the connection time series classifier for text prediction, and quickly output the final recognition result.

[0067] Compared with the traditional road vehicle license plate recognition method, adding an image defogging module can identify more blurred and complex scenes and realize end-to-end license plate recognition in haze weather. At the same time, the graph reasoning module is introduced to simplify the pixel similarity of the non-local mean filter into a more compact node similarity, which improves the computational efficiency. The spatial reasoning module and the channel reasoning module are used to establish and capture the interaction of different regions and different dimensions with similar structures, eliminate non-uniform haze, well preserve the texture and color of the image, and solve the problem of limited use of contextual information in license plate recognition in haze weather. It has broad application prospects.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing vehicle license plates on roads under haze weather conditions, characterized by: The following steps are involved: Step 1: Image dehazing processing; Step 1-1. Detect the area containing vehicle information in the processed image to obtain a vehicle image block; Step 1-2: Convolve the vehicle image block to obtain the feature code X; Step 1-3: Perform graph projection, graph reasoning, and graph reprojection on the feature code X in sequence; in the graph projection, the feature code X is projected onto the spatial graph and the channel graph to obtain the node features V of the spatial graph and the channel graph respectively. i ; Graph reasoning interacts with different node features to obtain features Graph reprojection reprojects the spatial graph nodes and channel graph nodes to the original pixel space, and obtains the reprojection features F corresponding to the spatial graph and channel graph respectively. i ; Step 1-4: Reproject the features F corresponding to the spatial map and channel map i Pooling, residual connection, standardization, activation function, multi-layer convolution stacking, global pooling and decoding are performed in sequence to obtain the defogged image. Step 2: Detect the image obtained in step 1 The license plate position in the image block of the license plate is obtained; Step 3: Recognize the license plate image block obtained in step 2 to obtain the license plate information.

2. The method for recognizing vehicle license plates on roads under haze weather conditions according to claim 1, characterized in that: The image processed in step 1 is obtained by frame processing through image acquisition by cameras on the road.

3. The method for recognizing vehicle license plates on roads in haze weather conditions according to claim 1, characterized in that: The specific process of obtaining the vehicle image block in step 1-1 is: aggregating the on-site prior knowledge of various traffic scenes, converting it into nonlinear inequality constraints, and constructing a multiple prior knowledge neural network model; inputting the processed image into the multiple prior knowledge neural network model to obtain the area where the vehicle reasonably appears, and obtaining the vehicle image block.

4. The method for recognizing vehicle license plates on roads under haze weather conditions according to claim 1, characterized in that: In steps 1-3, the feature code X is projected to the node features V of the spatial graph and channel graph i The expression is: According to the node feature V i Construct the connectivity matrix A i as follows: Among them, a i (p,q) is the connectivity matrix A i The element in the pth row and qth column represents the feature V from the pth node i (p) to the qth node feature V i (q); N represents the total number of graph nodes; θ(·) and θ'(·) represent two linear embeddings with different parameters.

5. The method for recognizing vehicle license plates on roads in haze weather conditions according to claim 4, characterized in that: In steps 1-3, the features obtained by the interaction of different node features The expression is as follows: Among them, σ(·) represents the activation function, W i Weights belonging to the graph convolution.

6. The method for recognizing vehicle license plates on roads in haze weather conditions according to claim 1, characterized in that: The specific process of step 2 is: Step 2-1. Image obtained in step 1 The corner points are used as feature points; the positions of feature points in the next frame of image are predicted by optical flow calculation based on Lucas-Konede constraint method; motion vectors are described and optical flow maps are generated; Use speed as the threshold to filter out valid feature points, eliminate noise feature points, obtain the vehicle's location information and shape, and determine the range of license plates. Step 2-2: Place the image Convert to GRAY color space; perform median filtering and edge detection on the image, and erode interference features; Perform Hover transform on the range interval where the license plate exists determined in step 2-1 to obtain line features, define the license plate range and crop the image to obtain the license plate image block.

7. The method for recognizing vehicle license plates on roads in haze weather conditions according to claim 1, characterized in that: The specific process of step three is: The license plate image block obtained in step 2 is input into the spatial transformer to obtain a flat license plate image; the flat license plate image is input into the license plate inference model to recognize the license plate information.

8. The method for recognizing vehicle license plates on roads in haze weather conditions according to claim 1, characterized in that: The training process of the inference model is as follows: use the backbone convolutional neural network to extract the convolutional features of the license plate, then use the cross-entropy loss of the attention classifier to train the spatial transformer, the backbone convolutional neural network and the attention branch, use the connection temporal classifier to train the connection temporal branch, and integrate the expression of multiple text features to obtain the trained license plate inference model; the license plate inference model is pruned and optimized; the final license plate inference model only retains the connection temporal classifier to recognize the flat license plate image.

9. A road vehicle license plate recognition system, characterized in that: The road vehicle license plate recognition method as described in claim 1; the road vehicle license plate recognition system includes an image defogging module, a license plate detection module and a license plate recognition module; the image defogging module includes a multiple prior knowledge neural network model, an encoder and a decoder; the encoder includes a convolution layer, a two-dimensional graph reasoning module, a pooling layer, a residual connection layer, a normalization layer, an activation function, a multi-layer convolution stacking layer and a global pooling layer; the license plate detection module is used to detect the position of the license plate in the image processed by the image defogging module, and select the image block containing the license plate number; the license plate recognition module is used to recognize the image block detected by the license plate detection module to obtain the license plate number text information.

Images