License plate detection and correction method based on semantic segmentation

Abstract

The present invention provides a license plate detection and correction method based on semantic segmentation, comprising the following steps: step 1, performing license plate semantic segmentation processing on a vehicle image based on a preset Unet semantic segmentation network to obtain a license plate mask map; the preset Unet semantic segmentation network comprises three mask channels, each mask channel corresponds to a license plate color; each mask channel comprises two convolution layers with 3*3 convolution kernels, four downsampling modules, four upsampling modules and a convolution layer with a 3*3 convolution kernel, the downsampling module comprises an upsampling layer and two convolution layers, and the upsampling module comprises a pooling layer and two convolution layers; step 2, locating four vertices of the license plate in the license plate mask map by using a license plate positioning algorithm; step 3, aligning the license plate area in the license plate mask map by using perspective transformation according to the four vertices of the license plate, and completing the license plate correction.

Description

Technical Field

[0001] The present invention relates to a vehicle license plate detection method, and in particular to a vehicle license plate detection and correction method based on semantic segmentation. Background Art

[0002] As an important part of the intelligent transportation system, license plate recognition technology is widely used in highway toll booths, parking lots, checkpoints and other occasions, bringing great convenience to people's lives. Such application scenarios are relatively simple, the location of the license plate is fixed, the license plate accounts for a large proportion of the image and the image is clear, and it is not easily disturbed by external environments such as light. When the light is insufficient, it can be illuminated by fill light equipment, and license plate recognition has achieved satisfactory results. However, with the development of intelligent technology, license plate recognition has gradually transitioned from simple application scenarios to complex traffic scenarios, such as real-time road monitoring, vehicle tracking, violations, etc.

[0003] License plate detection, as a key part of license plate recognition, plays a decisive role in license plate recognition. Most of the existing license plate positioning methods locate the license plate area from the perspective of target detection. Patent CN202011210086.2 discloses a license plate recognition method based on a neural network, and CN202110767061.0 discloses a tilted license plate correction method based on deep learning. Both patents use the Yolo model for license plate detection. When locating the license plate area, this method will locate some redundant backgrounds in the license plate area, resulting in a rectangular license plate area, which is inaccurately positioned.

[0004] Moreover, in the face of complex application scenarios, existing license plate detection methods face the following problems: first, the different distances between the target and the camera cause large changes in the proportion of the license plate in the image; the shooting angle is not fixed, and changes in the camera angle will cause the shape of the license plate to tilt and deform irregularly due to the viewing angle, so it is necessary to consider correcting the detected license plate before license plate recognition; the diversity of license plate types requires the simultaneous detection of different types of license plates.

[0005] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and thus provide a license plate detection and correction method based on semantic segmentation.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a license plate detection and correction method based on semantic segmentation, comprising the following steps:

[0008] Step 1: Perform license plate semantic segmentation on the vehicle image based on the preset Unet semantic segmentation network to obtain a license plate mask image;

[0009] The preset Unet semantic segmentation network includes three mask channels, each mask channel corresponds to a license plate color; each mask channel includes two convolutional layers with 3*3 convolutional kernels, four downsampling modules, four upsampling modules and one convolutional layer with 3*3 convolutional kernels, the downsampling module includes one upsampling layer and two convolutional layers, and the upsampling module includes one pooling layer and two convolutional layers;

[0010] Step 2, using the license plate positioning algorithm to locate the four vertices of the license plate in the license plate mask image;

[0011] Step 3: Use perspective transformation to align the license plate area in the license plate mask image according to the four vertices of the license plate to complete the license plate correction.

[0012] The present invention has outstanding substantive features and significant progress compared to the prior art. Specifically,

[0013] (1) Completely different from the method of locating the license plate position from the perspective of target detection, the present invention starts from the idea of ​​semantic segmentation, segments the specific position of the license plate in the image, and realizes the detection of the license plate. The segmented license plate position no longer contains redundant background information, and the license plate is accurately located. At the same time, this method is also applicable to different proportions of license plates in the image, detects license plates of different sizes, and supports the detection of different types of license plates;

[0014] (2) From the semantically segmented license plate mask image, the four vertices of the license plate are directly located and the license plate is corrected using the perspective transformation method. This method is simple, effective, and has good results. It can accurately locate the license plate, does not contain redundant background information, and is applicable to license plate tilt and deformation. It solves the problem of license plate tilt and irregular deformation caused by camera angle problems.

[0015] (3) The proposed semantic segmentation network includes two convolutional layers with 3*3 convolutional kernels, four downsampling modules, four upsampling modules and one convolutional layer with 3*3 convolutional kernels. The downsampling module includes one upsampling layer and two convolutional layers, and the upsampling module includes one pooling layer and two convolutional layers, which greatly reduces the number of network layers and convolutional kernels. Through continuous tuning, the detection reasoning speed is improved without a significant decrease in model performance, making the network real-time and efficient in license plate detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process of the present invention.

[0017] Figure 2It is a schematic diagram of the Unet semantic segmentation network of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0019] like Figure 1 As shown, the present invention provides a license plate detection and correction method based on semantic segmentation, comprising the following steps:

[0020] Step 1: Perform license plate semantic segmentation on the vehicle image based on the preset Unet semantic segmentation network to obtain a license plate mask image;

[0021] Step 2, using the license plate positioning algorithm to locate the four vertices of the license plate in the license plate mask image;

[0022] Step 201, performing edge detection on the license plate area of ​​the license plate mask image to obtain the edge contour and the minimum circumscribed rectangle of the license plate area;

[0023] Specifically, sort the four vertices of the minimum circumscribed rectangle in the order of upper left, lower left, upper right, and lower right, and obtain: l0(x0, y0), l1(x1, y1), l2(x2, y2), l3(x3, y3);

[0024] Step 202, calculate the distances dis0, dis1, dis2 and dis3 from any point (x, y) on the edge contour to the four vertices of the minimum circumscribed rectangle:

[0025]

[0026] Calculate the distance from the point (x, y) to the upper and lower lines of the minimum circumscribed rectangle. The upper line distance is denoted as d_up, and the lower line distance is d_down.

[0027] Step 203, calculating the rotation angle of the minimum circumscribed rectangle, and adding different weights to the distances obtained in step 202 according to the rotation angle, and calculating the distances and d0, d1, d2, d3;

[0028]

[0029] Among them, Angle is the rotation angle, w1 and w2 are weights;

[0030] Step 204, repeat steps 202-203, traverse all points on the edge contour, and obtain the distance and d0, d1, d2, d3 of each point;

[0031] Step 205, compare the distances of all points with d0 to select the minimum value d0, compare the distances of all points with d1 to select the minimum value d1, compare the distances of all points with d2 to select the minimum value d2, compare the distances of all points with d3 to select the minimum value d3, and the four points corresponding to the minimum values ​​d0, d1, d2, and d3 are the four vertices of the license plate;

[0032] Step 3: Use perspective transformation to align the license plate area in the license plate mask image according to the four vertices of the license plate to complete the license plate correction.

[0033] The present invention directly locates the four vertices of the license plate from the semantically segmented license plate mask image and uses the perspective transformation method to correct the license plate. The method is simple, effective, and has good effects. It has accurate positioning, does not contain redundant background information, and is applicable to license plate tilt, license plate deformation, etc., solving the problem of license plate tilt and irregular deformation caused by camera angle problems.

[0034] Completely different from the method of locating the license plate position from the perspective of target detection, the present invention starts from the idea of ​​semantic segmentation, segments the specific position of the license plate in the image, realizes license plate detection, and the segmented license plate position no longer contains redundant background information, and the license plate is accurately located. At the same time, this method is also applicable to different proportions of license plates in the image, detects license plates of different sizes, and supports detection of different types of license plates.

[0035] In a specific implementation, the preset Unet semantic segmentation network includes three mask channels, each mask channel corresponds to a license plate color; each mask channel includes two convolution layers with 3*3 convolution kernels, four downsampling modules, four upsampling modules and one convolution layer with 3*3 convolution kernels, the downsampling module includes one upsampling layer and two convolution layers, and the upsampling module includes one pooling layer and two convolution layers;

[0036] After the 512*512*3 vehicle image is input into the network, it is first convolved through two convolution layers Conv with a convolution kernel size of 3*3, and the output is 512*512*64; then it is processed by the downsampling module 4 times to extract the image features; then the upsampling module is repeated 4 times to restore the image feature map size. At the same time, due to the multi-layer feature fusion operation adopted in the upsampling module, the accuracy of semantic segmentation is improved; finally, after a layer of convolution operation, a segmentation result map of 512*512*(N+1) size is obtained, where 1 represents the background and N represents the number of license plates of different colors.

[0037] The above network greatly reduces the number of network layers and convolution kernels. Through continuous tuning, the detection reasoning speed is improved without a significant decrease in model performance, making the network real-time and efficient in license plate detection tasks.

[0038] In specific implementation, different network parameters can be set for the network structure according to different requirements, mainly setting the number of input channels and the number of output channels in the down-sampling module and the up-sampling module.

[0039] This embodiment provides two parameter settings:

[0040] Parameter setting 1

[0041] Layer In_channels Out_channels Down1 64 128 Down2 128 256 Down3 256 512 Down3 512 1024 Up1 1024 512 Up2 512 256 Up3 256 128 Up4 128 64

[0042] Parameter setting 2:

[0043] Layer In_channels Out_channels Down1 8 16 Down2 16 32 Down3 32 64 Down3 64 128 Up1 128 64 Up2 64 32 Up3 32 16 Up4 16 8

[0044] In the first parameter setting, the number of output channels is larger, the output feature dimension is high, and the extracted features are rich, but at the same time, the computational complexity of the Unet semantic segmentation network will be large, the inference speed will be slow, and the model performance will be better.

[0045] In the second parameter setting, the number of output channels is smaller, the output feature dimension is low, and the richness of the extracted features is reduced. However, at the same time, the amount of computation of the Unet semantic segmentation network will be smaller, the inference speed will be faster, and the model performance will decline to a certain extent.

[0046] Specifically, since the background of the license plate mask image obtained after semantic segmentation is black and the license plate area is a bright area, in order to further distinguish license plates of different color types in the license plate mask image, when the vehicle image is subjected to license plate semantic segmentation processing based on the preset license plate segmentation network, the grayscale values ​​of the license plate area in the license plate mask image obtained by different mask channels are different.

[0047] For example, in actual data processing, the three mask channels are the blue license plate channel, the yellow license plate channel and the green license plate channel; the grayscale value of the license plate area in the license plate mask image obtained by the blue license plate channel is set to 255; the grayscale value of the license plate area in the license plate mask image obtained by the yellow license plate channel is set to 160; the grayscale value of the license plate area in the license plate mask image obtained by the green license plate channel is set to 80; and it can also be expanded as needed to support the detection of special license plates, such as white license plates (military license plates, police license plates), black license plates, etc.

[0048] At the same time, in order to improve the applicability of the Unet semantic segmentation network to different proportions of license plates in pictures, a large number of samples with different proportions of license plates are needed to train the Unet semantic segmentation network.

[0049] And when training the Unet semantic segmentation network, the cross entropy loss function is used to calculate the loss.

[0050] Among them, assuming there are K categories, the cross entropy loss function is:

[0051]

[0052] Among them, p(y k ) represents the predicted probability of the kth category, q(y k ) represents the true probability of the kth ID category.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A license plate detection and correction method based on semantic segmentation, characterized in that: The following steps are involved: Step 1: Perform license plate semantic segmentation on the vehicle image based on the preset Unet semantic segmentation network to obtain a license plate mask image; The preset Unet semantic segmentation network includes three mask channels, each mask channel corresponds to a license plate color; each mask channel includes two convolutional layers with 3*3 convolutional kernels, four downsampling modules, four upsampling modules and one convolutional layer with 3*3 convolutional kernels, the downsampling module includes one upsampling layer and two convolutional layers, and the upsampling module includes one pooling layer and two convolutional layers; Step 2, using the license plate positioning algorithm to locate the four vertices of the license plate in the license plate mask image; Step 2 is as follows: Step 201, performing edge detection on the license plate area of ​​the license plate mask image to obtain the edge contour and the minimum circumscribed rectangle of the license plate area; Step 202, calculating the distances from any point on the edge contour to the four vertices of the minimum circumscribed rectangle and to the upper and lower straight lines of the minimum circumscribed matrix; Step 203, calculate the rotation angle of the minimum circumscribed rectangle, and add different weights to the distance obtained in step 202 according to the rotation angle, and calculate the distance and ; Among them, Angle is the rotation angle, and is the weight; is the distance from the point (x, y) to the straight line on the minimum circumscribed rectangle, is the distance from the point (x, y) to the straight line under the minimum circumscribed rectangle; , , as well as They are the distances from the point (x, y) to the four vertices of the minimum circumscribed rectangle; Step 204, repeat steps 202-203, traverse all points on the edge contour, and obtain the distance and ; Step 205, compare the distances of all points and Select the minimum value , compare the distances of all points and Select the minimum value , compare the distances of all points and Select the minimum value , compare the distances of all points and Select the minimum value , minimum The corresponding four points are the four vertices of the license plate; Step 3: Use perspective transformation to align the license plate area in the license plate mask image according to the four vertices of the license plate to complete the license plate correction.

2. The method for license plate detection and correction based on semantic segmentation according to claim 1, characterized in that: In step 1, when the vehicle image is subjected to license plate semantic segmentation processing based on the preset license plate segmentation network, the grayscale values ​​of the license plate area in the license plate mask image obtained by different mask channels are different.

3. The method for license plate detection and correction based on semantic segmentation according to claim 2, characterized in that: When training the Unet semantic segmentation network, the cross entropy loss function is used to calculate the loss.

Images