An arbitrary inclination angle license plate detection method based on a segmentation network

Abstract

This invention belongs to the field of image recognition technology. It discloses a method for detecting license plates at arbitrary tilt angles based on a segmentation network. The method involves training an image segmentation network model; inputting the license plate image to be predicted into the image segmentation network model to obtain the license plate outline and a binarized image of the license plate region; extracting the license plate outline point set from the binarized image of the license plate region; obtaining the minimum envelope rectangle corresponding to the license plate outline based on the license plate outline point set; obtaining the linear equations of the four sides of the minimum envelope rectangle; calculating the area of ​​the license plate outline; calculating the area of ​​the minimum envelope rectangle; obtaining the area ratio Q between the minimum envelope rectangle and the license plate outline; if Q equals 1, taking the four vertices of the rectangle as the four corner points of the license plate; performing an affine transformation using the four corner points of the license plate to obtain the corrected license plate image; effectively solving the detection and localization problem of tilted license plate images, improving detection accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to a method for detecting license plates at arbitrary tilt angles based on segmentation networks. Background Technology

[0002] Chinese Patent CN111914837A discloses a license plate detection method, apparatus, device, and storage medium based on YOLO object detection. The license plate detection method includes: Step 1, continuously capturing a preset number of video frames from a video stream to be detected; Step 2, detecting a first object target in the first frame image of the preset number of video frames using YOLO object detection, obtaining a first localization result corresponding to the first object target; Step 3, detecting a second object target in non-first frame images of the preset number of video frames according to the range determined by the first localization result, obtaining a second localization result corresponding to the second object target in the non-first frame images, until the preset number of video frames have been detected. This invention can improve the detection capability of small targets while achieving high-speed detection and reducing false positives.

[0003] Chinese Patent CN114267030A discloses an end-to-end license plate detection and key point detection method and apparatus. The method includes the following steps: acquiring license plate image data and preprocessing it; constructing a license plate detection network based on an improved YOLO algorithm; training the improved YOLO-based license plate detection network using the preprocessed license plate image data; and inputting the license plate image data to be identified into the trained license plate detection network for detection to obtain the license plate detection and key point detection results. The end-to-end license plate detection and key point detection method and apparatus provided by this invention, based on an improved YOLO license plate detection network, can simultaneously achieve multi-task detection of multiple license plates and corresponding key points, improving detection speed and saving computational resources.

[0004] Currently, the common methods for license plate detection in the industry include:

[0005] 1. Using deep learning object detection methods such as YOLO and SSD, the license plate is used as a rectangular target box for category learning and bounding box coordinate feature learning to find the position coordinates of the rectangular box where the license plate is located;

[0006] The target detection method detects standard horizontal and vertical rectangular targets. When the target is tilted, it appears as a non-rectangular target in the image. The detection results obtained by this rectangular target detection method will include a large proportion of the background area, especially in large-angle tilt fields. The localization deviation of this rectangular target detection is more serious.

[0007] 2. A keypoint detection method is used to directly regress the four corner points of the license plate, and then affine transformation is used to flatten the tilted license plate. This method relies on the accuracy of the detection of the four corner points. In scenarios where the corner points of the license plate are missing due to perspective, partial occlusion, dirt, etc., the corner point positions detected by the keypoint detection method will be inaccurate, and the license plate obtained by affine transformation will be severely deformed.

[0008] 3. Using traditional image processing methods, license plates are characterized based on traditional features such as contours, colors, and HOG feature points. This method has relatively simple feature selection, but it cannot handle complex scenarios in actual use and will have a high error rate.

[0009] In view of this, the present invention proposes a license plate detection method based on segmentation network with arbitrary tilt angle to solve the above problems. Summary of the Invention

[0010] To overcome the above-mentioned defects of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a license plate detection method based on a segmentation network with arbitrary tilt angle, comprising: S1. training an image segmentation network model; inputting the license plate image to be predicted into the trained image segmentation network model to obtain the license plate outline and the binary image of the license plate region;

[0011] S2. Extract the license plate contour point set from the binary image of the license plate region, obtain the minimum envelope rectangle corresponding to the license plate contour based on the license plate contour point set, and obtain the straight line equations of the four sides of the minimum envelope rectangle.

[0012] S3. Calculate the area of ​​the license plate outline, calculate the area of ​​the minimum enclosing rectangle according to the equation of the straight line, and calculate the ratio Q between the area of ​​the minimum enclosing rectangle and the area of ​​the license plate outline.

[0013] If Q equals 1, then the four vertices of the smallest envelope rectangle are taken as the four corner points of the license plate.

[0014] If Q is less than 1, then perform a contour correction operation; after the contour correction operation, the four corner points of the license plate are obtained;

[0015] S4. Perform an affine transformation on the four corner points of the license plate to obtain the corrected license plate image.

[0016] Further, step S1 includes:

[0017] The DBNet network was selected as the image segmentation network model.

[0018] Training the DBNet network includes:

[0019] S101. Collect an image dataset containing license plates at different tilt angles as training data;

[0020] S102. Use annotation tools to perform pixel-level annotations on each image in the image dataset, marking the license plate area as the foreground and the non-license plate area as the background;

[0021] S103. Merge the labeled image dataset and the original image dataset to divide them into a training set and a validation set. The training set and validation set contain the original images and the labeled images.

[0022] S104. Train the DBNet model using the training set, set the training hyperparameters, iteratively optimize the training hyperparameters during training, and use the validation set to verify the accuracy of the DBNet model.

[0023] Furthermore, the license plate image to be predicted is converted into the input format of the DBNet model and input into the DBNet model to obtain the foreground probability value of each pixel;

[0024] Set a threshold b, and use pixels with a foreground probability value greater than or equal to the threshold b as license plate pixels;

[0025] Pixels with a foreground probability value less than a threshold b are used as background pixels; license plate pixels form the license plate outline, and license plate pixels and background pixels form a binarized image of the license plate region.

[0026] Further, step S2 includes:

[0027] S201. Use OpenCV's approxPolyDP function to binarize the license plate region image to obtain the license plate outline point set;

[0028] S202. Input the license plate outline point set into the minAreaRect function of OpenCV to obtain the minimum envelope rectangle corresponding to the license plate outline;

[0029] S203. Use OpenCV's minAreaRect function to obtain the center coordinates, width, height, and rotation angle of the minimum envelope rectangle, and calculate the coordinates of the four vertices of the minimum envelope rectangle;

[0030] S204. Based on the coordinates of the four vertices of the minimum envelope rectangle, obtain the equations of the lines that form the four sides corresponding to the coordinates of any two adjacent vertices.

[0031] Further, step S3 includes:

[0032] Input the license plate outline point set into the OpenCV contourArea function to obtain the area of ​​the license plate outline;

[0033] Take the equations of two adjacent sides from the four lines of the minimum enclosing rectangle, namely side 1 and side 2; calculate the coordinates (x0, y0) of the intersection point of side 1 and side 2; obtain the length t1 of side 1 and the length t2 of side 2; obtain the distance d from the intersection point (x0, y0) of side 1 and side 2 to the vertex of the minimum enclosing rectangle; combine the intersection point coordinates (x0, y0), d, and the lengths t1 and t2 of the two sides to obtain the area S of the minimum enclosing rectangle;

[0034] Furthermore, step S3 also includes:

[0035] The contour correction operation is as follows:

[0036] S301. Calculate the distance from each contour point in the license plate contour point set to the four sides of the minimum envelope rectangle. According to the principle of closest distance, divide the contour points in the license plate contour point set into four point sets representing the four sides of the rectangle; calculate the distance from each contour point in the license plate contour point set to the four sides of the minimum envelope rectangle.

[0037] S302. Sort each contour point in ascending order of distance, and take the top n% of contour points as the contour point set P1 for a long side;

[0038] S303. Continue to sort the remaining contour points in ascending order of distance, and take the top n% of contour points as the contour point set P2 for another long side;

[0039] S304. Obtain the contour point sets P3 and P4 of the two short sides in sequence according to the method of S302-S303;

[0040] Furthermore, step S3 also includes:

[0041] S305. Obtain the center point C1 of contour point set P1 and the center point C2 of contour point set P2;

[0042] S306. Calculate the angle α1 of the line connecting each contour point in P1 to the center point C1 relative to the horizontal direction; calculate the angle α2 of the line connecting each contour point in P2 to the center point C2 relative to the horizontal direction.

[0043] S307. Sort the point set P1 in ascending or descending order according to angle α1, remove the first m% and the last m% of the contour points after sorting, and take the remaining contour points in P1 as the filtered contour point set F1.

[0044] Sort the point set P2 in ascending or descending order according to angle α2, remove the first m% and last m% of the contour points after sorting, and use the remaining contour points in P2 as the filtered contour point set F2.

[0045] S308. Fit the contour point sets F1 and F2 with straight lines to obtain two long side lines L1 and L2;

[0046] Set a threshold G; obtain the tilt angles of L1 and L2. If the difference between the tilt angles of L1 and L2 is greater than the threshold G, then take the longer side line Lk with more contour points in L1 and L2, where k is 1 or 2, as a standard longer side line.

[0047] S309. Obtain the straight line Lm of the contour points that are parallel to Lk and pass through the contour point set corresponding to the other long side line, and use it as the other long side line of the corrected standard.

[0048] Similarly, P3 and P4 are processed according to steps S305-S309 to obtain two standard short side lines;

[0049] Obtain the intersection of two standard long lines and two standard short lines to get the four corner points of the license plate.

[0050] Further, step S4 includes:

[0051] S401. Obtain the coordinates of the four corner points corresponding to the four corner points of the license plate; denoted as p1, p2, p3, p4, where p1 and p2 are not adjacent corner points, and p3 and p4 are not adjacent corner points;

[0052] S402. Construct the license plate correction transformation matrix, including:

[0053] Calculate the width w and height h based on the coordinates of the four corner points;

[0054] Construct the transformation matrix M = [1 / w, 0, 00, 1 / h, 0tx, ty, 1]; in the matrix, tx and ty are the translations along the x-axis and y-axis, respectively.

[0055] Furthermore, step S4 also includes:

[0056] Set the target image size d after transformation; input the license plate image B to be predicted, the transformation matrix M, and the target image size d into the OpenCV warpPerspective function to obtain the corrected license plate image.

[0057] A license plate detection system for arbitrary tilt angles based on a segmentation network is provided to implement the license plate detection method for arbitrary tilt angles based on a segmentation network as described in any one of claims 1 to 9. The system includes: a segmentation network module for training an image segmentation network model; and processing the license plate image to be predicted using the trained image segmentation network model to obtain a license plate outline and a binary image of the license plate region.

[0058] The position refinement module is used to extract the license plate contour point set from the binary image of the license plate region, obtain the minimum envelope rectangle corresponding to the license plate contour based on the license plate contour point set, and obtain the straight line equations of the four sides of the minimum envelope rectangle.

[0059] Calculate the area of ​​the license plate outline, calculate the area of ​​the minimum enclosing rectangle based on the equation of the line, and calculate the ratio Q between the area of ​​the minimum enclosing rectangle and the area of ​​the license plate outline.

[0060] If Q equals 1, then the four vertices of the smallest envelope rectangle are taken as the four corner points of the license plate.

[0061] If Q is less than 1, a contour correction operation is performed. After the contour correction operation, the four corner points of the license plate are obtained. The license plate correction module is used to perform affine transformation using the four corner points of the license plate to obtain the corrected license plate image.

[0062] The technical effects and advantages of this invention, a license plate detection method based on a segmentation network for arbitrary tilt angles, are as follows: It employs a deep learning-based image segmentation network for license plate region detection. Compared to traditional bounding box detection methods, the segmentation network can perform pixel-level prediction, enabling more accurate positioning of the license plate outline. After acquiring the license plate region image, this invention obtains the four standard corner points of the license plate through processes such as analyzing the minimum bounding rectangle of the license plate, contour point classification, and contour fitting. Finally, affine transformation is performed based on the four corner points to correct the tilted license plate. Compared to existing technologies, this invention can detect license plates at arbitrary tilt angles, resist the influence of partial occlusion, and return a complete license plate region. The prediction based on the segmentation network is more accurate and reliable, and the corrected license plate facilitates subsequent character recognition. Overall, this invention effectively solves the problem of detecting and locating tilted license plate images, improves detection accuracy and robustness, and has excellent technical effects. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a license plate detection method based on a segmentation network at arbitrary tilt angle according to the present invention.

[0064] Figure 2 This is a schematic diagram of an arbitrary tilt angle license plate detection system based on a segmentation network according to the present invention.

[0065] Figure 3 This is a schematic diagram of a license plate with an inclined angle in Embodiment 1 of the present invention;

[0066] Figure 4 This is a schematic diagram of the binarized image detected by the segmentation network in Embodiment 1 of the present invention;

[0067] Figure 5 This is a schematic diagram of the license plate outline and the minimum envelope rectangle in Embodiment 1 of the present invention;

[0068] Figure 6 This is a schematic diagram of the four-class classification of license plate outline point sets in Embodiment 1 of the present invention;

[0069] Figure 7 This is a schematic diagram of the point set fitting effect in Embodiment 1 of the present invention;

[0070] Figure 8 This is a schematic diagram of the electronic device of the present invention;

[0071] Figure 9 This is a schematic diagram of the storage medium of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] Example 1

[0074] Please see Figure 1 As shown in this embodiment, a license plate detection method based on a segmentation network at arbitrary tilt angles includes:

[0075] S1. Train the image segmentation network model; input the license plate image to be predicted into the trained image segmentation network model to obtain the license plate outline and the binary image of the license plate region;

[0076] S2. Extract the license plate contour point set from the binary image of the license plate region, obtain the minimum envelope rectangle corresponding to the license plate contour based on the license plate contour point set, and obtain the straight line equations of the four sides of the minimum envelope rectangle.

[0077] S3. Calculate the area of ​​the license plate outline, calculate the area of ​​the minimum enclosing rectangle according to the equation of the straight line, and calculate the ratio Q between the area of ​​the minimum enclosing rectangle and the area of ​​the license plate outline.

[0078] If Q equals 1, then the four vertices of the smallest envelope rectangle are taken as the four corner points of the license plate.

[0079] If Q is less than 1, then perform a contour correction operation; after the contour correction operation, the four corner points of the license plate are obtained;

[0080] S4. Perform an affine transformation on the four corner points of the license plate to obtain the corrected license plate image;

[0081] Further, step S1 specifically includes:

[0082] DBNet network was selected as the image segmentation network model for predicting image pixels.

[0083] Training the DBNet network includes:

[0084] S101. Collect an image dataset containing license plates at different tilt angles as training data, such as... Figure 3 As shown;

[0085] S102. Use annotation tools to perform pixel-level annotations on each image in the image dataset, marking the license plate area as the foreground and the non-license plate area as the background;

[0086] S103. Merge the labeled image dataset and the original image dataset to divide them into a training set and a validation set. The training set and validation set contain the original images and the labeled images.

[0087] S104. Train the DBNet model using the training set, set the training hyperparameters, iteratively optimize the training hyperparameters during training, and use the validation set to verify the accuracy of the DBNet model;

[0088] The license plate image to be predicted is converted into the input format of the DBNet model and input into the DBNet model to obtain the foreground probability value of each pixel;

[0089] Set a threshold b, and use pixels with a foreground probability value greater than or equal to the threshold b as license plate pixels;

[0090] Pixels with foreground probability values ​​less than threshold b are designated as background pixels; license plate pixels form the license plate outline, and the license plate pixels and background pixels together form a binarized image of the license plate region, such as... Figure 4 As shown;

[0091] It should be noted that the threshold b can be set in the following ways:

[0092] On the training set, the foreground probability values ​​of all license plate pixels are counted to obtain the license plate pixel probability distribution a1; the foreground probability values ​​of all background pixels are counted to obtain the background pixel probability distribution a2; the average probability values ​​u1 and u2 of a1 and a2 are calculated respectively; the threshold b is initially taken as the mean of the two average probabilities: (u1+u2) / 2; different thresholds b are tested on the validation set, and the threshold with the highest segmentation accuracy is selected; finally, the threshold b used on the test data is determined.

[0093] Pixel-level segmentation can locate license plates more precisely and is more accurate than traditional bounding box detection, especially for tilted license plates. Binarization segmentation based on foreground probability output can effectively resist some license plate occlusion and influence, improving detection robustness. Compared with directly regressing the four coordinates of the license plate, predicting the license plate region is more in line with the original design intention of image segmentation networks and can obtain more reliable prediction results. The segmentation network predicts the semantics of the image and can detect specific targets without being affected by deformation and appearance changes.

[0094] Further, step S2 specifically includes:

[0095] S201. Use OpenCV's approxPolyDP function to binarize the license plate region image to obtain the license plate outline point set;

[0096] S202. Input the license plate outline point set into the OpenCV minAreaRect() function to obtain the minimum envelope rectangle corresponding to the license plate outline, such as... Figure 5 As shown;

[0097] S203. Use OpenCV's minAreaRect() function to obtain the center coordinates, width, height, and rotation angle of the minimum envelope rectangle, and calculate the coordinates of the four vertices of the minimum envelope rectangle;

[0098] Specifically, S203,

[0099] The center coordinates c, width w, height h, and rotation angle a of the minimum envelope rectangle can be directly obtained using the minAreaRect() function in OpenCV.

[0100] Based on the center coordinates, the width and height of the rectangle, and the rotation angle, the coordinates of the four vertices can be calculated: Vertex 1 coordinates = c + mat × [-w / 2, -h / 2]; Vertex 2 coordinates = c + mat × [w / 2, -h / 2]; Vertex 3 coordinates = c + mat × [w / 2, h / 2]; Vertex 4 coordinates = c + mat × [-w / 2, h / 2]; where mat is the rotation transformation matrix constructed based on the rotation angle;

[0101] S204. Based on the coordinates of the four vertices of the minimum envelope rectangle, obtain the equations of the four sides corresponding to the lines connecting the coordinates of any two adjacent vertices.

[0102] Specifically, S204,

[0103] By connecting two adjacent vertices, we can find the equations of the four edges passing through those vertices.

[0104] Equation of edge line 1: Passing through vertex 1 and vertex 2;

[0105] Equation for edge line 2: Passing through vertex 2 and vertex 3;

[0106] Equation for edge line 3: Passing through vertex 3 and vertex 4;

[0107] Equation for edge line 4: Passing through vertex 4 and vertex 1;

[0108] Both can be obtained through two-point equations in mathematics;

[0109] Further, step S3 specifically includes:

[0110] Input the license plate outline point set into the OpenCV contourArea() function to get the area of ​​the license plate outline;

[0111] Take the equations of two adjacent sides from the four lines of the minimum enclosing rectangle, namely side 1 and side 2; calculate the coordinates (x0, y0) of the intersection point of side 1 and side 2; obtain the length t1 of side 1 and the length t2 of side 2; obtain the distance d from the intersection point (x0, y0) of side 1 and side 2 to the vertex of the minimum enclosing rectangle; combine the intersection point coordinates (x0, y0), d, and the lengths t1 and t2 of the two sides to obtain the area S of the minimum enclosing rectangle;

[0112] Contour correction operations include:

[0113] S301. Calculate the distance from each contour point in the license plate contour point set to the four sides of the smallest envelope rectangle. Based on the principle of shortest distance, divide the contour points in the license plate contour point set into four point sets representing the four sides of the rectangle, such as... Figure 6 As shown; calculate the distance from each contour point within the license plate contour point set to the four sides of the minimum envelope rectangle;

[0114] S302. Sort each contour point in ascending order of distance, and take the top n% of contour points as the contour point set P1 for a long side;

[0115] S303. Continue to sort the remaining contour points in ascending order of distance, and take the top n% of contour points as the contour point set P2 for another long side;

[0116] S304. Obtain the contour point sets P3 and P4 of the two short sides in sequence according to the method of S302-S303;

[0117] Since the minimum envelope rectangle may not perfectly fit the license plate edge, the four point sets obtained from S301-S304 may contain misclassified contour points. Therefore, the following filtering and sorting are performed:

[0118] S305. Obtain the center point C1 of contour point set P1 and the center point C2 of contour point set P2;

[0119] S306. Calculate the angle α1 of the line connecting each contour point in P1 to the center point C1 relative to the horizontal direction; calculate the angle α2 of the line connecting each contour point in P2 to the center point C2 relative to the horizontal direction.

[0120] S307. Sort the point set P1 in ascending or descending order according to angle α1, remove the first m% and the last m% of the contour points after sorting, and take the remaining contour points in P1 as the filtered contour point set F1.

[0121] Sort the point set P2 in ascending or descending order according to angle α2, remove the first m% and last m% of the contour points after sorting, and use the remaining contour points in P2 as the filtered contour point set F2.

[0122] S308. Fit the contour point sets F1 and F2 with straight lines to obtain two long side lines L1 and L2;

[0123] Set a threshold G; obtain the tilt angles of L1 and L2. If the difference between the tilt angles of L1 and L2 is greater than the threshold G, then take the long side line Lk with more contour points, where k is 1 or 2, as a standard long side line.

[0124] S309. Obtain the straight line Lm of the contour points that are parallel to Lk and pass through the contour point set corresponding to the other long side line, and use it as the other long side line of the corrected standard.

[0125] Similarly, P3 and P4 are processed according to steps S305-S309 to obtain two standard short side lines; as follows Figure 7 As shown;

[0126] Obtain the intersection of two standard long lines and two standard short lines to get the four corner points of the license plate;

[0127] It should be noted that the threshold G is set as follows:

[0128] In the experimental environment, n sets of test license plates were obtained to form a training set and a validation set. Through the above process, the tilt angle difference between the long edges of all license plates was statistically analyzed to obtain the distribution pattern of the angle difference. Multiple thresholds Gi, i = 1, 2, ..., n, were set, and long edge correction was performed m times. For each Gi, the degree of fit between the corrected long edge and the license plate outline, positioning accuracy, and other indicators were calculated. The threshold Gj that achieved the best correction effect was selected as the candidate threshold. The correction effect of threshold Gj was tested on the validation set. The threshold Gj was fine-tuned until the long edge correction effect on the validation set was optimal. Finally, G = Gj was determined as the threshold for long edge correction of the test data.

[0129] Calculating the area of ​​the minimum envelope rectangle determines whether the contour needs further correction, avoiding unnecessary calculations; dividing the contour points into four point sets, representing the four sides of the contour, improves the accuracy of subsequent line fitting; filtering and sorting the four point sets removes misclassified contour points, improving robustness; fitting two long sides and two short sides separately, and setting angle threshold constraints, can correct non-parallel lines; using the intersection of the two long sides and two short sides as contour corner points, a standard rectangular contour can be obtained.

[0130] Further, step S4 specifically includes:

[0131] S401. Obtain the coordinates of the four corner points corresponding to the four corner points of the license plate; denoted as p1, p2, p3, p4, where p1 and p2 are not adjacent corner points, and p3 and p4 are not adjacent corner points;

[0132] S402. Construct the license plate correction transformation matrix, including:

[0133] Calculate the width w and height h based on the coordinates of the four corner points; construct the transformation matrix M = [1 / w, 0, 00, 1 / h, 0tx, ty, 1];

[0134] tx and ty represent the translation along the x-axis and y-axis, respectively, with the aim of transforming the corner point p1 to the origin (0,0) of the image.

[0135] Set the size d of the target image after transformation; input the license plate image B to be predicted, the transformation matrix M, and the size d of the target image into the OpenCV warpPerspective() function to obtain the corrected license plate image;

[0136] It should be noted that correcting arbitrarily tilted license plates to a normal viewing angle simplifies the design of subsequent recognition algorithms; using mature OpenCV functions results in simple implementation and good performance; translating a corner point to the origin can remove the translation of the license plate area in the image; affine transformation can preserve the shape of the text and numbers on the license plate, facilitating subsequent recognition; compared to directly cropping the bounding box, affine transformation can return the complete license plate area.

[0137] This embodiment employs a deep learning-based image segmentation network for license plate region detection. Compared to traditional bounding box detection methods, the segmentation network can perform pixel-level prediction, enabling more accurate positioning of the license plate outline. After acquiring the license plate region image, this embodiment obtains the four standard corner points of the license plate through processes such as analyzing the minimum bounding rectangle of the license plate, contour point classification, and contour fitting. Finally, affine transformation is performed based on the four corner points to correct the tilted license plate. Compared to existing technologies, this embodiment can detect license plates at any tilt angle, resist the influence of partial occlusion, and return a complete license plate region. The prediction based on the segmentation network is more accurate and reliable, and the corrected license plate facilitates subsequent character recognition. Overall, this embodiment effectively solves the problem of detecting and locating tilted license plate images, improves detection accuracy and robustness, and has good technical effects.

[0138] Example 2

[0139] Please see Figure 2As shown in the figure, the parts not described in detail in this embodiment are described in Embodiment 1. A license plate detection system based on a segmentation network with arbitrary tilt angle is provided, including: a segmentation network module for training an image segmentation network model; and using the trained image segmentation network model to process the license plate image to be predicted to obtain the license plate outline and a binary image of the license plate region.

[0140] The position refinement module is used to extract the license plate contour point set from the binary image of the license plate region, obtain the minimum envelope rectangle corresponding to the license plate contour based on the license plate contour point set, and obtain the straight line equations of the four sides of the minimum envelope rectangle.

[0141] Calculate the area of ​​the license plate outline, calculate the area of ​​the minimum enclosing rectangle based on the equation of the line, and calculate the ratio Q between the area of ​​the minimum enclosing rectangle and the area of ​​the license plate outline.

[0142] If Q equals 1, then the four vertices of the smallest envelope rectangle are taken as the four corner points of the license plate.

[0143] If Q is less than 1, a contour correction operation is performed; after the contour correction operation, the four corner points of the license plate are obtained. The license plate correction module is used to perform affine transformation using the four corner points of the license plate to obtain the corrected license plate image; the modules are connected to each other via wired and / or wireless means to realize data transmission between modules.

[0144] Example 3

[0145] Please see Figure 8 As shown, according to another aspect of this application, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, can perform a license plate detection method based on a segmentation network at arbitrary tilt angles as described above.

[0146] The method or system according to the embodiments of this application can also be used by means of Figure 8 The architecture of the electronic device shown is used to implement this. For example... Figure 8 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store the license plate detection method based on a segmented network at arbitrary tilt angles provided in this application. Furthermore, the electronic device 500 may also include a user interface 508. Of course, Figure 3 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 3One or more components in the illustrated electronic device.

[0147] Example 4

[0148] Please see Figure 9 The diagram illustrates a computer-readable storage medium 600 according to one embodiment of this application. The computer-readable storage medium 600 stores computer-readable instructions. When executed by a processor, the computer-readable instructions can perform a license plate detection method based on a segmentation network according to an embodiment of this application, as described above with reference to the accompanying drawings. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0149] Furthermore, according to embodiments of this application, the processes described in the above-referenced flowcharts can be implemented as computer software programs. For example, this application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be executed by a processor to perform instructions corresponding to the method steps provided in this application, namely, a license plate detection method based on a segmentation network at arbitrary tilt angles. When this computer program is executed by a central processing unit (CPU), it performs the functions defined in the method of this application.

[0150] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0151] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0152] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0156] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0157] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting license plates at arbitrary tilt angles based on segmentation networks, characterized in that, include: S1. Train the image segmentation network model; The image of the license plate to be predicted is input into a trained image segmentation network model to obtain the license plate outline and a binarized image of the license plate region. S2. Extract the license plate contour point set from the binary image of the license plate region, obtain the minimum envelope rectangle corresponding to the license plate contour based on the license plate contour point set, and obtain the straight line equations of the four sides of the minimum envelope rectangle. S3. Calculate the area of ​​the license plate outline, calculate the area of ​​the minimum enclosing rectangle according to the equation of the straight line, and calculate the ratio Q between the area of ​​the minimum enclosing rectangle and the area of ​​the license plate outline. If Q equals 1, then the four vertices of the smallest envelope rectangle are taken as the four corner points of the license plate. If Q is less than 1, then perform a contour correction operation; after the contour correction operation, the four corner points of the license plate are obtained; The contour correction operation includes: S301. Calculate the distance from each contour point in the license plate outline point set to the four sides of the minimum envelope rectangle. Based on the principle of closest distance, divide the outline points in the license plate outline point set into four point sets representing the four sides of the rectangle; calculate the distance from each contour point in the license plate outline point set to the four sides of the minimum envelope rectangle. S302. Sort each contour point in ascending order of distance, and take the top n% of contour points as the contour point set P1 for one long side; S303. Continue sorting the remaining contour points in ascending order of distance, and take the top n% of contour points as the contour point set P2 for the other long side; S304. Obtain the contour point sets P3 and P4 of the two short sides in sequence according to the method of S302-S303; S305. Obtain the center point C1 of contour point set P1 and the center point C2 of contour point set P2; S306. Calculate the angle α1 of the line connecting each contour point in P1 to the center point C1 relative to the horizontal direction; calculate the angle α2 of the line connecting each contour point in P2 to the center point C2 relative to the horizontal direction. S307. Sort the point set P1 in ascending or descending order according to the angle α1, remove the first m% and the last m% of the contour points after sorting, and take the remaining contour points in P1 as the filtered contour point set F1. Sort the point set P2 in ascending or descending order according to angle α2, remove the first m% and last m% of the contour points after sorting, and use the remaining contour points in P2 as the filtered contour point set F2. S308. Fit the contour point sets F1 and F2 with straight lines to obtain two long side lines L1 and L2; Set a threshold G; obtain the tilt angles of L1 and L2. If the difference between the tilt angles of L1 and L2 is greater than the threshold G, then take the longer side line Lk with more contour points in L1 and L2, where k is 1 or 2, as a standard longer side line. S309. Obtain the straight line Lm of the contour points that are parallel to Lk and pass through the contour point set corresponding to the other long side line, and use it as the other long side line of the corrected standard. Similarly, P3 and P4 are processed according to steps S305-S309 to obtain two standard short side lines; Obtain the intersection of two standard long lines and two standard short lines to get the four corner points of the license plate; S4. Perform an affine transformation on the four corner points of the license plate to obtain the corrected license plate image.

2. The method for detecting license plates at arbitrary tilt angles based on segmentation networks according to claim 1, characterized in that, Step S1 includes: The DBNet network was selected as the image segmentation network model. Training the DBNet network includes: S101. Collect an image dataset containing license plates at different tilt angles as training data; S102. Use annotation tools to perform pixel-level annotations on each image in the image dataset, marking the license plate area as the foreground and the non-license plate area as the background; S103. Merge the labeled image dataset and the original image dataset to divide them into a training set and a validation set. The training set and validation set contain the original images and the labeled images. S104. Train the DBNet model using the training set, set the training hyperparameters, iteratively optimize the training hyperparameters during training, and use the validation set to verify the accuracy of the DBNet model.

3. The method for detecting license plates at arbitrary tilt angles based on segmentation networks according to claim 2, characterized in that, The license plate image to be predicted is converted into the input format of the DBNet model and then input into the DBNet model to obtain the foreground probability value of each pixel; Set a threshold b, and use pixels with a foreground probability value greater than or equal to the threshold b as license plate pixels; Pixels with a foreground probability value less than the threshold b are designated as background pixels; The license plate pixels form the outline of the license plate, and the license plate pixels and background pixels form a binarized image of the license plate region.

4. The method for detecting license plates at arbitrary tilt angles based on segmentation networks according to claim 3, characterized in that, Step S2 includes: S201. Use OpenCV's approxPolyDP function to binarize the license plate region image to obtain the license plate outline point set; S202. Input the license plate outline point set into the minAreaRect function of OpenCV to obtain the minimum envelope rectangle corresponding to the license plate outline; S203. Use OpenCV's minAreaRect function to obtain the center coordinates, width, height, and rotation angle of the minimum envelope rectangle, and calculate the coordinates of the four vertices of the minimum envelope rectangle; S204. Based on the coordinates of the four vertices of the minimum envelope rectangle, obtain the equations of the lines that form the four sides corresponding to the coordinates of any two adjacent vertices.

5. The method for detecting license plates at arbitrary tilt angles based on segmentation networks according to claim 4, characterized in that, Step S3 includes: Input the license plate outline point set into the OpenCV contourArea function to obtain the area of ​​the license plate outline; Take the equations of two adjacent sides from the four straight lines of the minimum enclosing rectangle, namely side 1 and side 2; calculate the coordinates (x0, y0) of the intersection point of side 1 and side 2; obtain the length t1 of side 1 and the length t2 of side 2; obtain the distance d from the intersection point (x0, y0) of side 1 and side 2 to the vertex of the minimum enclosing rectangle; combine the intersection point coordinates (x0, y0), d, and the lengths t1 and t2 of the two sides to obtain the area of ​​the minimum enclosing rectangle.

6. A license plate detection method based on a segmentation network at arbitrary tilt angles as described in claim 1. The method is characterized by, Step S4 includes: S401. Obtain the coordinates of the four corner points corresponding to the four corner points of the license plate; denoted as p1, p2, p3, p4, where p1 and p2 are not adjacent corner points, and p3 and p4 are not adjacent corner points; S402. Construct the license plate correction transformation matrix, including: Calculate the width w and height h based on the coordinates of the four corner points; Construct the transformation matrix M = [1 / w, 0, 0, 0, 1 / h, 0, tx, ty, 1]; in the matrix, tx and ty are the translations along the x-axis and y-axis, respectively.

7. A license plate detection method based on a segmentation network at arbitrary tilt angles as described in claim 6. The method is characterized by, Step S4 also includes: Set the target image size d after transformation; input the license plate image B to be predicted, the transformation matrix M, and the target image size d into the OpenCV warpPerspective function to obtain the corrected license plate image.

8. A license plate detection system based on a segmentation network at arbitrary tilt angles, used to implement the license plate detection method based on a segmentation network at arbitrary tilt angles as described in any one of claims 1 to 7, characterized in that, include: The segmentation network module is used to train the image segmentation network model; the trained image segmentation network model is used to process the license plate image to be predicted to obtain the license plate outline and the binary image of the license plate region; The position refinement module is used to extract the license plate contour point set from the binary image of the license plate region, obtain the minimum envelope rectangle corresponding to the license plate contour based on the license plate contour point set, and obtain the straight line equations of the four sides of the minimum envelope rectangle. Calculate the area of ​​the license plate outline, calculate the area of ​​the minimum enclosing rectangle based on the equation of the line, and calculate the ratio Q between the area of ​​the minimum enclosing rectangle and the area of ​​the license plate outline. If Q equals 1, then the four vertices of the smallest envelope rectangle are taken as the four corner points of the license plate. If Q is less than 1, a contour correction operation is performed. After the contour correction operation, the four corner points of the license plate are obtained. The license plate correction module is used to perform affine transformation using the four corner points of the license plate to obtain the corrected license plate image.

Images