A license plate recognition method and device combining license plate contour NMS refinement algorithm

Abstract

The present application discloses a license plate recognition method and device combined with a license plate contour NMS refinement algorithm. The method includes: license plate detection; license plate refinement based on the NMS refinement algorithm; license plate correction; license plate recognition. The device includes: a license plate detection module, a license plate refinement module, a license plate correction module and a license plate recognition module. The present application not only retains the effective contour, but also reduces the overlap of the contour, so that the license plate number boundary fitted subsequently is more in line with the actual situation, and the accuracy of recognizing the license plate number is improved. The block NMS algorithm is further adopted to divide the license plate image into blocks, and the divided images are NMS-ed separately, and finally an image is synthesized again, which improves the calculation efficiency and saves calculation resources.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a license plate recognition method and device combined with a license plate contour NMS refinement algorithm. Background Art

[0002] At present, mature license plate recognition solutions generally require four steps: (1) License plate detection; (2) License plate refinement; (3) License plate correction; (4) License plate recognition. Among them, license plate refinement is a key step. It requires refining the license plate rectangular frame detected in the previous license plate detection step, that is, determining the true boundary of the license plate number to eliminate redundant information. The result of refinement will directly affect the effect of subsequent license plate recognition. The existing mainstream refinement method first binarizes the image, then performs contour detection, and finally fits the boundary of the license plate number part based on the coordinates of each scattered contour vertex.

[0003] This method works well when the license plate number is clear and the brightness difference between different parts of the license plate image is relatively small. However, when the license plate number is blurred and the brightness difference between different parts of the license plate is large, the refinement effect will be significantly reduced. The reason is that when too few effective contours are detected, the coverage will not be in place, and too many contours will easily cause uneven distribution. Therefore, both of them may cause large deviations in the boundaries of the subsequent fitting, resulting in the truncation of the license plate number text, affecting the recognition accuracy.

[0004] With regard to the problem in related technologies that the license plate recognition accuracy rate decreases due to the license plate refinement algorithm, no effective solution has been proposed so far. Summary of the invention

[0005] The main purpose of this application is to provide a license plate recognition method and device combined with a license plate contour NMS refinement algorithm to solve the problem in the related art that the license plate recognition accuracy rate decreases due to the license plate refinement algorithm.

[0006] In order to achieve the above objectives, in a first aspect, the present application provides a license plate recognition method combined with a license plate contour NMS refinement algorithm, comprising the following steps:

[0007] License plate detection;

[0008] License plate refinement based on NMS refinement algorithm;

[0009] License plate correction;

[0010] License plate recognition.

[0011] The license plate detection includes collecting the image of the license plate to be detected;

[0012] The license plate refinement based on the NMS refinement algorithm includes:

[0013] Graying the license plate image to be detected to obtain a grayscale image;

[0014] The grayscale image is binarized by using multiple sets of different parameters to obtain multiple sets of corresponding binarized images;

[0015] For the plurality of groups of binary images, using a contour detection algorithm, detecting dispersed contours of the plurality of groups of binary images;

[0016] Preliminarily screening the dispersed profile to obtain a screened dispersed profile;

[0017] Performing NMS algorithm on the filtered scattered contours to obtain multiple groups of target contours;

[0018] superimposing the multiple groups of target contours in the same image, fitting the license plate boundary, and obtaining a refined license plate image;

[0019] The license plate correction corrects the refined license plate image by affine transformation to obtain a corrected license plate image.

[0020] The license plate recognition performs license plate recognition on the corrected license plate image to obtain a recognized license plate number.

[0021] The multiple groups of different parameters are used for binarization operation, including selecting different threshold constants (C) and neighborhood sizes (B) to perform multiple rounds of binarization operation on the grayscale image.

[0022] The preliminary screening sets a size threshold and an aspect ratio threshold according to the size and the aspect ratio. If the size and the aspect ratio are both within the threshold, the output is the filtered scattered contour. If one or both of the size and the aspect ratio are outside the threshold, the scattered contour at this time is filtered out.

[0023] The NMS algorithm comprises the following steps:

[0024] Add all the filtered scattered contours to the candidate list;

[0025] Select the first contour from the candidate list to add to the output list, and remove it from the candidate list.

[0026] Calculate the IOU of the currently selected contour and all other contours in the candidate list in turn, delete the candidate contours that are greater than the given contour threshold, and repeat the above process until the candidate list is empty.

[0027] The candidate contours in the output list are the final target contours.

[0028] The contours are processed using the same confidence level.

[0029] The license plate boundary fitting, for fitting the upper and lower boundaries of the license plate, selects the upper left corner point and the lower right corner point of each target contour respectively.

[0030] The NMS algorithm further adopts a block NMS algorithm to improve the computing efficiency and avoid the waste of computing resources. The process is as follows:

[0031] The collected license plate image to be detected is divided into N blocks, N<5, and each block is processed using the license plate refinement process based on the NMS refinement algorithm to obtain a refined license plate image for each block, and then all the blocks are combined to obtain a complete refined license plate image.

[0032] In the block NMS algorithm, the collected license plate image to be detected is divided into N blocks, and the edge filling method is used to eliminate the error caused by the block division, that is, each block will be expanded to both sides by a predetermined proportion during the NMS operation.

[0033] In a second aspect, the present application also provides a license plate recognition device combined with a license plate contour NMS refinement algorithm, including: a license plate detection module, a license plate refinement module, a license plate correction module and a license plate recognition module.

[0034] The license plate detection module, the license plate refinement module, the license plate correction module and the license plate recognition module are connected in sequence;

[0035] The license plate detection module collects the image of the license plate to be detected;

[0036] The license plate refinement module uses the NMS refinement algorithm to refine the license plate;

[0037] The license plate correction module corrects the refined license plate image by affine transformation to obtain a corrected license plate image;

[0038] The license plate recognition module performs license plate recognition on the corrected license plate image to obtain a recognized license plate number.

[0039] Beneficial technical effects:

[0040] The present application proposes a license plate recognition method and device combined with a contour NMS refinement algorithm. The main purpose is to perform multiple rounds of binarization operations and contour frame detection by adjusting the threshold constant (C) and the neighborhood size (B) during the dynamic threshold binarization of the image during the license plate refinement stage. Then, all detected contours are integrated and initially filtered to form candidate contours, and then the candidate contours are subjected to NMS operations. The present application not only retains the effective contours, but also reduces the overlap of contours, so that the license plate number boundaries subsequently fitted are more in line with the actual situation, thereby improving the accuracy of recognizing the license plate number. The block NMS algorithm is further used to divide the license plate image into blocks, and the divided images are subjected to NMS respectively, and finally an image is synthesized again, which improves the computational efficiency and saves computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0042] Figure 1 It is a flow chart of a license plate recognition method combined with a license plate contour NMS refinement algorithm provided according to an embodiment of the present application;

[0043] Figure 2 It is a license plate refinement flow chart based on the NMS refinement algorithm provided in an embodiment of the present application;

[0044] Figure 3 A license plate image with low recognition accuracy provided according to an embodiment of the present application;

[0045] Figure 4 It is a binary simulation effect diagram provided according to an embodiment of the present application;

[0046] Figure 5 This is a preliminary screening effect diagram provided according to the embodiment of the present application;

[0047] Figure 6 is a schematic diagram of overlapping contours provided according to an embodiment of the present application;

[0048] Figure 7 It is a schematic diagram after NMS algorithm processing according to an embodiment of the present application;

[0049] Figure 8 is a schematic diagram of an edge filling method provided according to an embodiment of the present application;

[0050] Fig. 9 is a schematic diagram of boundary fitting provided according to an embodiment of the present application;

[0051] Fig.10It is a schematic diagram of a license plate recognition device combined with a license plate contour NMS refinement algorithm provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0055] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to specific circumstances.

[0056] In addition, the term "plurality" shall mean two or more.

[0057] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0058] In the first aspect, the present application provides a license plate recognition method combined with a license plate contour NMS refinement algorithm, such as Figure 1 As shown, the following steps are included:

[0059] Step S100: license plate detection;

[0060] Step S101: refining the license plate based on the NMS refinement algorithm;

[0061] Step S102: License plate correction;

[0062] Step S103: License plate recognition.

[0063] The license plate detection includes collecting the image of the license plate to be detected;

[0064] In the license plate detection process, the detected license plate image usually carries different degrees of background information, affected by the detection method (rectangular frame), license plate angle, vehicle distance, license plate blur, etc. Figure 3 shown.

[0065] Directly using this type of image for license plate recognition will greatly reduce the recognition accuracy. Further license plate refinement can eliminate redundant information and obtain more accurate license plate boundaries, which can greatly optimize the recognition effect. In this application, the license plate grayscale image is binarized and contour detected multiple times by adjusting the threshold constant (C) and the neighborhood size (B). Finally, all detected contours are integrated, and the remaining contours are subjected to NMS operation after preliminary filtering to obtain the final target contour. The license plate boundary is fitted according to the vertex coordinates of each scattered target contour.

[0066] The license plate refinement based on the NMS refinement algorithm is as follows: Figure 2 As shown, including:

[0067] Step S200: graying the license plate image to be detected to obtain a grayscale image;

[0068] Step S201: performing a binarization operation on the grayscale image using multiple groups of different parameters to obtain corresponding multiple groups of binary images;

[0069] Step S202: detecting dispersed contours of the plurality of sets of binary images by using a contour detection algorithm;

[0070] Step S203: Preliminarily screening the dispersed profiles to obtain screened dispersed profiles;

[0071] Step S204: performing NMS algorithm on the filtered scattered contours to obtain multiple groups of target contours;

[0072] Step S205: superimposing the multiple groups of target contours in the same image, fitting the license plate boundary, and obtaining a refined license plate image;

[0073] The license plate correction corrects the refined license plate image by affine transformation to obtain a corrected license plate image.

[0074] The license plate recognition performs license plate recognition on the corrected license plate image to obtain a recognized license plate number.

[0075] The multiple groups of different parameters are used for binarization operation, including selecting different threshold constants (C) and neighborhood sizes (B) to perform multiple rounds of binarization operation on the grayscale image.

[0076] Different parts of the license plate image usually have different degrees of brightness. The adaptive threshold method is used in the binarization process. The threshold at this time is calculated based on each small area on the image. At the same time, the threshold constant (C) is added for fine-tuning. In order to reduce missed detection, the binarization operation is performed by adjusting different threshold constants (C) and neighborhood size (B) to generate multiple groups of binary images, and then contour detection is performed on each image. The binary simulation effect is shown in the figure below. Figure 4 shown.

[0077] The preliminary screening sets a size threshold and an aspect ratio threshold according to the size and the aspect ratio. If the size and the aspect ratio are both within the threshold, the output is the filtered scattered contour. If one or both of the size and the aspect ratio are outside the threshold, the scattered contour at this time is filtered out.

[0078] The ideal contour should contain all the independent letters or numbers in the license plate number. However, due to the influence of the external environment and the condition of the license plate itself, the shapes and sizes of the contours detected after binarization vary, and there are many contours that are obviously inconsistent. If they are sent directly to the next step without processing, the calculation burden will be seriously increased, and the final boundary fitting will be affected. Therefore, according to the requirements of size, aspect ratio, etc., all detected contours are filtered according to the empirical values ​​to form candidate contours. The preliminary screening effect is shown in the figure below. Figure 5 shown.

[0079] Different parts of the license plate have different brightness and clarity. During the multi-group binarization and contour detection process, for bright and clear parts, contours of similar size and proportion will be concentrated, while for relatively dim and blurred parts, contours are sparsely distributed. The area with concentrated distribution has a larger weight when fitting the boundary, causing the boundary to tilt towards it and produce deviation. How to suppress overlapping contours while retaining valid contours as much as possible is a major problem that needs to be solved. The schematic diagram of overlapping contours is shown in the figure below. Figure 6 shown.

[0080] In this application, a non-maximum suppression algorithm (NMS) is creatively introduced to filter the candidate contours. The schematic diagram after NMS algorithm processing is as follows: Figure 7 shown.

[0081] The NMS algorithm comprises the following steps:

[0082] Add all the filtered scattered contours to the candidate list;

[0083] Select the first contour from the candidate list to add to the output list, and remove it from the candidate list.

[0084] The IOU (Intersection over Union) of the currently selected contour and all other contours in the candidate list is calculated in turn, and the candidate contours that are greater than the given contour threshold are deleted. The above process is repeated until the candidate list is empty.

[0085] The candidate contours in the output list are the final target contours.

[0086] The contours are processed with the same confidence level. Because the difference in confidence level between contours is not involved, they can be processed with the same confidence level when performing NMS.

[0087] The fitting of the license plate boundary, for the fitting of the upper and lower boundaries of the license plate, selects the upper left corner point and the lower right corner point of each target contour respectively, and the boundary fitting schematic diagram is as follows: Fig. 9 shown.

[0088] Classic NMS needs to calculate all target contours uniformly, without considering the influence of position factors, which easily leads to a waste of computing resources. For example, when the positions of two contour boxes are far apart, there is no need to sort and calculate IOU. At the same time, this overall calculation method is not convenient for parallel execution, which limits the computing efficiency. For mobile terminals or embedded devices with limited computing power, it is particularly important to eliminate redundant calculations and improve computing efficiency.

[0089] The NMS algorithm further adopts a block NMS algorithm to improve the computing efficiency and avoid the waste of computing resources. The process is as follows:

[0090] The collected license plate image to be detected is divided into N blocks, N<5, and each block is processed using the license plate refinement process based on the NMS refinement algorithm to obtain a refined license plate image for each block, and then all the blocks are synthesized to obtain a complete refined license plate image.

[0091] The possible errors of the block NMS algorithm come from the block boundary part, because the contour box of the boundary part may be affected by the adjacent contour box during the suppression process.

[0092] In the block NMS algorithm, the collected license plate image to be detected is divided into N blocks, and the edge filling method is used to eliminate the error caused by the block division, that is, each block will be expanded to a predetermined ratio on both sides during the NMS operation. The schematic diagram of the edge filling method is as follows: Figure 8 shown.

[0093] In a second aspect, the present application also provides a license plate recognition device combined with a license plate contour NMS refinement algorithm, including: a license plate detection module, a license plate refinement module, a license plate correction module and a license plate recognition module.

[0094] The license plate detection module, license plate refinement module, license plate correction module and license plate recognition module are connected in sequence. Fig.10 As shown;

[0095] The license plate detection module collects the image of the license plate to be detected;

[0096] The license plate refinement module uses the NMS refinement algorithm to refine the license plate;

[0097] The license plate correction module corrects the refined license plate image by affine transformation to obtain a corrected license plate image;

[0098] The license plate recognition module performs license plate recognition on the corrected license plate image to obtain a recognized license plate number.

[0099] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A license plate recognition method combined with a license plate contour NMS refinement algorithm, characterized in that: include: License plate detection; License plate refinement based on NMS refinement algorithm; License plate correction; License plate recognition; The license plate detection includes collecting the image of the license plate to be detected; The license plate refinement based on the NMS refinement algorithm includes: Graying the license plate image to be detected to obtain a grayscale image; The grayscale image is subjected to binarization operation by using multiple sets of different parameters to obtain corresponding multiple sets of binarized images; the multiple sets of different parameters are subjected to binarization operation, including selecting different threshold constants and neighborhood sizes to perform multiple rounds of binarization operation on the grayscale image; For the plurality of groups of binary images, using a contour detection algorithm, detecting dispersed contours of the plurality of groups of binary images; Performing a preliminary screening on the dispersed contours to obtain a dispersed contour after screening; in the preliminary screening, a size threshold and an aspect ratio threshold are set according to the size and the aspect ratio, if the size and the aspect ratio are both within the threshold, the dispersed contour after screening is output, if one or both of the size and the aspect ratio are outside the threshold, the dispersed contour at this time is filtered out; The NMS algorithm is performed on the filtered scattered contours to obtain multiple groups of target contours; the NMS algorithm includes the following steps: adding all the filtered scattered contours to a candidate list; selecting the first contour from the candidate list to add to the output list, and deleting it from the candidate list; calculating the IOU of the currently selected contour and all other contours in the candidate list in turn, deleting the candidate contours greater than a given contour threshold, and repeating the above process until the candidate list is empty; the candidate contour in the output list is the final target contour; superimposing the multiple groups of target contours in the same image, fitting the license plate boundary, and obtaining a refined license plate image; The NMS algorithm further adopts a block NMS algorithm, and the process is as follows: the collected license plate image to be detected is divided into N blocks, each block is processed using the license plate refinement process based on the NMS refinement algorithm, and the refined license plate image of each block is obtained respectively, and then all the blocks are synthesized to obtain a complete refined license plate image; in the block NMS algorithm, the collected license plate image to be detected is divided into N blocks, and the edge filling method is adopted, that is, each block will perform a predetermined proportion of image expansion on both sides during the NMS operation.

2. The license plate recognition method combined with the license plate contour NMS refinement algorithm as claimed in claim 1, characterized in that: The license plate correction corrects the refined license plate image by affine transformation to obtain a corrected license plate image.

3. The license plate recognition method combined with the license plate contour NMS refinement algorithm as claimed in claim 2, characterized in that: The license plate recognition performs license plate recognition on the corrected license plate image to obtain a recognized license plate number.

4. A license plate recognition device combined with a license plate contour NMS refinement algorithm, characterized in that: The method for license plate recognition combined with the license plate contour NMS refinement algorithm as described in any one of claims 1 to 3 is implemented, comprising: a license plate detection module, a license plate refinement module, a license plate correction module and a license plate recognition module; The license plate detection module, the license plate refinement module, the license plate correction module and the license plate recognition module are connected in sequence; The license plate detection module collects the image of the license plate to be detected; The license plate refinement module uses the NMS refinement algorithm to refine the license plate; The license plate correction module corrects the refined license plate image by affine transformation to obtain a corrected license plate image; The license plate recognition module performs license plate recognition on the corrected license plate image to obtain a recognized license plate number.

Images