Multi-row license plate recognition method, system, device and readable storage medium

Abstract

The present invention provides a multi-row license plate recognition method, system, device and readable storage medium, belonging to the field of artificial intelligence. The method includes vehicle detection, license plate detection and character recognition. Vehicle detection and license plate detection complete the license plate detection task in two steps. Vehicle detection can effectively reduce the false recognition rate of the method of directly detecting license plates from natural images, while improving the recognition rate of unlicensed vehicles. License plate detection is performed on vehicle images obtained by vehicle detection, which can significantly reduce the disturbance caused by the natural background and effectively improve the license plate detection rate. Character recognition can accurately identify the license plate characters in the license plate image and perform statistics. The present invention uses deep learning to complete vehicle detection, license plate detection and character recognition, and can correctly identify single-row license plates, multiple-row license plates and no license plates. It can realize one-time recognition of the content of one-row license plates and multiple-row license plates, improve license plate recognition efficiency, and meet the current license plate recognition needs of highway toll stations.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a multi-row license plate recognition method, system, device and readable storage medium. Background Art

[0002] License plate recognition refers to the extraction of license plate information from image information. Since the license plate is an important information identifier of the vehicle, this technology is widely used in various occasions where vehicle control is required.

[0003] There are two main types of license plate recognition processes: one involves vehicle detection, license plate detection, and character recognition, and the other involves license plate detection and character recognition. The difference lies in whether the vehicle is detected before the license plate is detected. Both methods focus on license plate detection, and the ability to accurately detect the license plate position is crucial for subsequent character recognition. Current license plate recognition methods are primarily based on deep learning, which offers advantages such as high recognition accuracy, strong generalization, rapid deployment, and vehicle information retention.

[0004] Currently, license plate detection technology is widely used, and many commercial systems have emerged, but there are still some shortcomings. In special cases, existing license plate recognition systems cannot accurately detect the location and number of license plates, which may lead to license plate recognition errors. Summary of the Invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a multi-row license plate recognition method, system, device and readable storage medium, which complete vehicle detection, license plate detection and character recognition through deep learning, and can correctly identify single-row license plates, multiple rows of license plates and no license plates. The present invention can realize one-time recognition of the content of one row of license plates and multiple rows of license plates, improve the efficiency of license plate recognition, and meet the current license plate recognition needs of highway toll stations.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-row license plate recognition method includes the following process:

[0008] Pre-collected images containing vehicles are used as a vehicle image set. The vehicle image set is annotated using a pre-trained model to obtain a vehicle label set. The vehicle image set and vehicle label set together constitute a vehicle dataset. The target box detection model is trained using the vehicle dataset to obtain a vehicle detection model.

[0009] Vehicle images are captured from the vehicle dataset and resized as a license plate image set. The license plate image set is annotated using a pre-trained model, and single-row license plates and multi-row license plates are divided to obtain a license plate label set. The license plate image set and license plate label set together constitute the license plate dataset. The target box detection model is trained using the vehicle dataset to obtain a license plate detection model.

[0010] Extract license plate images from the license plate dataset and resize them as a character image set, annotate the character image set using a pre-trained model to obtain a character label set, the character image set and the character label set together constitute a character dataset, and train an optical character recognition model using the character dataset to obtain a character recognition model;

[0011] The vehicle detection module preprocesses the input image to obtain the vehicle image to be identified and the vehicle entry and exit status; the license plate detection module preprocesses the vehicle image to be identified to obtain the license plate image to be identified and the license plate type of the license plate image; when the license plate type of the license plate image to be identified is "multi-line", the license plate image to be identified is segmented and merged to obtain a license plate image of a single-line license plate type; the character statistics module recognizes the license plate image of the single-line license plate type and puts it into a dictionary; when the vehicle entry and exit status is "out", the correct license plate number is counted according to the dictionary.

[0012] Preferably, the generation of the vehicle data set specifically includes the following steps:

[0013] Step S1-1, collecting a data set of images with vehicles to form a vehicle image set;

[0014] Step S1-2: Use the existing multi-target detection pre-trained model to annotate the vehicle image set. The annotation content is the width and height of the rectangular box, denoted as w and h respectively, and the horizontal and vertical coordinates of the center point of the rectangular box. The upper left point of the image is the coordinate origin, the horizontal direction is the horizontal coordinate axis, and the vertical direction is the vertical coordinate axis, denoted as x and y respectively. The annotation format is (x, y, w, h); the detection results are cleaned and filtered to obtain the vehicle label set;

[0015] In step S1-3, the vehicle image set and the vehicle label set together constitute a vehicle dataset.

[0016] Preferably, the generation of the license plate dataset specifically includes the following steps:

[0017] Step S2-1, obtaining a license plate image set;

[0018] Step S2-2: Use the existing multi-target detection pre-training model to annotate the license plate image set. The license plate area must be completely enclosed in a rectangular box as small as possible. The annotation content is the license plate type, denoted as class, the width and height of the rectangular box, denoted as w and h respectively, the horizontal and vertical coordinates of the center point, with the upper left point of the image as the coordinate origin, the horizontal direction as the horizontal coordinate axis, and the vertical direction as the vertical coordinate axis, denoted as x and y respectively. The annotation format is (class, x, y, w, h); when annotating, only single-row license plates and multi-row license plates are divided into double-layer license plates and double license plates. The area where the multi-row license plates are located is annotated as a license plate area; the detection results are cleaned and filtered to obtain a license plate label set;

[0019] In step S2-3, the license plate image set and the license plate label set together constitute the license plate dataset.

[0020] Preferably, the generation of the character data set specifically includes the following steps:

[0021] Step S3-1, obtaining a character image set; the character image set is obtained by intercepting character images from the license plate image set based on the position information represented by the license plate label set;

[0022] Step S3-2: intercepting character images from the license plate image set according to the position information represented by the license plate label set, and then preprocessing the intercepted character images to form a license plate image set to be corrected;

[0023] Step S3-3, perform license plate correction on the license plate of the vehicle to be corrected, and calculate the transmission transformation matrix based on the four-point coordinate position of the license plate and the target position; the target position varies according to the type of license plate; the corrected image obtained, where the license plate type is a multi-row license plate image, is cut out from the original image and merged; the corrected image is denoted as M, with width and height (w, h) respectively, and the cutout upper part is denoted as a, with (0, 0) as the upper left point, and width and height respectively Zoom to The lower part of the image is b, with is the upper left point, and its width and height are Figure b is scaled to a width and height of Figure a and Figure b are merged left and right into Figure M′ to form the corrected image set;

[0024] Step S3-4: Use the pre-trained character recognition model to perform character recognition on the corrected image set and save it, modify the characters that are recognized incorrectly, and the reviewed character set is called the character label set;

[0025] In step S3-5, the above character image set and character label set together constitute a character dataset.

[0026] Preferably, the multi-row license plate includes a double-row license plate, and the double license plate, performing type conversion on the license plate image to be identified of the multi-row license plate type to obtain a license plate image of the single-row license plate type includes:

[0027] Based on preset segmentation parameters, the position of a segmentation line is determined on the image of a multi-row license plate type to be identified; the image of the multi-row license plate type to be identified is divided along the determined segmentation line to obtain an upper license plate image and a lower license plate image; the upper license plate image is resized to obtain a resized upper license plate image; and the resized upper license plate image and the lower license plate image are horizontally spliced ​​to obtain a converted license plate image of a single-row license plate type.

[0028] Preferably, preprocessing the vehicle image to obtain the license plate image to be identified and the license plate type includes:

[0029] The license plate detection model extracts image features from the license plate image. If no vehicle is detected, a preset blank license plate image is output; if a license plate is detected, a character image to be recognized and the license plate type of the character image to be recognized are obtained.

[0030] Preferably, the license plate image of the one-way license plate type is recognized and put into a dictionary including:

[0031] Extracting image features from a character image using a character detection model to obtain a character sequence; and storing the character sequence in a dictionary;

[0032] Storing the character sequence in a dictionary includes:

[0033] Construct a dictionary that can store keywords and values ​​corresponding to the keywords; when the input character sequence meets the license plate number regulations, the dictionary is searched using the character sequence as the keyword. If the keyword does not exist in the dictionary, the dictionary is added with the keyword and the corresponding value is set to one; if the keyword exists in the dictionary, the corresponding value is set to be increased by one.

[0034] A multi-row license plate recognition system includes: a vehicle detection module, a license plate detection module, a character recognition module and a character counting module;

[0035] The vehicle detection module is used to detect whether there is a vehicle in the input image. If not, it waits for the next input image. If it is, it intercepts the vehicle part of the input image and adjusts the image size to form a vehicle image to be identified;

[0036] The license plate detection module is used to detect whether there is a license plate in the image of the vehicle to be identified and determine the type of license plate. If not, a preset blank license plate image is inserted. If present, the license plate portion of the image to be identified is intercepted and corrected. Then, based on the license plate type, whether to intercept the license plate is selected. If it is a normal license plate, no operation is performed. If there are multiple rows of license plates, the license plate is intercepted into two parts, which are then spliced ​​together to form the license plate image to be identified.

[0037] The character recognition module is used to identify the license plate characters in the license plate image to be identified, determine whether the license plate character recognition result meets the license plate character regulations, and output it to the character statistics module if it is correct;

[0038] The character statistics module is used to store license plate characters in a dictionary, and the above modules are cycled until the vehicle detection module detects that the vehicle has left. All license plate numbers in the dictionary are screened, and the license plate number with the most occurrences is selected as the correct license plate.

[0039] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned methods for multi-row license plate recognition when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a multi-row license plate recognition method as described in any one of the above.

[0041] Compared with the prior art, the present invention has the following beneficial technical effects:

[0042] The present invention provides a multi-row license plate recognition method. By adopting a two-layer detection network, the license plate detection task is completed in two steps: vehicle detection and license plate detection. Vehicle detection can effectively reduce the false recognition rate of the method of directly detecting license plates from natural images, while improving the recognition rate of unlicensed vehicles; license plate detection is performed on vehicle images obtained from vehicle detection, which can greatly reduce the disturbance caused by the natural background and effectively improve the license plate detection rate.

[0043] This invention uses license plate detection to classify license plates into two categories: single-line and multi-line. The difference is that multi-line license plates have multiple lines of characters. Based on the classification results, different license plate correction parameters are applied to obtain a correctly proportioned image, thereby enhancing the accuracy of license plate character recognition. Furthermore, the license plate detection module segments the multi-line license plates into single-line plates, enabling accurate character recognition of both single-line and multi-line license plates. This method is a universal license plate detection and recognition method.

[0044] The license plate recognition method provided by the present invention uses a multi-frame statistical approach to identify the target vehicle's license plate. During the detection process, character recognition results that do not meet the license plate character standards are eliminated, effectively reducing the misrecognition rate of license plate characters. The correct license plate number can be determined based on the multi-frame recognition results, especially in the case of low image quality. The present invention can quickly generate a large number of labels for data samples, requiring only a simple screening of the detection results, effectively reducing the work of annotating data. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a schematic diagram of a single-frame image detection implementation flow of a multi-row license plate recognition method for a highway toll station according to an embodiment of the present invention.

[0046] Figure 2 The figure is a flow chart of a method for multi-row license plate recognition at a highway toll station according to an embodiment of the present invention.

[0047] Figure 3 The present invention is a flowchart of a method for multi-row license plate recognition applied at a highway toll station entrance according to an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the training of various models of a multi-row license plate recognition system for a highway toll station in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and description are to be considered as illustrative in nature and not restrictive.

[0050] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0051] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0052] In the present invention, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection, electrical connection, or communication; direct connection or indirect connection through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0053] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0054] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0055] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0057] The accompanying drawings illustrate various schematic diagrams of structures according to embodiments disclosed herein. These figures are not drawn to scale; for clarity, some details are exaggerated and some details may be omitted. The shapes of the various regions and layers shown in the figures, as well as their relative sizes and positional relationships, are merely exemplary and may deviate in practice due to manufacturing tolerances or technical limitations. Those skilled in the art may design regions / layers with different shapes, sizes, and relative positions as needed.

[0058] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0059] Example 1

[0060] A method for multi-row license plate recognition at a highway toll station according to an embodiment of the present invention includes the following steps:

[0061] Step S1: Constructing a vehicle dataset, specifically including the following steps:

[0062] In step S1-1, a dataset of images containing vehicles is collected, which is called a vehicle image dataset.

[0063] In step S1-2, use an existing pre-trained multi-target detection model, such as the Yolo series, to annotate the vehicle image set. The requirement is to use a small rectangular box to completely enclose the vehicle area. The annotation content is the width and height of the rectangular box, recorded as w and h respectively, and the horizontal and vertical coordinates of the rectangular box center (with the upper left point of the image as the coordinate origin, the horizontal direction as the horizontal axis, and the vertical direction as the vertical axis), recorded as x and y respectively. The annotation format is (x, y, w, h). The detection results are cleaned and filtered to obtain the vehicle label set.

[0064] Step S1-3: The above vehicle image set and vehicle label set together constitute the vehicle dataset

[0065] Step S2, constructing a license plate dataset, includes the following steps:

[0066] Step S2-1: Obtain a license plate image set. License plate images are captured from the vehicle image set based on the position information represented by the vehicle tag set, which is called a license plate image set.

[0067] Step S2-2, use the existing multi-target detection pre-training model, such as the Yolo series, to annotate the license plate image set. It is required to use a rectangular frame as small as possible to completely wrap the license plate area. The annotation content is the license plate type, recorded as class, the width and height of the rectangular frame, recorded as w and h respectively, the horizontal coordinates and vertical coordinates of the center point (with the upper left point of the image as the coordinate origin, the horizontal direction as the horizontal coordinate axis, and the vertical direction as the vertical coordinate axis), recorded as x and y respectively, and the annotation format is (class, x, y, w, h). When annotating, it is only divided into single-row license plates and multi-row license plates, among which multi-row license plates are double-layer license plates and double license plates. The area where the multi-row license plates are located is marked as a license plate area. The detection results are cleaned and filtered to obtain a license plate label set.

[0068] In step S2-3, the license plate image set and the license plate label set together constitute the license plate dataset.

[0069] Step S3, constructing a character dataset, includes the following steps:

[0070] Step S3-1, obtaining a character image set. Character images are captured from the license plate image set according to the position information indicated by the license plate label set, which is called a character image set.

[0071] Step S3-2, based on the position information represented by the license plate label set, character images are captured from the license plate image set, and then the captured character images are preprocessed, such as image scaling, denoising, grayscale processing, histogram equalization, binarization operations, etc., which are called the license plate image set to be corrected.

[0072] Step S3-3, perform license plate correction on the license plate to be corrected, and calculate the transmission transformation matrix based on the four-point coordinate position of the license plate and the target position. The target position varies depending on the license plate type. The corrected image obtained, where the license plate type is a multi-row license plate image, is cut out from the original image and merged. The corrected image is denoted as M, with width and height (w, h) respectively. The cutout upper part is denoted as a, with (0, 0) as the upper left point, and width and height respectively. Zoom to The lower part of the image is b, with is the upper left point, and its width and height are Figure b is scaled to a width and height of Figure a and Figure b are merged left and right into Figure M′, which is called the corrected image set.

[0073] In step S3-4, the pre-trained character recognition model is used to perform character recognition on the corrected image set and save the image set, and the incorrectly recognized characters are modified. The reviewed character set is called a character label set.

[0074] In step S3-5, the above character image set and character label set together constitute a character dataset.

[0075] Step S4: Based on the above vehicle dataset, YoloV8s is used as a pre-trained model to train a vehicle detection model; based on the above license plate dataset, YoloV8s is used as a pre-trained model to train a license plate detection model; based on the above character dataset, LPR-NET is used as a pre-trained model to train a character recognition model.

[0076] In step S5, the vehicle detection module preprocesses the input image to obtain the image of the vehicle to be identified and the vehicle entry and exit status; the license plate detection module preprocesses the vehicle image to be identified to obtain the license plate image to be identified and the license plate type of the license plate image; when the license plate type of the license plate image to be identified is "multi-line", the license plate image to be identified is segmented and merged to obtain a license plate image of a single-line license plate type; the character statistics module recognizes the license plate image of the single-line license plate type and puts it into a dictionary; when the vehicle entry and exit status is "out", the correct license plate number is counted according to the dictionary.

[0077] Example 2

[0078] A multi-row license plate recognition system for a highway toll station according to an embodiment of the present invention includes: a vehicle detection module, a license plate detection module, a character recognition module, and a character counting module.

[0079] The vehicle detection module is used to detect whether there is a vehicle in the input image. If not, it waits for the next input image. If it is, it captures the vehicle part of the input image and resizes the image, which is called the vehicle image to be identified.

[0080] The license plate detection module is used to detect whether there is a license plate in the above-mentioned vehicle image to be identified and determine the license plate type. If not, a preset blank license plate image is inserted. If it exists, the license plate part in the above-mentioned vehicle image to be identified is intercepted and corrected. Then, based on the license plate type, whether to intercept the license plate is selected. If it is a normal license plate, no operation is performed. If there are multiple rows of license plates, the license plate is intercepted into two parts, upper and lower, and spliced ​​together, which is called the license plate image to be identified.

[0081] The character recognition module is used to identify the license plate characters in the license plate image to be identified, and to determine whether the license plate character recognition result meets the license plate character regulations. If correct, it is output to the character statistics module.

[0082] The character statistics module stores the license plate characters in a dictionary, loops through the modules until the vehicle detection module detects that the vehicle has left, filters all license plate numbers in the dictionary, and selects the license plate number with the most occurrences as the correct one.

[0083] Example 3

[0084] like Figure 1 1 shows a schematic diagram of a single-frame image detection implementation process of a multi-row license plate recognition method for a highway toll station provided by an embodiment of the present application, which specifically includes the following steps S101 to S105.

[0085] In step S101 , the vehicle detection module processes the input image to obtain a vehicle image to be identified.

[0086] The input image in the embodiment of the present disclosure is not limited by its source and acquisition method. For example, it can be a picture directly taken by a roadside surveillance camera, or it can be a picture in a recorded video, or it can be a picture processed by a computer. The vehicle image to be identified is captured from the input image, and the position of the vehicle in the input image is determined by the vehicle detection model, and then it is captured from the input image based on the position information. The vehicle detection model can be a model of the Yolo series, which performs target detection tasks. The model uses a bounding box to describe the spatial position of the object. The bounding box is a rectangle, and the position is determined by the width and height of the rectangle and the horizontal and vertical coordinates of the center point. The coordinate system uses the upper left point of the image as the coordinate origin, the horizontal direction as the horizontal coordinate axis, and the vertical direction as the vertical coordinate axis. The vehicle types of the vehicle image to be identified include passenger cars, trucks, special operation vehicles, etc.

[0087] In step S102, the license plate detection module processes the image of the vehicle to be identified obtained in the above step S101 to obtain the image of the license plate to be identified and the license plate type of the image of the license plate to be identified. The image of the license plate to be identified is intercepted from the image of the vehicle to be identified. The license plate detection model is used to determine the position of the license plate in the input image, and then the position information is estimated to be intercepted from the image of the vehicle to be identified. The license plate detection model can be a model of the Yolo series, which performs target detection tasks. The model uses a bounding box to describe the spatial position of the object. The bounding box is a rectangle, and the position is determined by the width and height of the rectangle and the horizontal and vertical coordinates of the center point. The coordinate system of the horizontal and vertical coordinates takes the upper left point of the image as the coordinate origin, the horizontal direction is the horizontal coordinate axis, and the vertical direction is the vertical coordinate axis. The license plate detection model can recognize a variety of license plates currently in use in China, including but not limited to large vehicle license plates, trailer license plates, large new energy vehicle license plates, small vehicle license plates, small new energy vehicle license plates, embassy vehicle license plates, Hong Kong and Macao entry license plates, training vehicle license plates, and police vehicle license plates as specified in "GA36-2018 Motor Vehicle License Plates of the People's Republic of China." The license plate type of the image to be recognized is detected and acquired by the license plate detection model. It is categorized as single-line or multi-line license plates, distinguished by whether the license plate contains one or more lines of characters. For example, a small new energy vehicle license plate is a single-line license plate, while a large vehicle rear license plate and a trailer license plate are multi-line license plates.

[0088] In step S103, the license plate detection module corrects the license plate image to be identified obtained in the above step S102. The transmission transformation matrix is ​​calculated based on the coordinates of the four corners of the license plate and the target coordinates, and then the license plate image to be identified is corrected based on the transformation matrix. The resulting image is called the corrected license plate image. The coordinates of the four corners of the license plate are not limited by their source and acquisition method. For example, the straight line of the license plate edge can be detected based on the Hough transform to obtain the inclination angle, and the license plate corner point can be calculated based on the inclination angle; or the image features of the license plate image to be identified can be extracted based on the convolutional neural network to obtain the license plate corner point. The target coordinates are related to the license plate type of the above-mentioned license plate image to be identified. The following explanation is based on the example of the multi-row license plate being a double-row license plate. It should be understood that the multi-row license plate splicing and conversion processing of the disclosed embodiment is not limited to double-row license plates, but can be applied to any multi-row license plates. If it is a single-row license plate, the target coordinates are {(0,0), (440,0), (0,140), (440,140)}; if it is a double-row license plate, the target coordinates are {(0,0), (440,0), (0,240), (440,240)}.

[0089] In step S104, if the license plate type of the license plate image to be identified is a multi-row license plate, the license plate detection module segments and merges the above-mentioned corrected license plate image to obtain a license plate image of a single-row license plate type. The segmentation and merging refers to dividing the corrected license plate image of a multi-row license plate into two parts. The following description will be made by taking the multi-row license plate as a double-row license plate as an example. It should be understood that the multi-row license plate splicing and conversion processing of the embodiment of the present disclosure is not limited to double-row license plates, but can be applied to any multi-row license plates. Let the corrected image be M, the width and height be (w, h) respectively, the intercepted upper part of the image be a, with (0,0) as the upper left point, the width and height be Zoom to The lower part of the image is b, with is the upper left point, and its width and height are Figure b is scaled to a width and height of Figure a and Figure b are merged into a single-row license plate type license plate image, denoted as M′.

[0090] In step S105, the character recognition module recognizes the single-row license plate image obtained in step S104 to obtain the license plate number. This recognition process can be performed using an existing single-row license plate recognition model. The single-row license plate recognition model can select a convolutional neural network, a recurrent neural network, or a character segmentation method to recognize license plate characters based on actual needs.

[0091] Figure 2 A complete implementation flow chart of a multi-row license plate recognition method for a highway toll station provided by an embodiment of the present application is shown.

[0092] In step S201 , a natural image is acquired from an input device to obtain an input image.

[0093] In step S202, vehicle detection is performed on the input image. If a vehicle is detected in the input image in step S203, it is considered that the vehicle has entered, and the image of the vehicle to be identified is obtained. The license plate characters in the image of the vehicle to be identified are identified and counted. If no vehicle is detected in the input image in step S204, it is considered that the vehicle has left, and the dictionary is checked to see whether the license plate number is stored. If the dictionary is not empty, it is considered that a vehicle has passed by before, and the license plate number with the highest frequency is output.

[0094] As described above, in step S203, a vehicle is detected in the input image, and the license plate characters in the image of the vehicle to be identified need to be recognized and counted, which specifically includes the following sub-steps:

[0095] Step S2031 detects whether there is a license plate in the image of the vehicle to be identified and determines the type of license plate. If no license plate is detected, it is considered to be an unlicensed vehicle, and a preset blank license plate image is output, skipping step S2032. If a license plate is detected, it is divided into single-line license plates and multi-line license plates according to the number of rows of license plate characters, and the license plate part in the above-mentioned vehicle image to be identified is intercepted.

[0096] Step S2032 uses the coordinates of the four corners of the license plate as input points. The target coordinates are then determined based on the license plate type. A transmission transformation matrix is ​​calculated using these input and target coordinates, and the corrected license plate image is obtained through matrix calculation. Depending on the license plate type, whether to intercept the license plate is determined. If the license plate is single-row, no action is required. If the license plate is multi-row, the license plate image is converted to a single-row image, referred to as the license plate image to be identified.

[0097] Step S2033 performs character recognition on the license plate image to be identified and determines whether it meets the license plate number regulations, and outputs the license plate number that meets the regulations.

[0098] Step S2034 counts the output license plate numbers and puts them into a dictionary, and then loops through the above steps until step S202 detects no vehicle, deems the vehicle has left, and outputs the license plate number.

[0099] Figure 3 A schematic flow chart of a method of applying an embodiment of the present disclosure when using a camera at an entrance of a highway toll station to implement vehicle tracking is shown.

[0100] Reference Figure 3Next, the method described in the embodiment of the present disclosure will be further described in a real scenario of vehicle tracking through cameras at the entrance of a highway toll station. In the case of vehicle tracking across cameras, based on the input image containing the vehicle captured in the current lane, the method described in the embodiment of the present disclosure first pre-processes the input image based on step S301 to obtain an image of the vehicle to be identified. In this process, a convolutional neural network can be used to extract features from the input image to obtain an image of the vehicle to be identified.

[0101] Step S302: pre-process the image of the vehicle to be identified to obtain the license plate image to be identified and the license plate type of the license plate image to be identified. This process can use a convolutional neural network to extract features from the image of the vehicle to be identified to obtain the license plate image of the vehicle to be identified and the license plate type of the license plate image to be identified. Figure 3 As shown in , it is determined that the license plate type in the license plate image to be identified is "multi-line" and the license plate image is obtained. Further, the license plate image is rectified and mapped to a horizontal position using perspective transformation.

[0102] Thereafter, in step S303, based on the determination that the license plate type of the current license plate image to be identified is multi-row, multiple license plate sub-images are obtained based on preset segmentation parameters, which are then resized and spliced ​​to obtain a converted license plate image of a single-row license plate type.

[0103] Finally, in step S304, the license plate image of the single-row license plate type is subjected to character recognition to obtain the license plate number. This process can use a neural network to perform feature extraction, and ultimately obtain the license plate number of the vehicle body in the input image.

[0104] Figure 4 This paper presents a license plate recognition model training method that combines the YoloV8s object detection network and the LPR-NET network based on existing algorithms to recognize multiple rows of license plates. This method mainly involves training three models: 1) a vehicle detection model, 2) a license plate detection model, and 3) a character recognition model.

[0105] Vehicle detection model training includes the following steps:

[0106] Step S401 collects vehicle images with both single-row and multi-row license plates to form a vehicle image set. The number of images of each type must be evenly distributed. Vehicles in the collected vehicle images are then calibrated using rectangular frames to form vehicle region rectangular frames. This process can use an existing pre-trained model for preliminary labeling, followed by manual screening to correct misidentifications, missed identifications, and inaccurate identification positions to obtain a vehicle label set. The vehicle image set and vehicle label set together constitute the vehicle dataset.

[0107] Step S402: Use the vehicle dataset to perform model training to obtain a vehicle detection model. This process can use a classic classification model for transfer learning, such as using the YoloV8s model as a pre-trained model and using the vehicle dataset for transfer learning.

[0108] In step S403, according to the position information represented by the vehicle label set, the license plate image is cropped from the vehicle image set, and then the image size is adjusted to obtain a license plate image set. A rectangular frame is then used to calibrate the position and type of the license plates in the vehicle image set to form a license plate area rectangular frame and a license plate type. The license plate rectangular frame calibration process can use an existing pre-trained model for preliminary marking, and then perform manual screening to correct the marking results of misidentification, missed recognition, and inaccurate recognition positions. Ordinary license plates, such as small new energy vehicle license plates, are required to be single-row license plates, and special license plates, such as double license plates hung on cars entering Hong Kong and Macau, large car rear license plates, and trailer license plates, are required to be calibrated into a rectangular frame and multi-row license plates. The license plate image set and the license plate label set together constitute the license plate dataset.

[0109] Step S404: Use the license plate dataset to perform model training to obtain a license plate detection model. This process can use a classic classification model for transfer learning, such as using the YoloV8s model as a pre-trained model and using the license plate dataset for transfer learning.

[0110] In step S405, based on the position information indicated by the license plate label set, character images are cropped from the license plate image set, the images are resized and corrected, and perspective transformation is used to map them to a horizontal position, thereby obtaining a character image set. Subsequently, the characters in the character image set are represented as text using a text format to form a character label set. This process can use an existing pre-trained model for preliminary labeling, followed by manual screening to correct misidentified or missed characters and obtain a character label set. The character image set and character label set together constitute the character dataset.

[0111] Step S406: Use the character dataset to perform model training to obtain a character recognition model. This process can use a classic optical character recognition model for transfer learning, such as using Paddle OCR as a pre-trained model and using the character dataset for transfer learning.

[0112] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0113] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a multi-row license plate recognition method.

[0114] In one embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-row license plate recognition method in the above embodiment.

[0115] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

[0120] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, from all points of view, the embodiments should be regarded as illustrative and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and range of equivalents of the claims are included in the present invention. Any reference signs in the claims should not be construed as limiting the claim to which they relate.

[0121] In addition, it should be understood that although this specification describes the embodiments, not every embodiment contains only one independent technical solution. This description is for clarity only. Those skilled in the art should consider the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for the purpose of illustrating the technical concept of the present invention and cannot be used to limit the scope of protection of the present invention. Any changes made based on the technical solution in accordance with the technical concept proposed by the present invention fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-row license plate recognition method, characterized in that: Including the following process, Pre-collected images containing vehicles are used as a vehicle image set. The vehicle image set is annotated using a pre-trained model to obtain a vehicle label set. The vehicle image set and vehicle label set together constitute a vehicle dataset. The target box detection model is trained using the vehicle dataset to obtain a vehicle detection model. Vehicle images are captured from the vehicle dataset and resized as a license plate image set. The license plate image set is annotated using a pre-trained model, and single-row license plates and multi-row license plates are divided to obtain a license plate label set. The license plate image set and license plate label set together constitute the license plate dataset. The target box detection model is trained using the vehicle dataset to obtain a license plate detection model. Extract license plate images from the license plate dataset and resize them as a character image set, annotate the character image set using a pre-trained model to obtain a character label set, the character image set and the character label set together constitute a character dataset, and train an optical character recognition model using the character dataset to obtain a character recognition model; The generation of the character data set specifically includes the following steps: Acquire a character image set; the character image set is a character image intercepted from the license plate image set based on the position information represented by the license plate label set; intercepting character images from the license plate image set according to the position information represented by the license plate label set, and then preprocessing the intercepted character images to form a license plate image set to be corrected; The license plate of the vehicle to be corrected is corrected, and the transmission transformation matrix is ​​calculated based on the coordinate positions of the four points of the license plate and the target position; the target position varies according to the type of the license plate; the corrected image is obtained, where the license plate type is a multi-row license plate image, and the upper and lower images are cut out from the original image and merged; the corrected image is denoted as M, with width and height (w, h) respectively, and the cut upper part is denoted as a, with (0, 0) as the upper left point, and width and height respectively Zoom to The lower part of the image is b, with is the upper left point, and its width and height are Figure b is scaled to a width and height of Figure a and Figure b are merged left and right into Figure M′ to form the corrected image set; Use the pre-trained character recognition model to perform character recognition on the corrected image set and save it. Modify the characters that are recognized incorrectly. The reviewed character set is called the character label set. The above character image set and character label set together constitute the character dataset; The vehicle detection module preprocesses the input image to obtain the image of the vehicle to be identified and the vehicle entry and exit status; the license plate detection module preprocesses the vehicle image to be identified to obtain the license plate image to be identified and the license plate type of the license plate image; when the license plate type of the license plate image to be identified is "multi-line", the license plate image to be identified is segmented and merged to obtain a license plate image of a single-line license plate type; the character statistics module recognizes the license plate image of the single-line license plate type and puts it into a dictionary; when the vehicle entry and exit status is "out", the correct license plate number is counted according to the dictionary.

2. A multi-row license plate recognition method according to claim 1, characterized in that: The generation of the vehicle data set specifically includes the following steps: Collect a dataset of images with vehicles to form a vehicle image set; Using an existing pre-trained multi-target detection model, the vehicle image set is annotated. The annotation content includes the width and height of the rectangular box, denoted as w and h respectively, and the horizontal and vertical coordinates of the rectangular box center point. The upper left point of the image is the coordinate origin, the horizontal direction is the horizontal axis, and the vertical direction is the vertical axis, denoted as x and y respectively. The annotation format is (x, y, w, h). The detection results are cleaned and filtered to obtain the vehicle label set. The above vehicle image set and vehicle label set together constitute the vehicle dataset.

3. A multi-row license plate recognition method according to claim 1, characterized in that: The generation of the license plate data set specifically includes the following steps: Get the license plate image set; Use the existing multi-target detection pre-training model to annotate the license plate image set mentioned above. It is required to use a rectangular box as small as possible to completely enclose the license plate area. The annotation content is the license plate type, recorded as class, the width and height of the rectangular box, recorded as w and h respectively, the horizontal coordinates and vertical coordinates of the center point, with the upper left point of the image as the coordinate origin, the horizontal direction as the horizontal coordinate axis, and the vertical direction as the vertical coordinate axis, recorded as x and y respectively. The annotation format is (class, x, y, w, h); when annotating, only single-row license plates and multi-row license plates are divided into double-layer license plates and double license plates. The area where the multi-row license plates are located is marked as a license plate area; the detection results are cleaned and filtered to obtain the license plate label set; The license plate image set and the license plate label set together constitute the license plate dataset.

4. A multi-row license plate recognition method according to claim 1, characterized in that: The multi-row license plate includes a double-row license plate and a double license plate. Converting the image of the license plate to be identified of the multi-row license plate type to obtain a license plate image of the single-row license plate type includes: Based on preset segmentation parameters, the position of a segmentation line is determined on the image of a multi-row license plate type to be identified; the image of the multi-row license plate type to be identified is divided along the determined segmentation line to obtain an upper license plate image and a lower license plate image; the upper license plate image is resized to obtain a resized upper license plate image; and the resized upper license plate image and the lower license plate image are horizontally spliced ​​to obtain a converted license plate image of a single-row license plate type.

5. The multi-row license plate recognition method according to claim 1, characterized in that: Preprocess the vehicle image to obtain the license plate image to be identified and the license plate type including: The license plate detection model extracts image features from the license plate image. If no vehicle is detected, a preset blank license plate image is output; if a license plate is detected, a character image to be recognized and the license plate type of the character image to be recognized are obtained.

6. A multi-row license plate recognition method according to claim 1, characterized in that: Recognize the license plate image of the one-way license plate type and put it into the dictionary including: Extracting image features from a character image using a character detection model to obtain a character sequence; and storing the character sequence in a dictionary; Storing the character sequence in a dictionary includes: Construct a dictionary that can store keywords and values ​​corresponding to the keywords; when the input character sequence meets the license plate number regulations, the dictionary is searched using the character sequence as the keyword. If the keyword does not exist in the dictionary, the dictionary is added with the keyword and the corresponding value is set to one; if the keyword exists in the dictionary, the corresponding value is set to be increased by one.

7. A multi-row license plate recognition system, used to implement the multi-row license plate recognition method according to any one of claims 1 to 6, characterized in that: include: Vehicle detection module, license plate detection module, character recognition module and character statistics module; The vehicle detection module is used to detect whether there is a vehicle in the input image. If not, it waits for the next input image. If it is, it intercepts the vehicle part of the input image and adjusts the image size to form a vehicle image to be identified; The license plate detection module is used to detect whether there is a license plate in the image of the vehicle to be identified and determine the type of license plate. If not, a preset blank license plate image is inserted. If present, the license plate portion of the image to be identified is intercepted and corrected. Then, based on the license plate type, whether to intercept the license plate is selected. If it is a normal license plate, no operation is performed. If there are multiple rows of license plates, the license plate is intercepted into two parts, which are then spliced ​​together to form the license plate image to be identified. The character recognition module is used to identify the license plate characters in the license plate image to be identified, determine whether the license plate character recognition result meets the license plate character regulations, and output it to the character statistics module if it is correct; The character statistics module is used to store license plate characters in a dictionary, and the above modules are cycled until the vehicle detection module detects that the vehicle has left. All license plate numbers in the dictionary are screened, and the license plate number with the most occurrences is selected as the correct license plate.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the multi-row license plate recognition method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a multi-row license plate recognition method as described in any one of claims 1 to 6 are implemented.

Images