A license plate detection method, a license plate detector and related devices

Abstract

The application discloses a license plate detection method, a license plate detector and related equipment, and relates to the technical field of image processing, and in particular to a license plate detection method, a license plate detector and related equipment. The license plate detection method comprises the following steps: acquiring a to-be-detected vehicle image containing a license plate; correcting a plurality of prediction points detected from the to-be-detected vehicle image, wherein the prediction points are associated with the to-be-detected license plate; acquiring a license plate region image corresponding to a to-be-processed license plate from the to-be-detected vehicle image by using the plurality of corrected prediction points; and identifying image features of the license plate region image to obtain license plate information of the to-be-processed license plate. In the foregoing manner, the application can improve the detection accuracy of the license plate.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a license plate detection method, a license plate detector and related equipment. Background Art

[0002] With the rapid growth of my country's car ownership, road traffic problems have become a global issue. Traffic violations and illegal parking are common, seriously affecting road safety. License plates are a key feature of vehicle identification, making license plate detection particularly important. However, when the license plate is at a large angle, detection can be inaccurate. Summary of the Invention

[0003] The main technical problem solved by the present invention is to provide a license plate detection method, a license plate detector and related equipment, which can improve the detection accuracy of license plates.

[0004] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a license plate detection method, which includes: obtaining a vehicle image to be detected containing a license plate; correcting multiple prediction points detected from the vehicle image to be detected, and the prediction points are associated with the license plate to be detected; using the corrected multiple prediction points, obtaining a license plate area image corresponding to the license plate to be processed from the vehicle image to be detected; identifying image features of the license plate area image to obtain license plate information of the license plate to be processed.

[0005] Among them, using the corrected multiple prediction points, the license plate area image corresponding to the license plate to be processed is obtained from the vehicle image to be detected, including: determining the target image area in the image to be processed that contains all the license plate area prediction points and whose size meets the preset requirements as the license plate area image, wherein: the size of the target image area is smaller than the size of the vehicle image to be detected.

[0006] Among them, correcting multiple prediction points detected from the image of the vehicle to be detected includes: extracting image feature information of the image of the vehicle to be detected; obtaining multiple prediction points and prediction offsets corresponding to each of the multiple prediction points from the image feature information; and correcting the prediction points using the prediction offsets to obtain multiple license plate area prediction points.

[0007] Among them, obtaining multiple prediction points and the prediction offset corresponding to each prediction point in the multiple prediction points from the image feature information includes: inputting the image feature information into a deformable convolutional network; the deformable convolutional network includes a main network, a prediction point detection subnetwork and an offset analysis subnetwork; using the main network to process the image feature information to obtain prediction feature information; using the prediction point detection subnetwork to process the prediction feature information to obtain multiple prediction points; using the offset analysis subnetwork to process the prediction feature information and the multiple prediction points to obtain prediction offsets corresponding to the multiple prediction points.

[0008] Among them, the main network of the positioning branch network includes three consecutive convolutional layers to perform convolution processing on the image feature information to obtain predicted feature information; the prediction point detection subnetwork includes a consecutive first convolutional layer and a second convolutional layer to perform convolution processing on the predicted feature information to obtain multiple prediction points. The first convolutional layer is a 3*3 convolutional layer, and the second convolutional layer is a 1*1 convolutional layer; the offset analysis subnetwork includes a consecutive third convolutional layer and a fourth convolutional layer. The third convolutional layer is a 3*3 convolutional layer, and the fourth convolutional layer is a 1*1 convolutional layer.

[0009] Among them, the prediction offset is the offset of the prediction point relative to the prediction point of the previous prediction. The initial prediction point information is corrected by using the prediction offset to obtain multiple license plate area prediction point information, including: fusing the first prediction point with the corresponding prediction offset to obtain the second prediction point; calculating the prediction offset corresponding to the second prediction point; taking the second prediction point as the new first prediction point, executing the steps of fusing the first prediction point with the corresponding prediction offset to obtain the second prediction point information, and calculating the prediction offset corresponding to the second prediction point, until the prediction offset reaches the termination condition to obtain multiple license plate area prediction points.

[0010] The deformable convolutional network also includes a classification branch network, and the method further includes: inputting image feature information into the classification branch network, and obtaining license plate category information of the license plate from the image feature information.

[0011] The step of extracting image feature information of the vehicle image to be detected includes extracting image feature information of the vehicle image to be detected by using a feature pyramid network to obtain image pyramid feature information.

[0012] Among them, obtaining the image of the vehicle to be detected containing the license plate includes: performing vehicle detection on the image of the vehicle to be detected containing the vehicle to obtain the vehicle area; performing magnification processing on the vehicle area with the center point of the vehicle area as the base point to obtain the vehicle area image; performing license plate hanging position detection on the vehicle area image to obtain the license plate hanging position area, and the license plate hanging position includes the front and / or rear of the vehicle; performing magnification processing on the license plate hanging position area with the center displacement point of the license plate hanging position area as the base point to obtain the image of the vehicle to be detected containing the license plate; the center displacement point is a point where the center point of the license plate hanging position area is displaced downward by a preset height.

[0013] To solve the above technical problems, another technical solution adopted by the present invention is: providing a license plate detector, which includes an acquisition module, a detection module and an identification module. The acquisition module is used to obtain a vehicle image to be detected containing a license plate; the detection module is used to perform license plate detection on the vehicle image to be detected based on a representative point set representation method to obtain a license plate area image; the identification module is used to identify image features of the license plate area image to obtain license plate information.

[0014] In order to solve the above technical problem, another technical solution adopted by the present invention is: providing a license plate detection device, the license plate detection device includes a processor, and the processor is used to execute to implement the above license plate detection method.

[0015] In order to solve the above technical problems, another technical solution adopted by the present invention is: providing a computer-readable storage medium, which is used to store instructions / program data, and the instructions / program data can be executed to implement the above license plate detection method.

[0016] The beneficial effects of the present invention are: different from the existing technology, the present invention addresses the problem of greater difficulty in license plate detection by using a representative point set representation method to obtain a license plate area image. This method has the potential to adapt to changes in the posture or shape of the license plate. The extracted point set can be adaptively distributed in local areas with important semantic significance in the license plate and can represent the geometric extension of the license plate, thereby providing a more detailed geometric description of the license plate, such as the four corner points of the license plate, thereby improving the detection accuracy of the license plate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a license plate detection method in an embodiment of the present application;

[0018] Figure 2 This is a flow chart of a method for obtaining an image of a vehicle to be detected containing a license plate in an embodiment of the present application;

[0019] Figure 3 This is a flow chart of a method for obtaining a license plate area image in an embodiment of the present application;

[0020] Figure 4 Schematic diagram of the process of the deformable convolutional network method in the embodiment of the present application;

[0021] Figure 5 is a structural diagram of the deformable convolutional network method in an embodiment of the present application;

[0022] Figure 6 It is a panoramic image of a large truck captured by a camera in a specific embodiment of the present application;

[0023] Figure 7 This is a schematic diagram of the result of the first frame zooming operation in this application;

[0024] Figure 8 This is a schematic diagram of the result of the second frame zooming operation in this application;

[0025] Figure 9 is a schematic diagram of the license plate detection frame obtained from the representative point set of this application;

[0026] Figure 10 This is a schematic diagram of the structure of the license plate detector in the embodiment of the present application;

[0027] Figure 11 This is a schematic diagram of the structure of the license plate detection device in the embodiment of the present application;

[0028] Figure 12 It is a schematic diagram of the structure of the computer-readable storage medium in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0030] The present application provides a license plate detection method. To address the difficulty of license plate detection, a representative point set representation method is used to obtain a license plate area image. This method has the potential to adapt to changes in the posture or shape of the license plate. The extracted point set can be adaptively distributed in local areas of the license plate with important semantic significance and can represent the geometric extension of the license plate, thereby providing a more detailed geometric description of the license plate, such as the four corner points of the license plate, thereby improving the detection accuracy of the license plate.

[0031] See also Figure 1 , Figure 1 It is a flowchart of a license plate detection method in the embodiment of the present application. It should be noted that if there are substantially the same results, this embodiment is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, this embodiment includes:

[0032] S110: Acquire an image of the vehicle to be detected including the license plate.

[0033] The image containing the vehicle license plate can be obtained by using the surveillance video captured by the surveillance camera, or the image containing the vehicle license plate can be obtained locally. In the embodiment of the present application, when obtaining the image using the surveillance video, the surveillance video captured by the surveillance camera is first obtained, and the video is frame-processed to obtain each frame of the surveillance video as the image of the vehicle to be detected.

[0034] S130: Correcting the plurality of predicted points detected from the image of the vehicle to be detected.

[0035] Because license plates are small and occupy a relatively small proportion of captured images or surveillance videos, direct license plate recognition reduces the quality of extracted features, thereby impairing recognition performance. Therefore, the license plate area is first detected, and multiple prediction points associated with the license plate to be detected are extracted. In this embodiment, a representative point set representation method is used to adaptively extract multiple semantically significant prediction points distributed within the license plate, and these prediction points are then corrected.

[0036] S150: Using the corrected multiple prediction points, obtain a license plate area image corresponding to the license plate to be processed from the vehicle image to be detected.

[0037] The corrected multiple predicted points can be used as points representing important semantics of the license plate. The corrected multiple predicted points are used to obtain the area containing these point sets from the vehicle image to be detected as the license plate area image corresponding to the license plate to be processed.

[0038] S170: Identify the image features of the license plate area image to obtain the license plate information of the license plate to be processed.

[0039] The recognition model is used to extract features from the license plate area image and identify the license plate information of the license plate to be processed.

[0040] In this implementation, to address the difficulty of license plate detection, a representative point set representation method is used to correct multiple predicted points detected from the vehicle image to be detected to obtain a license plate area image. This method has the potential to adapt to changes in the posture or shape of the license plate. The extracted point set can be adaptively distributed in local areas of important semantic significance in the license plate and can represent the geometric extension of the license plate, thereby providing a more detailed geometric description of the license plate, such as the four corner points of the license plate, thereby improving the detection accuracy of the license plate.

[0041] In this embodiment, the vehicle image to be processed usually contains not only the vehicle, but also some background objects unrelated to license plate detection, which will cause recognition interference when directly performing license plate detection. Therefore, when obtaining a vehicle image to be detected that contains a license plate, it is necessary to perform preliminary detection on the vehicle and license plate area image to facilitate subsequent license plate recognition. The target image area in the image to be processed that contains all license plate area prediction points and whose size meets the preset requirements is determined as the license plate area image, wherein the target image area meeting the preset requirements means that the size of the target image area is smaller than the size of the vehicle image to be detected. Please refer to Figure 2 , Figure 2 This is a flow chart of a method for obtaining a vehicle image to be detected including a license plate in an embodiment of the present application. It should be noted that if there is substantially the same result, this embodiment does not use Figure 2 The process sequence shown is limited. Figure 2 As shown, this embodiment includes:

[0042] S210: Perform vehicle detection on the image to be detected containing the vehicle to obtain a vehicle area.

[0043] The acquired image to be detected containing vehicles is input into the target detection area, the number of vehicles contained in the image and the coordinates of the vehicles are detected, and the vehicle area is obtained.

[0044] S230: performing zoom processing on the vehicle area with the center point of the vehicle area as a base point to obtain an image of the vehicle area.

[0045] First, the first zoom process is performed based on the center point of the vehicle area in the image. The magnification should not be too large, such as 1.5 times, to obtain the vehicle area image. At this time, the vehicle in the image is relatively complete, but larger than the vehicle in the original image, and the license plate is usually still not clear.

[0046] S250: Perform license plate hanging position detection on the vehicle area image to obtain the license plate hanging position area.

[0047] Based on the vehicle area image, the license plate hanging position is detected, which includes the front and / or rear of the vehicle. That is, the front and / or rear detectors are used to detect the license plate hanging position on the vehicle area image to obtain the license plate hanging position area of ​​the vehicle, that is, the front and / or rear detection frame.

[0048] S270: performing zoom processing on the license plate hanging position area based on the central displacement point of the license plate hanging position area, and obtaining an image of the vehicle to be detected including the license plate.

[0049] A second zoom process is performed based on the center displacement point of the license plate hanging area of ​​the vehicle as a base point to obtain an image of the vehicle to be detected that includes the license plate. The center displacement point is a point where the center point of the license plate hanging area is displaced downward by a preset height. In one embodiment, the center displacement point is a point where the center point of the license plate hanging area is displaced downward by 1 / 4 of the height of the license plate hanging area. At this point, the license plate is clear and close to the center of the image, facilitating subsequent license plate detection and recognition.

[0050] In this embodiment, based on the difficulty in selecting the center point of the frame zoom, an improved frame zoom multiple-magnification scheme that combines the front and rear of the vehicle is proposed. First, the zoom is performed once based on the detected vehicle frame, then the front and rear of the vehicle are detected, and then the zoom is performed a second time based on the detected front (or rear) of the vehicle. This ensures that the image after zooming contains a complete license plate, and the license plate is located in the middle area as much as possible. This solves the problem that improper selection of the center point of the frame zoom may cause the license plate to be out of the picture or incomplete after zooming, improves the accuracy of the zoom, and is beneficial to subsequent license plate detection.

[0051] In this embodiment, a representative point set representation method is used to obtain a license plate area image from a vehicle image to be detected that includes a license plate. Figure 3 , Figure 3 This is a flow chart of a method for obtaining a license plate area image in an embodiment of the present application. It should be noted that if there is substantially the same result, this embodiment does not use Figure 3 The process sequence shown is limited. Figure 3 As shown, this embodiment includes:

[0052] S310: Extracting image feature information of the vehicle image to be detected.

[0053] A feature pyramid network is used as a backbone network to extract image feature information of the vehicle image to be detected to obtain image pyramid feature information. In one embodiment, five layers of image pyramid feature information are generated, and the five-layer feature pyramid network ranges from stage 3 with a downsampling ratio of 8 to stage 7 with a downsampling ratio of 128.

[0054] S330: Acquire multiple prediction points and prediction offsets corresponding to each of the multiple prediction points from the image feature information, and use the prediction offsets to correct the prediction points to obtain multiple license plate area prediction points.

[0055] In this embodiment, a set of adaptive point sets is modeled using a representative point set representation method. First, a set of initial sample points is obtained. Where n represents the total number of sample points used. In one embodiment, the total number of sample points is 9. During the training process, we continue to use the multi-stage target detection method to gradually improve the representative point set. The formula is as follows: in, is the predicted offset of each new sample point relative to each initial sample point. At the same time, in order to utilize the existing rectangular box annotation information (the annotation cost is relatively small) and evaluate the effect of the detector based on the representative point set, we use a conversion function T to convert the representative point set into a pseudo bounding box. The conversion function is: T:R p →B p , where R p is the representative point set representation of the license plate, and B p As for the specific form of the conversion function T, this embodiment uses the Min-Max Function, that is, taking the maximum and minimum x, y among all n points to obtain a rectangular box containing all sample points, which is the pseudo bounding box.

[0056] The learning of representative point sets is driven by both localization loss and recognition loss. To calculate the localization loss, we first transform the representative point set into B p Then calculate B p Compared to the standard bounding box loss (ground truth bounding box), in this implementation, the SmoothL1 loss is used for positioning loss. The improved Focal Loss is used for classification loss. Driven by both the target positioning loss and the target recognition loss, this method can automatically learn the extreme points and semantic key points of the license plate.

[0057] In the implementation of this application, an anchor-free multi-stage object detector TypDet is designed based on a representative point set. This object detector TypDet is used to iterate multiple times to obtain multiple license plate area prediction points, and then obtain the minimum area image containing all license plate area prediction points as the license plate area image. The specific implementation process of the object detector TypDet is as follows:

[0058]

[0059]

[0060] …

[0061]

[0062] Among them, the multi-stage object detector TypDet is used to perform multiple detection and improvement on the object center, and each time multiple target prediction points TypPoints proposals are obtained.

[0063] In this embodiment, a deformable convolutional network is used to implement the above-mentioned method for representing representative point sets. The deformable convolutional network is used to obtain multiple prediction points and prediction offsets corresponding to the multiple prediction points from image feature information. The prediction offset is the offset of the prediction point relative to the prediction point of the previous prediction. The prediction offset is used to correct the prediction point to obtain multiple license plate area prediction points. Figure 4 and Figure 5 , Figure 4 is a flow chart of the deformable convolutional network method in the embodiment of the present application, Figure 5 It is a structural diagram of the deformable convolutional network method in the embodiment of the present application. It should be noted that if there is substantially the same result, this embodiment does not use Figure 4 The process sequence shown is limited. Figure 4 As shown, this embodiment includes:

[0064] S410: Using the main network to process the image feature information to obtain predicted feature information.

[0065] Deformable Convolutional Networks include localization branch networks, see Figure 5 , Figure 5 The left branch is the positioning branch network. First, the image feature information of the vehicle image to be detected is extracted and input into the main network of the positioning branch network. The main network of the positioning branch network includes three consecutive 256-d 3*3 convolutional layers to perform convolution processing on the image feature information to obtain predicted feature information.

[0066] S430: Process the prediction feature information using the prediction point detection sub-network to obtain multiple prediction points.

[0067] The prediction point detection subnetwork includes a continuous first convolutional layer and a second convolutional layer to perform convolution processing on the prediction feature information to obtain information of multiple prediction points. The first convolutional layer is a 256-d 3*3 convolutional layer, and the second convolutional layer is a 256-d 1*1 convolutional layer.

[0068] S450: Processing the prediction feature information and the multiple prediction points using the offset analysis sub-network to obtain a prediction offset corresponding to each of the multiple prediction points.

[0069] The offset analysis subnetwork includes a third and fourth consecutive convolutional layers to perform convolution processing on the predicted feature information and the information of multiple prediction points to obtain the predicted offset information corresponding to the multiple prediction points. The third convolutional layer is a 256-d 3*3 convolutional layer, and the fourth convolutional layer is a 256-d 1*1 convolutional layer.

[0070] S470: Fusing the first prediction point with the corresponding prediction offset to obtain a second prediction point.

[0071] The multiple prediction points obtained by convolution processing of the image feature information are used as the first prediction point, and the first prediction point is corrected by adding the corresponding prediction offset to obtain the second prediction point. Figure 5 As shown, Figure 5 The positioning branch network calculates and outputs the second predicted point.

[0072] In one embodiment, if Figure 5 As shown in Figure 1, the object detector TypDet is a two-stage object detector. It generates the first prediction point by initializing the center point hypothesis, then generates the second prediction point by modifying the first prediction point. The second prediction point is used as the final license plate region prediction point. Thus, we have obtained an object detector based on point set representation, which is better than the previous detector based on rectangular bounding box representation for detecting license plates at large angles.

[0073] In another embodiment, the second prediction point is further calculated using the same calculation method as described above. Using the positioning branch network of the deformable convolutional network, the prediction offset corresponding to the second prediction point is calculated, and the second prediction point is used as the new first prediction point. The first prediction point and the corresponding prediction offset are added to obtain the second prediction point information, and the prediction offset corresponding to the second prediction point is calculated again until the prediction offset reaches the termination condition, thereby obtaining the license plate area prediction point. In one embodiment, the prediction offset reaches the termination condition when the number of prediction offset calculations reaches a first threshold; in another embodiment, the prediction offset reaches the termination condition when the prediction offsets corresponding to all prediction points are less than or equal to a second threshold.

[0074] In another embodiment, the deformable convolutional network further includes a classification branch network, see Figure 5 , Figure 5The right branch is the classification branch network, which shares the calculation results of the classification branch network with some positioning branch networks. The image feature information is input into the classification branch network, and the license plate category information of the license plate is obtained from the image feature information. First, the image feature information of the extracted vehicle image to be detected is input into the main network of the classification branch network. The main network of the classification branch network includes three consecutive 256-d 3*3 convolutional layers to perform convolution processing on the image feature information to obtain classification feature information. The classification feature information and the information of multiple prediction points are then input into the classification sub-network to obtain classification information. The prediction point detection sub-network includes consecutive fifth and sixth convolutional layers to perform convolution processing on the classification feature information to obtain classification information. The fifth convolutional layer is a 256-d 3*3 convolutional layer, and the sixth convolutional layer is a 256-d 1*1 convolutional layer.

[0075] In this implementation, the classification branch network operates synchronously with the localization branch network. While the localization branch network is iterating prediction points, the classification branch network is also iterating classification information. That is, while the third and fourth convolutional layers are calculating prediction offsets, the fifth and sixth convolutional layers are also calculating classification information. Ultimately, when the localization branch network terminates its calculations, the classification branch network outputs the final classification result.

[0076] After calculating the predicted points of the license plate area, obtain the smallest rectangular box containing all the predicted points, that is, take the maximum value x among all points max and xminimum value x min , y maximum value y max and y minimum value y min , with (x min ,y min ), (x max ,y min ), (x max ,y max ) and (x min ,y max ) as vertices, and the smallest rectangular frame containing all sample points is obtained, which is used as the license plate area image.

[0077] In this embodiment, in order to solve the problem of the difficulty of license plate detection, a target detector based on a representative point set representation method is proposed to obtain a license plate area image. This method has the potential to adapt to changes in the posture or shape of the license plate. The extracted point set can be adaptively distributed in the local area of ​​the license plate with important semantic meaning, and can represent the geometric extension of the license plate, thereby providing a more detailed geometric description of the license plate, such as the four corner points of the license plate, which improves the detection accuracy of the license plate. In the use of the target detector, this application uses the center point as the initial representation of the target, just like YOLO and DenseBox. The use of a center point-based method makes it easier to cover the two-dimensional space of the image. When selecting the representative point set, a deformable convolutional network with a branch network is used. While obtaining the license plate detection frame, the license plate is preliminarily classified to facilitate subsequent license plate recognition and detection.

[0078] See also Figure 6 , Figure 6 It is a panoramic view of a large truck captured by a camera in a specific embodiment of the present application. In one specific embodiment, the license plate detection method of the present application is applied to a method for capturing illegal parking of a large truck. When capturing illegal parking, a fixed-angle camera is generally used for shooting, and one or more no-parking areas are set in advance in the camera's image based on the no-parking areas. The video captured by the camera is framed, and the current frame image is used as the input image. The input image is scaled and the color space is converted, and the processed image is input into a target detector, and the target detector is used to obtain the number of vehicles in the input image and the coordinates of each vehicle. Each vehicle target is tracked to obtain the running trajectory of each vehicle.

[0079] When the trajectory of vehicle A crosses the prohibited parking area and remains stationary for more than a preset time, such as 5 seconds, it is determined that the vehicle may have illegally parked, and the frame image is cached. The camera is used to zoom in on the vehicle. However, due to the wide vehicle body and the vehicle detection frame, the center point of the camera's zoom is difficult to select. If the center point of the detection frame is directly zoomed, the license plate is likely to be out of the zoomed picture. Therefore, the above-mentioned zoom processing method of this application is used. First, the center point of the large truck's vehicle inspection frame is zoomed for the first time, but the magnification should not be too large. Please refer to Figure 7 , Figure 7 This is a schematic diagram of the result of the first frame zooming in this application. Then, the front and / or rear of the truck are detected to obtain the license plate hanging position area. Then, the center point O of the license plate hanging position area is found and displaced downward by 1 / 4 of the height of the license plate hanging position area as the center displacement point P. The license plate hanging position area is subjected to a second zooming process based on the center displacement point P of the license plate hanging position area to obtain the image of the vehicle to be detected containing the license plate. Please refer to Figure 8 , Figure 8This is a schematic diagram of the result of the second frame zooming in this application. At this time, the license plate is clear and close to the center of the picture, which is conducive to subsequent license plate detection and recognition.

[0080] Since the actual illegal parking scene is quite complex and the license plate angle is often large, the license plate detection is difficult. The above representative point set representation method is used to extract the semantically important point set on the large truck license plate, and the point set is used to obtain the rectangular bounding box containing the license plate. Figure 9 , Figure 9 is a schematic diagram of the license plate detection frame obtained by the representative point set of this application. In this embodiment, 9 sample points are used, such as Figure 9 As shown, nine predicted points are found around the license plate, and the smallest rectangular box containing the nine predicted points is obtained as the license plate detection frame. The recognition model is used to identify the license plate within the rectangular bounding box, obtaining its license plate information, and obtaining the first license plate information, plate_A. After recognition is completed, the camera returns to the preset position.

[0081] The camera preset position continuously detects and tracks the vehicle. If vehicle A remains stationary for a period exceeding the preset illegal parking time, such as 2 minutes, the vehicle is judged to be illegally parked and the image is cached. The camera takes a zoom close-up of the vehicle again, detects the vehicle and identifies the license plate, obtains its license plate information, and records it as the second license plate information plate_B. If the first license plate information plate_A and the second license plate information plate_B are consistent, it is determined that the license plates are exactly the same, and the evidence chain of illegal parking is reported, and the camera returns to the preset position; if the first license plate information plate_A and the second license plate information plate_B differ by 1 digit, the license plate is identified again and recorded as the third license plate information plate_C. The first license plate information plate_A, the second license plate information plate_B, and the third license plate information plate_C are voted, and the license plate information with the most votes is selected as the license plate identification result and output, and the evidence chain of illegal parking is reported, and the camera returns to the preset position; if the first license plate information plate_A and the second license plate information plate_B differ by more than or equal to 2 digits, it is determined that they are not the same vehicle, and it is determined that vehicle A has left and vehicle B has newly appeared. Its parking time information is updated, the camera returns to the preset position, and the above steps are repeated.

[0082] In this implementation, a complete set of illegal parking capture solutions is proposed for the situation where large trucks park illegally. Since large trucks have wider bodies and larger vehicle detection frames, it is difficult to select the center point of the frame zoom. When zooming in close-up, the angle of the license plate is also large, which makes it easy for the license plate to not be detected or to be detected inaccurately. An improved frame zoom multiple-zoom solution that combines the front and rear of the vehicle is proposed. First, the zoom is performed once based on the detected vehicle frame, and then the front and rear of the vehicle are detected. Then, the zoom is performed a second time based on the detected front (or rear) of the vehicle, so that the complete license plate is included in the zoomed picture, and the license plate is located in the middle area as much as possible. This solves the problem that improper selection of the center point of the frame zoom may cause the license plate to be out of the picture or incomplete after the zoom, improves the accuracy of the zoom, and is beneficial to subsequent license plate detection. At the same time, to address the difficulty of detecting license plates at large angles, a target detector based on a representative point set representation method is proposed to obtain license plate area images. This method has the potential to adapt to changes in the posture or shape of the license plate. The extracted point set can be adaptively distributed in local areas of the license plate with important semantic meaning and can represent the geometric extension of the license plate, thereby providing a more detailed geometric description of the license plate, such as the four corner points of the license plate, and improving the detection accuracy of the license plate. When selecting the representative point set, a deformable convolutional network with a branch network is used to obtain the license plate detection frame and perform preliminary classification of the license plate, facilitating subsequent license plate recognition and detection. This improves the detection accuracy of license plates at large angles.

[0083] See also Figure 10 , Figure 10 1 is a schematic diagram of the structure of the license plate detector in the embodiment of the present application. In this embodiment, the license plate detector includes an acquisition module 101, a detection module 102, a correction module 103 and a recognition module 104.

[0084] The acquisition module 101 is used to acquire an image of a vehicle to be detected that includes a license plate; the detection module 102 is used to correct multiple predicted points detected from the image; the correction module 103 is used to use the corrected predicted points to acquire a license plate region image corresponding to the license plate to be processed from the image; and the recognition module 104 is used to identify image features of the license plate region image to obtain license plate information for the license plate to be processed. This license plate detector uses a representative point set representation method to acquire a license plate region image. This method has the potential to adapt to changes in the posture or shape of the license plate. The extracted point set can be adaptively distributed in semantically important local areas of the license plate and can represent the geometric extension of the license plate, thereby providing a more detailed geometric description of the license plate, such as the four corner points of the license plate, and improving the detection accuracy of the license plate.

[0085] See also Figure 11 , Figure 111 is a schematic diagram of the structure of the license plate detection device in the embodiment of the present application. In this embodiment, the license plate detection device 11 includes a processor 12.

[0086] The processor 12 may also be referred to as a CPU (Central Processing Unit). The processor 12 may be an integrated circuit chip having signal processing capabilities. The processor 12 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or the processor 12 may be any conventional processor.

[0087] The license plate detection device 11 may further include a memory (not shown in the figure) for storing instructions and data required for the processor 12 to operate.

[0088] The processor 12 is configured to execute instructions to implement the method provided by any embodiment of the license plate detection method of the present application and any non-conflicting combination thereof.

[0089] See also Figure 12 , Figure 12 : is a schematic diagram of the structure of the computer-readable storage medium in the embodiment of the present application. The computer-readable storage medium 21 of the embodiment of the present application stores instructions / program data 22, and when the instructions / program data 22 are executed, the method provided by any embodiment of the license plate detection method of the present application and any non-conflicting combination is implemented. Among them, the instructions / program data 22 can form a program file and be stored in the above-mentioned storage medium 21 in the form of a software product, so that a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) executes all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium 21 includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0091] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0092] The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A license plate detection method, characterized in that: The method comprises: Acquiring an image of a vehicle to be detected containing a license plate to be processed, comprising: performing vehicle detection on the image of the vehicle to be detected containing the vehicle to obtain a vehicle area; performing zoom processing on the vehicle area with a center point of the vehicle area as a base point to obtain a vehicle area image; performing license plate hanging position detection on the vehicle area image to obtain a license plate hanging position area, wherein the license plate hanging position includes the front and / or rear of the vehicle; performing zoom processing on the license plate hanging position area with a center displacement point of the license plate hanging position area as a base point to obtain the image of the vehicle to be detected containing the license plate; the center displacement point is a point where the center point of the license plate hanging position area is displaced downward by a preset height; Correcting a plurality of predicted points detected from the image of the vehicle to be detected, the predicted points being associated with the license plate to be detected; Using the corrected multiple prediction points, obtaining a license plate area image corresponding to the license plate to be processed from the vehicle image to be detected; Identifying image features of the license plate area image to obtain license plate information of the license plate to be processed; The step of correcting the plurality of predicted points detected from the image of the vehicle to be detected includes: Extracting image feature information of the vehicle image to be detected; Obtaining the plurality of prediction points and a prediction offset corresponding to each of the plurality of prediction points from the image feature information, wherein the prediction offset is an offset of the prediction point relative to a prediction point predicted in a previous prediction; The prediction point is corrected using the prediction offset to obtain multiple license plate area prediction points, including: fusing the first prediction point with the corresponding prediction offset to obtain a second prediction point; calculating the prediction offset corresponding to the second prediction point; taking the second prediction point as a new first prediction point, executing the steps of fusing the first prediction point with the corresponding prediction offset to obtain second prediction point information, and calculating the prediction offset corresponding to the second prediction point, until the prediction offset corresponding to the second prediction point reaches a termination condition, thereby obtaining the multiple license plate area prediction points, wherein the prediction offset reaches the termination condition when the number of times the prediction offset is calculated reaches a first threshold or the prediction offsets corresponding to all prediction points are less than or equal to a second threshold.

2. The license plate detection method according to claim 1, characterized in that: The method of obtaining a license plate area image corresponding to the license plate to be processed from the image of the vehicle to be detected by using the corrected multiple prediction points includes: The target image area in the image to be processed that contains all the license plate area prediction points and whose size meets the preset requirements is determined as the license plate area image, wherein: The size of the target image area is smaller than the size of the image of the vehicle to be detected.

3. The license plate detection method according to claim 1, characterized in that: The acquiring the plurality of prediction points and the prediction offset corresponding to each of the plurality of prediction points from the image feature information includes: Inputting the image feature information into a deformable convolutional network; the deformable convolutional network includes a main network, a prediction point detection subnetwork, and an offset analysis subnetwork; Processing the image feature information using the main network to obtain predicted feature information; Processing the prediction feature information using the prediction point detection subnetwork to obtain the multiple prediction points; The offset analysis subnetwork is used to process the prediction feature information and the multiple prediction points to obtain a prediction offset corresponding to each prediction point in the multiple prediction points.

4. The license plate detection method according to claim 3, characterized in that: The main network includes three consecutive convolutional layers to perform convolution processing on the image feature information to obtain the predicted feature information; The predicted point detection subnetwork includes a first convolutional layer and a second convolutional layer that are continuous to perform convolution processing on the predicted feature information to obtain the multiple predicted points, wherein the first convolutional layer is a 3*3 convolutional layer and the second convolutional layer is a 1*1 convolutional layer; The offset analysis subnetwork includes a third convolutional layer and a fourth convolutional layer, wherein the third convolutional layer is a 3*3 convolutional layer and the fourth convolutional layer is a 1*1 convolutional layer.

5. The license plate detection method according to claim 3, characterized in that: The deformable convolutional network also includes a classification branch network, and the method further includes: The image feature information is input into the classification branch network, and the license plate category information of the license plate is obtained from the image feature information.

6. The license plate detection method according to claim 1, characterized in that: The extracting of image feature information of the vehicle image to be detected comprises: A feature pyramid network is used to extract image feature information of the vehicle image to be detected to obtain image pyramid feature information.

7. A license plate detector, characterized in that: The license plate detector comprises: An acquisition module is used to acquire an image of a vehicle to be detected that includes a license plate to be processed, comprising: performing vehicle detection on the image of the vehicle to be detected that includes the vehicle to obtain a vehicle area; performing zoom processing on the vehicle area with a center point of the vehicle area as a base point to obtain a vehicle area image; performing license plate hanging position detection on the vehicle area image to obtain a license plate hanging position area, wherein the license plate hanging position includes the front and / or rear of the vehicle; performing zoom processing on the license plate hanging position area with a center displacement point of the license plate hanging position area as a base point to obtain the image of the vehicle to be detected that includes the license plate; the center displacement point is a point that is displaced downward by a preset height from the center point of the license plate hanging position area; a detection module, configured to correct a plurality of predicted points detected from the image of the vehicle to be detected, the predicted points being associated with the license plate to be detected; A correction module, configured to obtain a license plate area image corresponding to the license plate to be processed from the image of the vehicle to be detected using the corrected multiple prediction points; A recognition module, configured to recognize image features of the license plate area image and obtain license plate information of the license plate to be processed; The detection module is configured to correct the plurality of predicted points detected from the image of the vehicle to be detected, including: Extracting image feature information of the vehicle image to be detected; Obtaining the plurality of prediction points and a prediction offset corresponding to each of the plurality of prediction points from the image feature information, wherein the prediction offset is an offset of the prediction point relative to a prediction point predicted in a previous prediction; The prediction point is corrected using the prediction offset to obtain multiple license plate area prediction points, including: fusing the first prediction point with the corresponding prediction offset to obtain a second prediction point; calculating the prediction offset corresponding to the second prediction point; taking the second prediction point as a new first prediction point, executing the steps of fusing the first prediction point with the corresponding prediction offset to obtain second prediction point information, and calculating the prediction offset corresponding to the second prediction point, until the prediction offset corresponding to the second prediction point reaches a termination condition, thereby obtaining the multiple license plate area prediction points, wherein the prediction offset reaches the termination condition when the number of times the prediction offset is calculated reaches a first threshold or the prediction offsets corresponding to all prediction points are less than or equal to a second threshold.

8. A license plate detection device, characterized in that: The method comprises a processor configured to execute instructions to implement the license plate detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store instructions / program data, and the instructions / program data can be executed to implement the license plate detection method according to any one of claims 1 to 6.

Images