Vehicle license plate recognition method, electronic equipment and storage medium

Abstract

The invention discloses a vehicle license plate recognition method, electronic equipment and a storage medium. The vehicle license plate recognition method provided by the invention comprises the following steps: processing a vehicle video stream to obtain a plurality of license plate information in each frame of image; based on a target tracking model, processing the information of the plurality of license plates in each frame of image in sequence to obtain license plate tracks of the plurality of license plates; and determining a final license plate number of the license plate based on the license plate track. According to the scheme, the license plate of the vehicle far away from the camera is recognized, and the license plate recognition effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle data processing, and in particular to a vehicle license plate recognition method, electronic equipment and storage medium. Background Art

[0002] License plate capture technology is an important part of the intelligent transportation system. Its main purpose is to automatically identify and record the license plate information of passing vehicles through roadside high-definition cameras and image processing technology. This technology plays an important role in traffic monitoring, illegal capture, accident investigation, and illegal vehicle identification.

[0003] When using roadside cameras for license plate recognition, due to the limited coverage of a single camera, vehicles that are far away from the camera cannot be recognized. Summary of the invention

[0004] In view of this, the present invention proposes a vehicle license plate recognition method, electronic device and storage medium, which solves the problem in the related art that, during the process of recognizing license plates by cameras, the camera has a small coverage range and cannot recognize the license plates of vehicles far away from the camera.

[0005] On the one hand, an embodiment of the present invention provides a vehicle license plate recognition method, comprising: Process the vehicle video stream to obtain several license plate information in each frame image; Based on the target tracking model, the license plate information in each frame of the image is processed in turn to obtain the license plate trajectories of the license plates; Based on the license plate trajectory of the license plate, the final license plate number of the license plate is determined.

[0006] In some implementations, based on the target tracking model, the plurality of license plate information in each frame of the image is processed in sequence to obtain the license plate trajectories of the plurality of license plates, including performing the following steps based on the target tracking model: Construct the cost matrix of the Hungarian algorithm based on the license plate information in the current frame and several previous frames; The cost matrix is ​​optimized using the Hungarian algorithm to obtain the target matching solution; Update existing license plate tracks or add new license plate tracks based on the target matching solution.

[0007] In some implementations, a cost matrix of the Hungarian algorithm is constructed based on the license plate information in the current frame and the previous frames thereof, including: Based on the Kalman filter algorithm, according to the license plate information in the current frame image and the license plate information in the previous frame image of the current frame image, the license plate prediction information of the current frame image is obtained; According to some license plate prediction information and some license plate information of the current frame image, the cost matrix of the Hungarian algorithm is constructed.

[0008] In some implementations, based on the Kalman filter algorithm, based on the license plate information in the current frame image and the license plate information in the previous frame image of the current frame image, the license plate prediction information of the current frame image is obtained, including: Based on the Kalman filter algorithm, according to the license plate information in the current frame image and the license plate information in the previous frame image of the current frame image, several license plate prediction frames of the current frame image are obtained, as well as the overlap degree between each license plate detection frame obtained according to the current frame image and each license plate prediction frame in the several license plate prediction frames.

[0009] In some implementations, a cost matrix of the Hungarian algorithm is constructed based on a plurality of license plate prediction information and a plurality of license plate information of the current frame image, including: Based on the spatial distance algorithm, the spatial distance between the license plate detection frame and the license plate prediction frame is obtained according to the overlap between the license plate detection frame and the license plate prediction frame in the license plate information; According to the license plate feature vector in the license plate information corresponding to the license plate detection frame and the license plate feature vector in the license plate information corresponding to the license plate prediction frame, the similarity of the license plate feature vector is obtained; Based on the obtained spatial distances of several license plate frames and the similarities of several license plate feature vectors, the cost matrix of the Hungarian algorithm is constructed.

[0010] In some implementations, updating an existing license plate track or adding a new license plate track based on a target matching solution includes: Based on the first preset condition, bad matches are eliminated from the target matching scheme, and vehicle information corresponding to the eliminated bad matches is added to the unmatched set, and license plate information corresponding to the remaining matches in the target matching scheme is added to the matched set; Filtering out license plate information that meets a second preset condition from all license plate information in the unmatched set, and adding the license plate information that meets the second preset condition to the matching set; Based on all the remaining license plate information in the matching set, the corresponding license plate trajectory in the existing license plate trajectory is updated, and based on all the license plate information in the unmatched set, the corresponding license plate trajectory is added.

[0011] In some embodiments, the first preset condition includes that the similarity between the two license plate information is lower than a first threshold; the second preset condition includes that the license plate feature lengths in the two license plate information are the same and the license plate feature similarity is greater than a second threshold.

[0012] In some implementations, the vehicle video stream is processed to obtain a number of license plate information in each frame of the image, including: Based on the target detection model, each frame of the vehicle video stream is detected to obtain the information of several license plate detection frames in each frame; Based on the license plate recognition model, the license plate detection frame information is detected to obtain the license plate number and license plate feature vector of the license plate detection frame.

[0013] In some implementations, determining a final license plate number of a license plate based on a license plate trajectory of the license plate includes: When all the license plate numbers in the license plate trajectory are the same, the final license plate number of the license plate is obtained based on any license plate number in the license plate trajectory; When all license plate numbers in the license plate trajectory are not exactly the same, the final license plate number of the license plate is obtained based on the license plate detection frame information and the license plate feature vector of each track point in the license plate trajectory.

[0014] In some implementations, when all license plate numbers in the license plate trajectory are not exactly the same, obtaining the final license plate number of the license plate based on the license plate detection frame information and the license plate feature vector of each track point in the license plate trajectory includes: Perform weighted calculation on all license plate detection frame information and all license plate feature vectors in the license plate trajectory to obtain a weighted license plate feature vector; The weighted license plate feature vector is calculated according to the normalized exponential function to obtain the final license plate number.

[0015] In some implementations, after obtaining the final license plate number of the license plate, the method further includes: Update all license plate numbers in the corresponding license plate trajectory according to the final license plate number of the license plate.

[0016] On the other hand, an embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory, the memory storing a computer program that can be run on the processor, characterized in that when the processor executes the program, the steps of the method of any of the above embodiments are performed.

[0017] On the other hand, an embodiment of the present invention further provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of any of the above embodiments are executed.

[0018] The present invention has at least the following beneficial effects: The present invention provides a vehicle license plate recognition method, electronic equipment and storage medium. The vehicle license plate recognition method provided by the present invention can process a plurality of license plate information in each frame image in sequence through a target tracking model, fuse the license plate information in multiple frames of images, obtain a series of license plate trajectories, and determine the final license plate number of the license plate through the obtained license plate trajectories, thereby realizing the recognition of the vehicle license plate far away from the camera and improving the effect of license plate recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flow chart of a vehicle license plate recognition method provided by an embodiment of the present invention; Figure 2 A flowchart of performing license plate tracking in a license plate recognition method provided by an embodiment of the present invention; Figure 3a A flowchart of constructing a cost matrix of the Hungarian algorithm in a license plate recognition method provided by an embodiment of the present invention; Figure 3b A flow chart of generating a license plate trajectory in a license plate recognition method provided by an embodiment of the present invention; Figure 4 A flow chart of a vehicle license plate recognition method provided by an embodiment of the present invention; Figure 5 A flowchart of a license plate tracking method in a license plate recognition method provided by an embodiment of the present invention; Figure 6 A schematic diagram of a minimum circumscribed rectangle of two license plate detection frames provided in an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a computer storage medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the embodiments of the present invention are further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0022] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0023] License plate capture technology is an important part of the intelligent transportation system. Its main purpose is to automatically identify and record the license plate information of passing vehicles through roadside high-definition cameras and image processing technology. This technology plays an important role in traffic monitoring, illegal capture, accident investigation, and illegal vehicle identification.

[0024] When using roadside cameras for license plate recognition, due to the limited coverage of a single camera, the license plates of vehicles that are far away from the camera cannot be recognized due to the small number of license plate pixels and tilted license plates, resulting in license plate recognition errors and partial loss of license plate tracks.

[0025] In view of this, in order to solve at least one of the above technical problems, an embodiment of the present invention provides a vehicle license plate recognition method. The present invention is described in detail below in conjunction with the embodiments and drawings.

[0026] A first aspect of an embodiment of the present invention provides a vehicle license plate recognition method, such as Figure 1 As shown, the method may include steps S10 to S12.

[0027] S10, processing the vehicle video stream to obtain a number of license plate information in each frame image.

[0028] The vehicle video stream may be a video stream collected by a roadside camera. Each frame of the video stream may include a license plate of a vehicle or multiple license plates of multiple vehicles according to the actual road conditions. No specific limitation is made here.

[0029] The license plate information may include: license plate identification, license plate detection frame, license plate number, license plate number length, license plate features, license plate feature length and other information.

[0030] Each license plate may be configured with a unique identifier, through which a series of license plate trajectories of the license plate may be tracked.

[0031] S11. Based on the target tracking model, the license plate information in each frame of the image is processed in turn to obtain the license plate trajectories of the license plates.

[0032] The target tracking model refers to the improved Sort (Simple Online and Realtime Tracking) model.

[0033] In the license plate tracking related technology, the Sort model can be used to process each frame of images collected in real time by roadside equipment to obtain the license plate tracks of several license plates. However, when the Sort model is used to process each frame of images, for images collected by roadside equipment that are far away from the roadside equipment, the Sort model cannot recognize the license plates in such images due to the small number of license plate pixels in such images and the tilted license plate viewing angle.

[0034] Therefore, based on the Sort model, the present invention improves the image processing flow of the Sort model to obtain an improved Sort model.

[0035] The vehicle video stream is processed, and the obtained license plate information in each frame image is input into the improved Sort model in sequence. Each license plate information includes license plate detection frame information and license plate feature information. The license plate trajectories of several license plates can be obtained by processing the input license plate detection frame information and license plate feature information through the improved Sort model.

[0036] The license plate trajectory may include the number of license plate detections, license plate detection frame information, license plate number information, the time of each detection, and other information.

[0037] In a specific implementation, a new license plate track can be generated or an existing license plate track can be updated by sequentially detecting a number of license plate information in each frame of the image through an improved Sort model.

[0038] In the process of detecting several license plate information in the current frame image through the target tracking model, generating a new license plate track or updating the existing license plate track, if the license plate corresponding to the detected license plate information has appeared before, a new track point can be added to the corresponding license plate track based on the license plate identifier in the license plate information (i.e., updating the existing license plate track), and the newly added track point information can include: license plate frame information, license plate feature vector, license plate number, detection time of this detection, etc. If the license plate corresponding to the detected license plate information has not appeared before, a new license plate track is initialized based on the license plate identifier in the license plate information (i.e., generating a new license plate track), the license plate identifier in the license plate information is used as the identifier of the new license plate track, and the license plate information detected this time is used as the information of the first license plate track point in the newly added license plate track.

[0039] S12. Determine the final license plate number of the license plate based on the license plate trajectory of the license plate.

[0040] The final license plate number of the vehicle can be determined based on whether all the license plate numbers in the license plate trajectory are consistent over a period of time. If all the license plate numbers in the license plate trajectory are consistent over a period of time, any license plate number among all the license plate numbers can be used as the final license plate number of the vehicle. If all the license plate numbers in the license plate trajectory are not completely consistent over a period of time, a weighted algorithm can be used to calculate all the license plate information in the license plate trajectory to determine the final license plate number of the vehicle.

[0041] The scheme of the present invention processes a number of license plate information in each frame image in turn through a target tracking model to obtain a series of license plate trajectories. The final license plate number of the license plate is determined through the obtained license plate trajectories, thereby realizing the recognition of vehicle license plates that are far away from the camera and improving the effect of license plate recognition.

[0042] In some embodiments of the present invention, Figure 1 The step S10 shown, processing the vehicle video stream to obtain a number of license plate information in each frame image, may include: based on the target detection model, detecting each frame image in the vehicle video stream to obtain a number of license plate detection frame information in each frame image; based on the license plate recognition model, detecting the license plate detection frame information to obtain the license plate number and license plate feature vector of the license plate detection frame.

[0043] The target detection model may be a YOLO, SSD or other model, but is not limited thereto, and may also be other models for detecting targets. The target detection model may be used to identify the license plate frame information of each frame in the video stream. The license plate frame information may include the position and size information of the license plate frame.

[0044] The license plate recognition model can be a model such as YOLO, CNN, PaddleOCR, etc., but is not limited to these, and can also be other models for recognizing license plate numbers.

[0045] Through the license plate recognition model, the license plate numbers in each license plate frame output by the target detection model can be identified. At the same time, the input vector (i.e., the license plate feature vector) of the last loss layer (e.g., the softmax layer) in the license plate recognition model is saved. . Among them, i is the number of characters in the license plate, which is generally 7-8, and j is the total number of character categories in the license plate.

[0046] In some embodiments of the present invention, Figure 1 Step S11 shown in the figure, based on the target tracking model, sequentially processes the license plate information in each frame image to obtain the license plate tracks of the license plates, which may include executing the following steps based on the target tracking model: Figure 2 Steps S20~S22 are shown.

[0047] S20, constructing a cost matrix of the Hungarian algorithm based on the license plate information in the current frame and the previous frames.

[0048] The Hungarian algorithm is a classic combinatorial optimization algorithm used to solve the allocation problem, that is, how to optimally allocate tasks among multiple tasks and multiple agents to minimize the total cost.

[0049] The cost matrix may be an n×m matrix, wherein each element may be used to represent the cost of assigning the nth license plate to the mth license plate.

[0050] S21. Use the Hungarian algorithm to optimize the cost matrix and obtain the target matching solution.

[0051] By optimizing the cost matrix through the Hungarian algorithm, the allocation scheme with the minimum cost (i.e. the target allocation method) can be found.

[0052] The target matching scheme may be several license plate groups, each of which may be two license plate frames, or two license plate frames + two license plate numbers, which are not specifically limited here.

[0053] S22: Update the existing license plate trajectory or add a new license plate trajectory based on the target matching solution.

[0054] According to preset conditions, license plate frames that do not meet the preset conditions can be screened out from the target matching scheme, the existing license plate trajectory can be updated based on the license plate information corresponding to the remaining license plate frames in the target matching scheme, and a new license plate trajectory can be generated based on the license plate information corresponding to the license plate frames that do not meet the preset conditions.

[0055] In an embodiment of the present invention, a cost matrix of the Hungarian algorithm is constructed through information of several license plates in a current frame and several previous frames of images, and the cost matrix is ​​optimized using the Hungarian algorithm to obtain a target matching solution, and an existing license plate trajectory is updated or a new license plate trajectory is added based on the target matching solution, so that the recognition of license plate information in images at a long distance from the roadside equipment can be realized, thereby improving the license plate recognition effect.

[0056] In some embodiments of the present invention, Figure 2 Step S20 shown in the figure constructs a cost matrix of the Hungarian algorithm based on the license plate information in the current frame and the previous frames, which may include: Figure 3a Steps S31 and S32 are shown.

[0057] S31. Based on the Kalman filter algorithm, obtain some license plate prediction information of the current frame image according to some license plate information in the current frame image and some license plate information in the previous frame image of the current frame image.

[0058] S32. Construct a cost matrix of the Hungarian algorithm based on a plurality of license plate prediction information and a plurality of license plate information of the current frame image.

[0059] In some embodiments of the present invention, Figure 3a The step S31 shown, based on the Kalman filter algorithm, obtains some license plate prediction information of the current frame image according to some license plate information in the current frame image and some license plate information in the frame image before the current frame image, and may include: based on the Kalman filter algorithm, according to some license plate information in the current frame image and some license plate information in the frame image before the current frame image, obtains some license plate prediction frames of the current frame image, and the overlap degree of each license plate detection frame obtained according to the current frame image with each license plate prediction frame in some license plate prediction frames.

[0060] Specifically, the Kalman filter parameters can be first configured based on the license plate frame information obtained from the license plate information in the current frame image; based on the Kalman filter algorithm after the parameters are configured, the license plate information in the frame image before the current frame image is calculated to obtain the license plate prediction frames of the current frame image. Then, based on the Kalman filter algorithm after the parameters are configured, the overlap degree of each license plate detection frame in the license plate detection frames and each license plate prediction frame in the license plate prediction frames is calculated to obtain the overlap degree (which can be expressed by IOU) between each license plate detection frame in the license plate detection frames and each license plate prediction frame in the license plate prediction frames.

[0061] In some embodiments of the present invention, Figure 3a The step S32 shown, constructing the cost matrix of the Hungarian algorithm according to several license plate prediction information and several license plate information of the current frame image, may include: based on the spatial distance algorithm, obtaining the license plate frame spatial distance between the license plate detection frame and the license plate prediction frame according to the overlap between the license plate detection frame in the license plate information and the license plate prediction frame; obtaining the license plate feature vector similarity according to the license plate feature vector in the license plate information corresponding to the license plate detection frame and the license plate feature vector in the license plate information corresponding to the license plate prediction frame; constructing the cost matrix of the Hungarian algorithm based on the obtained several license plate frame spatial distances and several license plate feature vector similarities.

[0062] Specifically, the cost matrix of the Hungarian algorithm can be constructed through several license plate frame spatial distances and several license plate feature similarities. Each element in the cost matrix includes a license plate frame spatial distance and a license plate feature similarity.

[0063] According to the similarity of the license plate feature vectors between the license plate feature vector corresponding to the license plate detection frame and the license plate feature vector corresponding to the license plate prediction frame, the similarity of the license plate features of the two license plate frames can be evaluated, thereby improving the accuracy of subsequent license plate tracking.

[0064] Through the spatial distance algorithm, for example, GIOU can calculate the spatial distance between the license plate detection frame and the license plate prediction frame, thereby evaluating the spatial distance between the two license plate frames.

[0065] The solution of the present invention calculates the spatial distance of the license plate frame and the similarity of the license plate features, and constructs the cost matrix of the Hungarian algorithm through the calculated spatial distance of the license plate frame and the similarity of the license plate features, thereby improving the accuracy of license plate recognition and improving the recognition effect of license plates in images at a long distance from roadside equipment.

[0066] In some embodiments of the present invention, Figure 2 Step S20 shown in the figure, updating the existing license plate track or adding a new license plate track based on the target matching scheme, may include: Figure 3b Steps S301~S303 shown.

[0067] S301. Eliminate bad matches from the target matching solution based on a first preset condition, add vehicle information corresponding to the eliminated bad matches to an unmatched set, and add license plate information corresponding to the remaining matches in the target matching solution to a matching set.

[0068] The first preset condition is used to determine whether the license plate group in the target matching scheme is a bad match, and the first preset condition may include that the similarity of the two license plate information is lower than the first threshold. For example, the first preset condition may be that the similarity of the two license plate detection frames is lower than the first threshold. For example, the first preset condition may also be that the similarity of the two license plate features is lower than the first threshold. Among them, the license plate feature may be a license plate feature vector, that is, the first preset condition may also be that the similarity of the two license plate feature vectors is lower than the first threshold.

[0069] The first threshold can be set according to the actual application scenario, for example, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8, etc., and is not specifically limited here.

[0070] A bad match can be understood as the low similarity between the two license plate information in the license plate group.

[0071] The matching set can be used to store license plate information. The license plate information stored in the matching set can be the license plate information corresponding to the license plate group after the bad matches are eliminated in the target matching scheme. The license plate information stored in the matching set can be the license plate number information, or the license plate frame information and the license plate number information, which are not specifically limited here. The matching set is used to update the license plate track.

[0072] The unmatched set can also be used to store license plate information. The license plate information stored in the unmatched set can be the license plate information corresponding to the license plate group in the bad match. The license plate information stored in the unmatched set can be license plate number information, or license plate frame information and license plate number information, which is not specifically limited here.

[0073] The unmatched set can be used to update the license plate trajectory, or to generate a new license plate trajectory, which is not specifically limited here.

[0074] In some examples, there are multiple license plate groups in the target matching scheme, but the similarity between the two matching license plates in some license plate groups is low, then the license plate information corresponding to the license plate groups with low similarity is deleted from the matching set.

[0075] S302: Filter out license plate information that meets a second preset condition from all license plate information in the unmatched set, and add the license plate information that meets the second preset condition to the matched set.

[0076] In a specific implementation, the second preset condition may include: the lengths of the license plate features in the two license plate information are the same, and the similarity of the license plate features is greater than a second threshold. For example, the second preset condition may be that the length of the license plate feature vector of one license plate is the same as the length of the license plate feature vector of the other license plate, and the similarity of the license plate feature vector of one license plate to the license plate feature vector of the other license plate is greater than a second threshold.

[0077] The second threshold can be set according to the actual application scenario, for example, 0.8, 0.85, 0.9, 0.95, etc., and is not specifically limited here.

[0078] Through the second preset condition, a type of license plates with equal license plate feature vector lengths and high license plate feature vector similarity in the bad matches can be screened out, and the license plate information corresponding to the type of license plates is added to the matching set.

[0079] S303: Update the corresponding license plate track in the existing license plate track based on all the remaining license plate information in the matching set, and add the corresponding license plate track based on all the license plate information in the unmatched set.

[0080] The solution of the present invention can reduce the probability of mismatching between license plates and improve the license plate tracking effect.

[0081] In some embodiments of the present invention, Figure 3b As shown, the target tracking method of the present invention may further include step S304 in addition to steps S301 to S303.

[0082] S304: In response to the presence of license plate information in the matching set without being updated for more than a preset period of time, the license plate information is deleted from the matching set.

[0083] If the license plate information in the matching set has not been updated for more than a preset period of time, it means that the vehicle may have driven out of the observation area. The license plate information can be deleted from the matching set, and the matching set can be updated. This reduces the amount of calculation in the license plate tracking process, reduces computing memory, and improves computing efficiency.

[0084] In some embodiments of the present invention, Figure 1 The step S12 shown, determining the final license plate number of the license plate based on the license plate trajectory of the license plate, may include: when all license plate numbers in the license plate trajectory are the same, obtaining the final license plate number of the license plate based on any license plate number among all the license plate numbers; when all license plate numbers in the license plate trajectory are not exactly the same, obtaining the final license plate number of the license plate based on the license plate detection frame information and the license plate feature vector of each trajectory point in the license plate trajectory.

[0085] In some examples, Figure 1 Step S11 shown in the figure obtains a series of license plate tracks , where each license plate track has a unique identifier , Specifically includes: License plate detection frame information: ,in Indicates The bounding box coordinates of the detection. License plate number: ,in Indicates The license plate number of the detection. Detection timestamp: ,in Indicates The timestamp of the detection. The license plate recognition model outputs a vector: ,in For the The model output vector for each detection.

[0086] If the trajectory All license plate numbers satisfy , the final license plate number of the license plate recognition can be output .

[0087] If the trajectory There is a license plate number in If other license plate numbers are not satisfied, then each track point in the track of the license plate and , calculate the final license plate number of the license plate.

[0088] In some embodiments of the present invention, when all license plate numbers in a license plate trajectory are not exactly the same, obtaining the final license plate number of the license plate based on the license plate detection frame information and the license plate feature vector of each trajectory point in the license plate trajectory may include: performing weighted calculation on all license plate detection frame information and all license plate feature vectors in the license plate trajectory to obtain a weighted license plate feature vector; calculating the weighted license plate feature vector according to a normalized exponential function to obtain the final license plate number of the license plate.

[0089] In some examples, if the trajectory There is a license plate number in If other license plate numbers are not satisfied, then each track point in the track of the license plate and , then get The license plate recognition model output vector in and Perform weighted calculation to obtain the weighted license plate feature vector The calculation method is:

[0090] in, for The pixel area, Fill the license plate set with license plates of different lengths The maximum length of the license plate, and fill the missing parts with 0.

[0091] right Execute the normalized exponential function (also known as the softmax function) and output the final license plate number .

[0092]

[0093] The solution of the present invention realizes the recognition of the vehicle license plate which is far away from the camera, and improves the accuracy of the license plate number recognition.

[0094] In some embodiments of the present invention, after the final license plate number of the license plate is obtained, all license plate numbers in the corresponding license plate track may be updated according to the final license plate number of the license plate.

[0095] Specifically, you can All license plates assigned to this tracking id, i.e. .

[0096] The embodiments of the present invention are described in detail below through specific examples.

[0097] Example 1: like Figure 4As shown, the vehicle license plate recognition method provided by the embodiment of the present invention may include: S40: Obtain license plate frame information.

[0098] The target detection model is used to identify the license plate frame information of each frame from the video stream.

[0099] S41. Obtaining license plate number information and license plate feature vector according to license plate frame information.

[0100] The license plate recognition model can identify the license plate number information in the license plate frame. At the same time, the input vector of the last softmax (activation function) layer in the model can also be saved. .

[0101] S42. Tracking the license plate according to the license plate information through the target tracking model.

[0102] All license plate information obtained by a roadside camera within a period of time (for example, 1 minute) can be obtained, such as license plate detection frame information, license plate number, detection timestamp, license plate feature vector and other information.

[0103] S43, judging whether the license plate number under the same track id is unique.

[0104] S44. If the license plate number under the same trajectory id is unique, output the license plate number.

[0105] S45. If the license plate number under the same trajectory ID is not unique, a weighted sum is performed on the license plate feature vector according to the size of the license plate detection frame.

[0106] S46. After executing the softmax function on the weighted sum result, the obtained license plate number is output.

[0107] Example 2: like Figure 5 As shown, the license plate tracking method provided by the embodiment of the present invention may include: S50, setting Kalman filter parameters according to the license plate status information, and initializing the Hungarian algorithm.

[0108] Kalman filter parameters can include state variables ,in, Respectively represent the center point position of the target box coordinate, represents the area of ​​the target box, Indicates the aspect ratio of the target box. Respectively represent horizontal , Vertical ,area The rate of change of movement.

[0109] Initializing the Hungarian algorithm parameters may include a threshold of the cost matrix. For example, the threshold of the cost matrix may include a spatial distance threshold of two license plate frames. Similarity threshold with license plate feature vector For example, you can set ,set up .

[0110] S51. For each frame of image, obtain the license plate detection frame information and the license plate feature matrix corresponding to the vehicle detection frame.

[0111] Get the boundaries of the license plate detection frame The license plate feature matrix corresponding to the vehicle detection frame .in, is the coordinate of the center point of the license plate detection frame, are the width and height of the license plate detection box.

[0112] S52. Calculate the license plate frame spatial distance and the license plate feature vector similarity between the license plate prediction frame of the current frame image and the license plate detection frame of the current frame through a Kalman filter.

[0113] The license plate prediction frame of the current frame image is obtained based on the license plate detection frame in the frame image before the current frame image.

[0114] The license plate frame spatial distance between the license plate prediction frame of the current frame image and the license plate detection frame of the current frame can be calculated using GIOU (a loss function for target detection and bounding box regression) This solves the problem that when two license plates do not overlap, the IOU value calculated by the Kalman filter is always 0.

[0115] The calculation method can be:

[0116] in, Represents the spatial distance between two detection boxes, , and There are two detection frames a and b respectively. for and The minimum enclosing rectangle of .

[0117] A minimum bounding rectangle of two license plate detection frames, such as Figure 6 shown.

[0118] The calculation method for calculating the similarity of the license plate feature vector between the license plate prediction result in the previous frame and the recognition result in the current frame by the Kalman filter can be:

[0119] in, Represents the similarity of two license plate feature vectors, Respectively represent the license plate feature vectors corresponding to the two license plate detection frames. Transpose. is a banded sparse matrix whose non-zero elements are concentrated near the diagonal, and the non-zero elements on the diagonal are all 1, and the other non-zero elements are , usually 0.6. Its band (diagonal) can be 1 or 2. For example, when The corresponding license plate has 8 characters. When the license plate has 7 characters and band=1, , banded sparse matrix It can be as follows: .

[0120] Get the similarity of feature vectors of all license plates in two adjacent frames Then, the feature vector similarity is normalized using 0-1. The similarity is normalized to 0-1 to obtain the normalized license plate feature vector similarity .

[0121]

[0122] S53. Construct the cost matrix of the Hungarian algorithm based on the spatial distance of the license plate frame and the similarity of the license plate feature vector. Calculate the cost of the u and v license plates in the previous and next frames in pairs. The calculation method is:

[0123] in, and The weights of the similarity between GIOU and the license plate feature vector can range from 0 to 1. For example, the weight It can be taken as 0.7. Indicates the spatial distance between the two license plate frames a and b. Represents the similarity between the feature vectors of license plates a and b after standardization.

[0124] The final output cost matrix is a A matrix where each element represents the matching cost of the license plate of the previous and next frames.

[0125] S54. Use the Hungarian algorithm to calculate the cost matrix Optimize and find the allocation plan with minimum cost.

[0126] S55, remove the bad matches in the allocation scheme with the minimum cost, and add the remaining matches in the minimum cost scheme with the bad matches removed to the matching set, and execute step S57.

[0127] The definition of bad mismatch is .in and They are respectively the license plate frame spatial distance similarity threshold and the license plate feature vector similarity threshold (initialized in S50).

[0128] S56, obtaining all unmatched license plates in frames before the current frame, and determining whether there are two unmatched license plates that meet preset conditions.

[0129] Determine whether there are two unmatched license plates among all unmatched license plates in the frames before the current frame and , which have the same length and the eigenvectors Similarity Greater than a threshold (the threshold can be >0.95). In this step, the feature vector similarity The calculation is as follows:

[0130] in is the length of vector dimension 1, that is, the number of license plate characters. To normalize the feature matrix row by row to 0-1. is the identity matrix.

[0131] If it exists, it is added to the matching set and step S57 is executed.

[0132] If it does not exist, initialize a new trajectory and execute step S58.

[0133] S57: Update the matching trajectory.

[0134] S58. Update Kalman filter parameters based on the license plate frame state information of the next frame.

[0135] S59: Determine whether the track has not been updated for a period of time.

[0136] If the track has not been updated for a period of time, the track is deleted and the process returns to step S51.

[0137] Otherwise, directly return to step S51 to use the updated Kalman filter to predict the target state of the next frame until the license plate tracking is completed.

[0138] The solution of the present invention constructs a Hungarian cost matrix based on the spatial distance of the license plate detection frame and the similarity of the license plate feature vector, which can reduce the probability of false matching.

[0139] The solution of the present invention can utilize the similarity of the license plate feature vector to retrieve the lost tracking, reduce the number of switching times of the license plate track identifier, and improve the effect of license plate tracking.

[0140] The solution of the present invention adopts a multi-feature fusion method to process the features of all time series of the license plate, and can determine the best license plate number from the license plate sequence over a period of time, thereby realizing the function of identifying the vehicle license plate far away from the camera.

[0141] Based on the same inventive concept, according to another aspect of the present invention, Figure 7 As shown, an embodiment of the present invention further provides an electronic device 70, which includes a processor 710 and a memory 720. The memory 720 stores a computer program 721 that can be run on the processor. When the processor 710 executes the program, the steps of the above method are performed.

[0142] Among them, the memory, as a non-volatile storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the compression method in the embodiment of the present application. The processor executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory, that is, realizing the compression method of the above method embodiment.

[0143] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the local module via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] Based on the same inventive concept, according to another aspect of the present invention, Figure 8 As shown, an embodiment of the present invention further provides a computer storage medium 800, which stores a computer program 810 for executing the above method when executed by a processor.

[0145] Finally, it should be noted that a person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium of the program can be a disk, an optical disk, a read-only storage memory (ROM) or a random access memory (RAM), etc. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding above-mentioned any method embodiments.

[0146] It will also be appreciated by those skilled in the art that various exemplary logic blocks, modules, circuits and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, a general description has been given to the functions of various schematic components, blocks, modules, circuits and steps. Whether this function is implemented as software or hardware depends on specific applications and the design constraints imposed on the entire system. Those skilled in the art can implement the function in various ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the disclosed scope of the embodiments of the present invention.

[0147] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0148] It should be understood that, as used herein, the singular forms "a", "an" are intended to include the plural forms as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations including one or more of the associated listed items.

[0149] A person skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples; under the concept of the embodiments of the present invention, the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes in different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A vehicle license plate recognition method, characterized in that: include: Process the vehicle video stream to obtain several license plate information in each frame image; Based on the target tracking model, the license plate information in each frame of the image is processed in turn to obtain license plate trajectories of the license plates; Based on the license plate trajectory of the license plate, a final license plate number of the license plate is determined.

2. The method according to claim 1, characterized in that Based on the target tracking model, the plurality of license plate information in each frame of the image is processed in turn to obtain license plate trajectories of the plurality of license plates, including performing the following steps based on the target tracking model: Construct the cost matrix of the Hungarian algorithm based on the license plate information in the current frame and several previous frames; The cost matrix is ​​optimized using the Hungarian algorithm to obtain a target matching solution; Update existing license plate tracks or add new license plate tracks based on the target matching solution.

3. The method according to claim 2, characterized in that The cost matrix of the Hungarian algorithm is constructed based on the license plate information in the current frame and the previous frames thereof, including: Based on the Kalman filter algorithm, according to the license plate information in the current frame image and the license plate information in the previous frame image of the current frame image, the license plate prediction information of the current frame image is obtained; According to the plurality of license plate prediction information and the plurality of license plate information of the current frame image, a cost matrix of the Hungarian algorithm is constructed.

4. The method according to claim 3, characterized in that Based on the Kalman filter algorithm, according to the license plate information in the current frame image and the license plate information in the previous frame image of the current frame image, the license plate prediction information of the current frame image is obtained, including: Based on the Kalman filter algorithm, according to the license plate information in the current frame image and the license plate information in the previous frame image of the current frame image, several license plate prediction frames of the current frame image are obtained, as well as the degree of overlap between each license plate detection frame obtained according to the current frame image and each license plate prediction frame in the several license plate prediction frames.

5. The method according to claim 3, constructing the cost matrix of the Hungarian algorithm based on the plurality of license plate prediction information and the plurality of license plate information of the current frame image, comprising: Based on the spatial distance algorithm, according to the overlap between the license plate detection frame and the license plate prediction frame in the license plate information, the license plate frame spatial distance between the license plate detection frame and the license plate prediction frame is obtained; Obtaining a license plate feature vector similarity based on a license plate feature vector in the license plate information corresponding to the license plate detection frame and a license plate feature vector in the license plate information corresponding to the license plate prediction frame; Based on the obtained spatial distances of several license plate frames and the similarities of several license plate feature vectors, the cost matrix of the Hungarian algorithm is constructed.

6. The method according to claim 2, characterized in that Update existing license plate tracks or add new license plate tracks based on the target matching solution, including: Eliminate bad matches from the target matching solution based on a first preset condition, add vehicle information corresponding to the eliminated bad matches to an unmatched set, and add license plate information corresponding to the remaining matches in the target matching solution to a matched set; Filtering out license plate information that meets a second preset condition from all license plate information in the unmatched set, and adding the license plate information that meets the second preset condition to the matching set; The corresponding license plate track in the existing license plate track is updated based on all the remaining license plate information in the matching set, and the corresponding license plate track is added based on all the license plate information in the unmatched set.

7. The method according to claim 6, characterized in that The first preset condition includes that the similarity between the two license plate information is lower than a first threshold; the second preset condition includes that the license plate feature lengths in the two license plate information are the same and the license plate feature similarity is greater than a second threshold.

8. The method according to any one of claims 1 to 7, characterized in that: The vehicle video stream is processed to obtain several license plate information in each frame, including: Based on the target detection model, each frame of the vehicle video stream is detected to obtain a number of license plate detection frame information in each frame of the image; Based on the license plate recognition model, the license plate detection frame information is detected to obtain the license plate number and license plate feature vector of the license plate detection frame.

9. The method according to claim 8, characterized in that Determining a final license plate number of the license plate based on the license plate trajectory of the license plate includes: When all the license plate numbers in the license plate trajectory are the same, obtaining the final license plate number of the license plate based on any license plate number among the license plate numbers; In the case that all the license plate numbers in the license plate trajectory are not completely the same, the final license plate number of the license plate is obtained based on the license plate detection frame information and the license plate feature vector of each track point in the license plate trajectory.

10. The method according to claim 9, characterized in that In the case that all the license plate numbers in the license plate trajectory are not exactly the same, obtaining the final license plate number of the license plate based on the license plate detection frame information and the license plate feature vector of each track point in the license plate trajectory includes: Performing weighted calculation on all license plate detection frame information and all license plate feature vectors in the license plate trajectory to obtain a weighted license plate feature vector; The weighted license plate feature vector is calculated according to a normalized exponential function to obtain a final license plate number of the license plate.

11. The method according to claim 10, characterized in that After obtaining the final license plate number of the license plate, the method further includes: All license plate numbers in the corresponding license plate track are updated according to the final license plate number of the license plate.

12. An electronic device comprising: at least one processor; as well as A memory storing a computer program executable on the processor, wherein the processor executes the steps of the method according to any one of claims 1 to 11 when executing the program.

13. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images