A berth entry and exit event analysis method based on long-term multi-target tracking

Through the long-term multi-objective tracking and analysis method, combined with multiple features and data correlation technology, the problem of identifying berth entry and exit events under vehicle occlusion is solved, and accurate and efficient intelligent management in roadside parking lots are achieved.

CN112836699BActive Publication Date: 2024-11-08AIPARK TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110172449.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-11-30
Filing Date
2021-02-08
Publication Date
2024-11-08
Estimated Expiration
2041-02-08

AI Technical Summary

Technical Problem

Existing vehicle behavior analysis technology is difficult to accurately identify berth entry and exit events under vehicle occlusion, resulting in a high misjudgment rate. Especially in the intelligent management of roadside parking lots, traditional methods are difficult to effectively deal with vehicle trajectory breakage and occlusion problems.

Method used

Using berth entry and exit event analysis method based on long-term multi-objective tracking, multiple features are obtained through vehicle apparent deep learning model and data association technology, combined with edge integral features and velocity features, multimodal similarity judgment is performed, and trajectory dynamically merged , ensure accurate identification of the inlet and outgoing events of berth vehicles.

Benefits of technology

Improve the accuracy and reliability of identification of berth entry and exit events, especially when the vehicle is completely or partially blocked, it can accurately identify inbound and exit events, reduce algorithm calculations, realize real-time processing, and improve roadside parking services. Intelligent management level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112836699B_ABST
    Figure CN112836699B_ABST
Patent Text Reader

Abstract

The present invention discloses a roadside parking entry and exit event analysis method based on long-term multi-target tracking. The method obtains vehicle multimodal features and a vehicle trajectory set, and performs layer-by-layer correlation processing on the two to obtain reliable and accurate long-term trajectory information. According to the relationship between the obtained parking space position of the vehicle and the long-term trajectory of the vehicle, the parking space vehicle entry and exit event information can be accurately obtained, especially the entry and exit events that occur after the vehicle is completely blocked for a short time or are partially blocked, so as to improve the intelligent level of roadside parking services.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent analysis of vehicle behavior, and in particular to a berth entry and exit event analysis method based on long-term multi-target tracking. Background Art

[0002] In the urban parking management system, the use of parking lots accounts for a large proportion. With the continuous increase in the occupancy rate of motor vehicles in the city, parking lots can no longer meet the growing parking demand. The roadside parking mode has gradually assumed an increasingly important role, and the intelligentization of roadside parking is particularly important.

[0003] With the rapid development of computer vision, artificial intelligence, pattern recognition and other fields, the original charging mode relying on parking metering personnel has been gradually replaced by a charging mode relying on intelligent automatic management of multiple sensors. The intelligence of closed parking lots and roadside open parking lots depends entirely on the accurate analysis of vehicle behavior. According to the different data sources used, vehicle behavior analysis technology can be divided into static analysis and dynamic analysis. Among them, the static analysis method uses single-frame or multi-frame (usually three-frame) images to determine whether there is a car. This type of method has developed from the early frame difference method to the classification and recognition method based on deep learning (i.e., the problem of car and no car binary classification), so as to determine whether there is a car in the parking space. Although the static analysis method has certain adaptability, it is easy to cause misjudgment when the camera shooting angle is not ideal, resulting in the occlusion of the front and rear vehicles in the parking space, or the occlusion of the passing large vehicle. The dynamic analysis method expands the single-frame / multi-frame image into video data analysis, that is, the trajectory of the vehicle in the video data related to the parking space is tracked, and the parking space entry and exit event information is obtained by analyzing the relationship between the trajectory and the parking space position. Compared with the static analysis method, this type of method has improved the accuracy of judging berth entry and exit events, but it also has many problems.

[0004] Based on the dynamic analysis method, the present invention proposes a long-term multi-target tracking berth entry and exit event accurate analysis method. The method first uses multiple features such as vehicle appearance deep learning model extraction features, motion state features, vehicle edge features, etc. to model and describe the vehicle, and uses data association technology to obtain the vehicle trajectory. Usually, due to occlusion, the trajectory of the same vehicle may be broken, that is, in the entire video process, the trajectory of the same vehicle can be split into multiple trajectories, targeting the problem of trajectory set merging.

[0005] Based on this, this method further uses edge integral features, speed features, license plate features, etc. to perform multimodal similarity judgment, complete the dynamic merging of trajectory sets, and give the analysis results of parking vehicle entry and exit events according to the relationship between the long-term trajectory of the vehicle and the parking position, which is used to accurately identify parking entry and exit events. Summary of the invention

[0006] The purpose of the present invention is to provide a parking space entry and exit event analysis method based on long-term multi-target tracking, aiming to solve the problem of real-time analysis of roadside parking space entry and exit events, especially the problem of accurately obtaining parking space entry and exit events in the case of vehicle occlusion.

[0007] To achieve the above object, the present invention provides a berth entry and exit event analysis method based on long-term multi-target tracking, the method comprising:

[0008] Obtain multiple consecutive frames of images in the surveillance video, extract the vehicle and license plate position information in each frame; determine the subordinate relationship between the license plate and the vehicle based on the vehicle and license plate position information;

[0009] Preprocessing the vehicle and license plate images to obtain multi-dimensional features of the vehicle and a comprehensive confidence level of the license plate recognition result;

[0010] Associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to obtain a vehicle trajectory set;

[0011] Segment the vehicle image, extract the edge features of the vehicle instance area, and calculate the edge integral features;

[0012] Match the vehicle trajectory set and edge integral features to obtain the long-term trajectory of the vehicle;

[0013] The parking space position of the vehicle is obtained, and the vehicle entry and exit events are confirmed based on the relationship between the parking space position and the long-term trajectory of the vehicle.

[0014] As a further improvement of the present invention, the step of determining the subordinate relationship between the license plate and the vehicle specifically includes preprocessing each frame of the image, extracting the vehicle area and the license plate area in the image, and determining the positional relationship between the license plate area and the vehicle area in the image. If the license plate area is located in the vehicle area in the image, it is determined that the license plate belongs to the vehicle. The preprocessing includes one or more of vehicle and license plate color channel conversion, scaling, and filtering.

[0015] As a further improvement of the present invention, preprocessing the license plate image to obtain the comprehensive confidence of the license plate recognition result specifically includes:

[0016] Correct the license plate area image and determine the license plate category based on the correction result.

[0017] Segment the license plate characters to obtain the number of license plate characters, and determine the complete confidence of the license plate based on the number of license plate characters.

[0018] Recognize the license plate characters and confirm the character recognition confidence;

[0019] The comprehensive confidence of the license plate recognition result is determined based on the license plate integrity confidence and character recognition confidence.

[0020] As a further improvement of the present invention, obtaining the multi-dimensional features of the vehicle at least includes obtaining one or more of the appearance feature description of the vehicle and the vehicle status information.

[0021] As a further improvement of the present invention, the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result are associated, and after obtaining the vehicle trajectory set, the vehicle trajectory set is also updated. The updating of the vehicle trajectory set specifically includes selecting a license plate with a high threshold to associate with the vehicle in the trajectory set, and adding one or more of new vehicle appearance features and vehicle status information.

[0022] As a further improvement of the present invention, updating the vehicle trajectory set specifically further includes deleting a vehicle trajectory set whose motion state cannot be obtained for a long time.

[0023] As a further improvement of the present invention, segmenting the vehicle image, extracting the edge features of the vehicle instance region, and calculating the edge integral features specifically include calculating the edge features of a moving vehicle and the edge features of a stationary vehicle.

[0024] As a further improvement to the present invention, calculating the integral feature of a stationary vehicle specifically includes: performing weighted accumulation on the acquired stable multi-frame edge features of the vehicle to obtain an edge integral feature.

[0025] As a further improvement of the present invention, calculating the moving vehicle specifically includes performing translation and / or rotation changes according to the trajectory set to obtain multi-segment edge integral features of the vehicle.

[0026] As a further improvement of the present invention, obtaining the long-term trajectory of the vehicle according to the edge integral feature specifically includes matching the vehicle edge integral and merging the trajectory sets to obtain the long-term trajectory of the vehicle.

[0027] A berth entry and exit event analysis device based on long-term multi-target tracking includes:

[0028] The acquisition and analysis device is used to obtain multiple consecutive frames of images in the surveillance video, extract the vehicle and license plate position information in each frame of the image; and determine the subordinate relationship between the license plate and the vehicle based on the vehicle and license plate position information;

[0029] Preprocessing device; used for preprocessing the vehicle and license plate images to obtain multi-dimensional features of the vehicle and comprehensive confidence of the license plate recognition result;

[0030] Acquisition device; associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to obtain a vehicle trajectory set;

[0031] A computing device for segmenting a vehicle image, extracting edge features of a vehicle instance region, and calculating edge integral features; a matching device for matching a vehicle trajectory set and edge integral features to obtain a long-term trajectory of the vehicle;

[0032] The judging device obtains the parking space position of the vehicle and confirms the entry and exit event of the vehicle according to the relationship between the parking space position and the long-term trajectory of the vehicle.

[0033] As a further improvement of the present invention, the device also includes an extraction device, which is used to pre-process each frame of the image, extract the vehicle area and the license plate area in the image, and determine the positional relationship between the license plate area and the vehicle area. If the license plate area is located in the vehicle area, it is determined that the license plate belongs to the vehicle. The preprocessing includes one or more of the vehicle and license plate color channel conversion, scaling, and filtering.

[0034] As a further improvement of the present invention, it also includes:

[0035] The correction device is used to correct the license plate area image through a correction algorithm and determine the license plate category according to the correction result.

[0036] A segmentation device, used to segment the license plate characters by a segmentation algorithm to obtain the number of license plate characters, and determine the complete confidence level of the license plate according to the number of license plate characters;

[0037] The recognition device is used to recognize the license plate characters through the recognition device and confirm the character recognition confidence; and determine the comprehensive confidence of the license plate recognition result according to the license plate integrity confidence and the character recognition confidence.

[0038] As a further improvement of the present invention, the acquisition device is also used to update the vehicle trajectory set. Updating the vehicle trajectory set specifically includes selecting license plates with high thresholds to associate with vehicles in the trajectory set, and adding one or more of new vehicle appearance features and vehicle status information.

[0039] As a further improvement of the present invention, the segmentation device is also used to segment the vehicle image through a segmentation algorithm to extract edge features of the vehicle instance region;

[0040] The computing device is also used to calculate edge integral features, specifically including calculating moving vehicle edge features and stationary vehicle edge features.

[0041] As a further improvement of the present invention, the computing device is also used to match the vehicle edge integrals and perform trajectory set merging to obtain the long-term trajectory of the vehicle.

[0042] Beneficial Effects

[0043] The invention discloses a roadside parking entry and exit event analysis method based on long-term multi-target tracking, which associates vehicle multimodal features and the comprehensive confidence of license plate recognition results layer by layer to obtain reliable and accurate long-term trajectory information. On the one hand, a more reliable long-term trajectory is obtained through the combination of multiple weak constraints; on the other hand, this method can greatly reduce the amount of algorithm calculation, including the following levels (note: the following time statistics are on the NVIDIA 1060 graphics card): first, a wide residual network with fewer layers is used to obtain the apparent characteristics of the vehicle area, and the single target characterization and recognition time is better than 1ms; secondly, the Kalman filter is used to establish the target motion feature description, and the motion feature establishment time is better than 0.5ms; thirdly, by establishing a license plate correction network and a character segmentation and recognition network, the traditional multi-step calculation is integrated into the deep learning network, so that the license plate recognition time is better than 1ms; in addition, in the vehicle detection algorithm, based on the existing classic detection algorithm, according to the characteristics of roadside parking, especially the frequent occlusion, the network structure is adjusted to obtain a model with both accuracy and time consumption; finally, through model acceleration, including model pruning and TensorRT acceleration, the algorithm time consumption of each link is further optimized. On a platform with Nvidia graphics card computing power superior to 1050Ti, single-channel video can be processed in real time to further accurately obtain information on parking vehicle entry and exit events, especially when the vehicle is fully or partially obscured, entry and exit events can be accurately identified, thereby improving the intelligence level of roadside parking services. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a berth entry and exit event analysis method based on long-term multi-target tracking of the present invention;

[0045] Figure 2 It is a flow chart of a berth entry and exit event analysis method based on long-term multi-target tracking of the present invention. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0047] The present invention provides a parking space entry and exit event analysis method based on long-term multi-target tracking, which can accurately obtain parking space vehicle entry and exit time information, especially when the vehicle is fully or partially blocked, it can accurately identify the entry and exit events, and can improve the intelligent management level of roadside parking.

[0048] Specifically, the method disclosed in the present invention comprises:

[0049] S1; obtaining multiple consecutive frames of images in the surveillance video, extracting the position of the vehicle and the license plate in each frame; determining the subordinate relationship between the license plate and the vehicle according to the position information of the vehicle and the license plate;

[0050] Specifically, a monitoring device is used to obtain a plurality of continuous frames of images in the monitoring video, and then each frame of the image is preprocessed.

[0051] When the license plate is twisted or exposed, the preprocessing method includes converting, scaling and filtering the license plate color and vehicle color channels in the image to obtain a clear license plate and vehicle image, and then further using a machine learning-based detection method to obtain the vehicle area and license plate area in the image, using the prior knowledge that a single license plate can only belong to one vehicle and the license plate should be in the vehicle area, and using the intersection-and-union ratio to determine the subordination of the license plate, the intersection-and-union ratio calculation method includes but is not limited to IOU, CIOU, DIOU, GIOU and other intersection-and-union ratio calculation methods. For example, in actual situations, due to mutual occlusion of vehicles (such as occlusion between the front and rear vehicles), a license plate recognition frame often appears in both vehicle detection frames in whole or in part. At this time, the attribution is determined as follows: the attribution is determined according to the size of the weighted IOU of the license plate frame and the vehicle frame, where the weight is: the position quadrant relationship of the license plate frame and the vehicle frame is weighted under different camera viewing angles.

[0052] While acquiring each frame of the surveillance video, it is necessary to track and monitor multiple license plates and vehicle targets in real time to obtain dynamic information of vehicle targets and license plate targets. The specific vehicle detection method includes but is not limited to detection and tracking through target detection networks such as YOLO, SSD, and CenterNet; the license plate detection method includes but is not limited to detection and tracking through target detection networks such as CRNN+CTC, YOLO, etc. to obtain the best vehicle and license plate image information.

[0053] The monitoring device includes one or a combination of multiple types of gun-type cameras or spherical cameras.

[0054] S2: preprocessing the vehicle and license plate images to obtain multi-dimensional features of the vehicle and comprehensive confidence of the license plate recognition result;

[0055] Since vehicles may encounter partial or complete occlusions when entering or leaving the vehicle, it is particularly important to obtain multi-dimensional features of the vehicle in order to improve the accuracy of vehicle entry or exit event judgment. Specifically, the backbone part of the detection network is first used to obtain the multi-dimensional appearance feature description of the vehicle, and the Kalman filter is further used to model the vehicle motion state to obtain vehicle state information such as vehicle speed and acceleration. The multi-dimensional features of the vehicle are obtained by the above method. The detection network backbone used for the vehicle appearance feature description includes but is not limited to Vgg, ResNet, Darknet53, ResNext, etc.

[0056] Furthermore, the license plate image needs to be preprocessed to obtain a comprehensive confidence of the license plate recognition result, where the comprehensive confidence of the license plate recognition result includes the complete confidence of the license plate and the confidence of the character recognition;

[0057] The specific steps for calculating the complete confidence of the license plate are as follows: first, correct the license plate area image; second, classify the corrected vehicle image to confirm that the vehicle belongs to one of the new energy license plate, ordinary license plate or double-layer character license plate; third, segment the characters in the license plate to obtain the number of license plate characters, and confirm the completeness of the license plate according to the number of license plate characters. In this embodiment, specifically, the license plate integrity confidence is compared and confirmed according to the ratio of the number of segmented characters to the correct number of characters according to different license plate forms (single-layer or double-sided, ordinary license plate or new energy license plate, etc.); the vehicle segmentation method includes but is not limited to MASK-RCNN, BlendMask, SOLO, PolarMask, etc. For example, when the color of the license plate is green, it is preliminarily determined that the vehicle is a new energy vehicle. If the color of the license plate is blue, it is preliminarily confirmed that the vehicle is an ordinary passenger car; if it is confirmed that the vehicle is an ordinary license plate and the number of characters of the license plate is 7, it can be preliminarily confirmed that the license plate is a complete license plate.

[0058] The license plate correction algorithm includes but is not limited to a correction method based on license plate corner point correction and a correction method based on deep learning to detect the left and right sides.

[0059] The steps for calculating the character recognition confidence are as follows: recognize the segmented characters to obtain the recognition results, and calculate the character recognition confidence based on the recognition results. Specifically, the character recognition confidence is the product of the minimum and mean values ​​of the recognition accuracy of each segmented character, but the confidence is not limited to the product mode, and other component modes are also possible (such as average value, L2 norm, etc.). Among them, for ordinary license plates, new energy license plates, double-layer license plates, etc., the number and position of their characters are specific patterns. If the number of segmented characters, the aspect ratio of the license plate, etc. obtained are closer to a certain type of real license plate (such as ordinary license plates), then the confidence of the license plate character segmentation is higher; in addition, each segmented character is recognized through a deep learning network, and a single character recognition confidence is obtained, and then the single character confidence is calculated by L2 norm to obtain the entire license plate character recognition confidence; the comprehensive segmentation confidence and recognition confidence give the single frame image license plate confidence. Finally, according to the time dimension, the confidence of the same license plate in multiple consecutive frames of images is dynamically optimized.

[0060] S3: Associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to obtain a vehicle trajectory set;

[0061] Specifically, first, according to the comprehensive confidence of the license plate, the vehicle corresponding to the license plate with a high threshold is selected, and then the vehicles corresponding to the license plates above the threshold are further associated with the vehicles in the trajectory set, and the license plate information in the trajectory set is updated to the highest confidence.

[0062] Then, for vehicles with low confidence or blocked license plates, the same processing is performed on the previous frame image or the next frame image obtained by S2, by obtaining the vehicle appearance features, motion state features, license plate feature matching and vehicle motion state matching, vehicle appearance feature similarity description and license plate feature similarity description, etc., combined with the vehicle detection results of the current frame image, the cost matrix is ​​constructed, and the data association method is used to obtain the optimal allocation result; according to the optimal allocation result, the trajectory state of each vehicle is updated. In the cost matrix, the horizontal axis is the number of vehicle trajectories retained in the previous frame, and the vertical axis is the number of vehicles obtained by the detection algorithm in the current frame. Its value is the difference between the multidimensional feature description corresponding to the vehicle trajectory of the previous frame and the multidimensional feature description of the vehicle obtained by the detection algorithm in the current frame, and through L2 normalization, the matching relationship corresponding to the minimum value is found from the cost matrix, which is the vehicle ID relationship in the current frame.

[0063] The so-called description of the degree of matching of vehicle motion states includes but is not limited to Mahalanobis distance, L2 distance, etc.

[0064] The so-called vehicle appearance feature similarity description includes but is not limited to minimum cosine distance, L2 distance, etc.

[0065] The so-called license plate feature similarity description includes but is not limited to L2 distance, Mahalanobis distance, etc.

[0066] Furthermore, the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result are associated. After obtaining the vehicle trajectory set, the vehicle trajectory set is also updated. Specifically, targets that cannot obtain status updates for a long time are eliminated, indicating that the vehicle is no longer in the monitoring area; for vehicles that cannot update their status temporarily due to occlusion and other conditions, their status is temporarily retained, and the status update can be selected at a time. For example, if the target reappears within 1 minute, its status will continue to be updated, otherwise it will be eliminated.

[0067] The so-called data association methods include but are not limited to the Hungarian algorithm, the Nearest algorithm, the K-Means algorithm, etc.

[0068] S4: Segment the vehicle image, extract the edge features of the vehicle instance area, and calculate the edge integral features;

[0069] Specifically, for a stationary vehicle, which is completely or partially occluded (occluded by the front and rear vehicles when parked due to the shooting angle), or gradually occluded and then revealed (occluded by a passing vehicle), its overall edge or partial edge has a relatively strong stability. Multiple consecutive frames of images are obtained, and the edge features that appear stably in multiple consecutive frames are weighted and accumulated to obtain the edge integral features.

[0070] For moving vehicles, according to the trajectory features obtained by S3, combined with the camera intrinsic parameters, scale + translation + rotation transformation is performed to obtain the multi-segment edge integral features of the moving vehicle. The edge integral features of the vehicle are calculated, including the edge integral features of the stationary vehicle and the moving vehicle respectively. The specific method is as follows: the camera intrinsic parameters and the berth edge are at the image vanishing point, and the camera perspective information is estimated (because the camera intrinsic parameters basically do not change after the camera is installed and fixed, the estimated camera extrinsic parameters can be calculated in the system initialization stage instead of in real time); on this basis, the vehicle motion state (i.e., translation vector: Δx, Δy) is used, combined with the camera intrinsic parameters and the estimated extrinsic parameters, to give the vehicle three-dimensional description change (including rotation, translation, and scale estimation value), and finally the vehicle frame of the current frame is reprojected, the edge features of the reprojected image are calculated, and the edge integral features are combined with the previous edge integral features for dynamic adjustment. In order to reduce the amount of calculation, the edge integral map is sparse (i.e., downsampled).

[0071] The so-called segmentation standard of the edge integral feature of the moving vehicle is: if the current vehicle has a large in-plane rotation motion, a new segment of edge integral feature is reconstructed;

[0072] S5: Match the vehicle trajectory set and edge integral features to obtain the long-term trajectory of the vehicle;

[0073] Match the edge integral features and merge the trajectory sets, remove the trajectory sets that cannot be merged and exceed the retention time, and then obtain the long-term trajectory of the vehicle. The specific method is as follows: For the edge integral feature maps corresponding to the trajectories that have no overlapping relationship in the two time dimensions, perform least squares matching to obtain the matching similarity data. When the matching similarity is higher than the preset threshold, it is considered that the two trajectories correspond to the same vehicle.

[0074] Specifically, for a stationary vehicle, the least squares matching calculation is directly performed; for a moving vehicle, multi-segment edge integral feature least squares matching is performed to obtain their association.

[0075] S6: Acquire the parking space position of the vehicle, and confirm the vehicle's entry and exit events based on the relationship between the parking space position and the long-term trajectory of the vehicle.

[0076] Specifically, for each vehicle's long-term trajectory in the video, we determine whether it is related to the parking space based on the position relationship between its head and tail ends and the parking space. If both the front and rear ends of the vehicle are in the parking space, then the vehicle is definitely in the parking space. If the front part of the vehicle is not in the parking space, then the vehicle has an outgoing or incoming motion. Then, based on the order in which the motions occur, we determine whether the vehicle is parked in or out of the parking space.

[0077] In the present invention, the accuracy of judging parking space entry and exit events depends to a large extent on the accuracy of vehicle trajectory recognition and the length of the effective trajectory. Compared with the existing methods based on a single tracking algorithm and vehicle association based solely on license plate recognition, the present invention has the following advantages: (1) It is robust and reliable, especially when the vehicle is partially or completely blocked, it still has good recognition performance. Compared with the vehicle trajectory recognition method based on license plate recognition, the present invention uses the license plate information as one of the constraints. When the license plate is blocked or the license plate recognition is inaccurate, the present invention adopts a multi-modal and multi-level trajectory association method. Effectively organizing multiple methods with "weaker" effects can effectively improve the accuracy of vehicle trajectory recognition. In addition, for situations where vehicle occlusion occurs frequently, this method can improve the recognition accuracy compared to the "vehicle trajectory recognition method based on a single tracking algorithm (such as KLT, particle filtering, Karaman filtering, etc.)" in the prior art; (2) Good real-time performance. The present invention merges the detection tasks, and a single frame image can obtain the license plate and vehicle positioning information in one detection; the appearance features are combined with the vehicle trajectory set, and the vehicle appearance feature description is obtained at the detection position at the same time, which reduces the calculation dimension and greatly improves the real-time performance of the algorithm.

[0078] The present invention also discloses a device for executing a parking space entry and exit event analysis method based on long-term multi-target tracking, and a collection and analysis device for acquiring continuous multiple frames of images in a surveillance video, extracting vehicle and license plate position information in each frame of the image; and determining the subordinate relationship between the license plate and the vehicle according to the vehicle and license plate position information;

[0079] Preprocessing device; used for preprocessing the vehicle and license plate images to obtain multi-dimensional features of the vehicle and comprehensive confidence of the license plate recognition result;

[0080] An acquisition device is provided for associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to acquire a vehicle trajectory set;

[0081] A computing device for segmenting a vehicle image, extracting edge features of a vehicle instance region, and calculating edge integral features;

[0082] A matching device, matching the vehicle trajectory set and the edge integral feature to obtain the long-term trajectory of the vehicle;

[0083] The judging device obtains the parking space position of the vehicle and confirms the entry and exit event of the vehicle according to the relationship between the parking space position and the long-term trajectory of the vehicle.

[0084] The device also includes an extraction device, which is used to pre-process each frame of the image, extract the vehicle area and the license plate area in the image, and determine the positional relationship between the license plate area and the vehicle area. If the license plate area is located in the vehicle area, it is determined that the license plate belongs to the vehicle. The pre-processing includes one or more of the vehicle and license plate color channel conversion, scaling, and filtering.

[0085] The correction device is used to correct the license plate area image and determine the license plate type based on the correction result.

[0086] A segmentation device, which segments the license plate characters to obtain the number of license plate characters, and determines the complete confidence level of the license plate according to the number of license plate characters;

[0087] The recognition device recognizes the license plate characters and confirms the character recognition confidence; and determines the comprehensive confidence of the license plate recognition result according to the license plate integrity confidence and the character recognition confidence.

[0088] The acquisition device is also used to update the vehicle trajectory set. Updating the vehicle trajectory set specifically includes selecting a license plate with a high threshold to associate with the vehicle in the trajectory set, and adding one or more of new vehicle appearance features and vehicle status information.

[0089] The segmentation device is also used to segment the vehicle image and extract the edge features of the vehicle instance area;

[0090] The computing device is also used to calculate the edge features of a moving vehicle and a stationary vehicle, specifically to perform weighted accumulation of the obtained stable multi-frame edge features of the vehicle to obtain edge integral features; and to perform translation and / or rotation changes according to the trajectory set to obtain multi-segment edge integral features of the vehicle.

[0091] The computing device is also used to match the vehicle edge integrals and perform trajectory set merging to obtain the long-term trajectory of the vehicle.

[0092] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0093] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0094] The disclosed embodiments are described above to enable any person skilled in the art to implement or use the present invention. Various modifications of these embodiments are obvious to those skilled in the art, and the general principles defined herein may also be applied to other embodiments without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments given herein, but is consistent with the broadest scope of the principles and novel features disclosed in this application.

[0095] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

[0096] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0097] The various illustrative logic blocks or units described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0098] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0099] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

[0100] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0101] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A berth entry and exit event analysis method based on long-term multi-target tracking, characterized in that: The method comprises: Obtain multiple consecutive frames of images in the surveillance video, extract the vehicle and license plate position information in each frame; determine the subordinate relationship between the license plate and the vehicle based on the vehicle and license plate position information; Preprocessing the vehicle and license plate images to obtain multi-dimensional features of the vehicle and a comprehensive confidence level of the license plate recognition result; Associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to obtain a vehicle trajectory set; The step of associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to obtain the vehicle trajectory set includes: according to the comprehensive confidence of the license plate, selecting the vehicle corresponding to the license plate with a confidence greater than a preset threshold and associating the vehicle corresponding to the license plate with the vehicle in the trajectory set; updating the license plate information in the trajectory set to the highest confidence; for the vehicle corresponding to the license plate with a license plate confidence lower than the preset threshold or the vehicle with the license plate blocked, obtaining the vehicle appearance characteristics and motion state characteristics and performing license plate feature matching, vehicle motion state matching, vehicle appearance feature similarity description and license plate feature similarity description; constructing a cost matrix according to the vehicle detection result of the current frame image, and obtaining the optimal allocation result by using the data association method; according to the optimal allocation result, updating the trajectory state of each vehicle, wherein the horizontal coordinate of the cost matrix is ​​the number of vehicle trajectories retained in the previous frame, and the vertical coordinate is the number of vehicles obtained by the detection algorithm in the current frame, and its value is obtained by performing a difference operation and normalization on the multi-dimensional feature description corresponding to the vehicle trajectory of the previous frame and the multi-dimensional feature description of the vehicle obtained by the detection algorithm in the current frame; Segment the vehicle image, extract the edge features of the vehicle instance area, and calculate the edge integral features; The segmentation of the vehicle image, extraction of the edge features of the vehicle instance region, and calculation of the edge integral features specifically include calculating the edge integral features of the moving vehicle and the edge integral features of the stationary vehicle; The calculating of the edge integral feature of the stationary vehicle specifically includes: weighting the acquired stable multi-frame edge features of the vehicle to obtain the edge integral feature; The calculating of the edge integral features of the moving vehicle specifically includes performing translation and / or rotation changes according to the trajectory set to obtain multi-segment edge integral features of the vehicle; Match the vehicle trajectory set and edge integral features to obtain the long-term trajectory of the vehicle; Merge the trajectory set based on the multi-dimensional features of the vehicle and the license plate recognition results; For the unmerged trajectory set, the edge integral feature is used for secondary calculation to obtain the long-term trajectory of the vehicle; Acquire the parking space position of the vehicle, and confirm the vehicle's entry and exit events based on the relationship between the parking space position and the vehicle's long-term trajectory; The step of determining the subordinate relationship between the license plate and the vehicle specifically includes: Preprocessing each frame of the image, extracting the vehicle area and the license plate area image in the image, determining the positional relationship between the license plate area and the vehicle area in the image, and if the license plate area in the image is located in the vehicle area, determining that the license plate belongs to the vehicle, wherein the preprocessing includes one or more of the vehicle and license plate color channel conversion, scaling, and filtering; Preprocessing the license plate image to obtain the comprehensive confidence of the license plate recognition result specifically includes: Correct the license plate area image and determine the license plate category based on the correction result. Segment the license plate characters to obtain the number of license plate characters, and determine the complete confidence of the license plate based on the number of license plate characters. Recognize the license plate characters and confirm the character recognition confidence; The comprehensive confidence of the license plate recognition result is determined based on the license plate integrity confidence and character recognition confidence.

2. The method according to claim 1, characterized in that Acquiring the multi-dimensional features of the vehicle includes at least acquiring one or more of the vehicle's appearance feature description and the vehicle's motion state information.

3. The method according to claim 2, characterized in that The multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result are associated. After obtaining the vehicle trajectory set, the vehicle trajectory set is also updated. The updating of the vehicle trajectory set specifically includes selecting a license plate with a high threshold to associate with the vehicle in the trajectory set, and adding one or more of new vehicle appearance features and vehicle motion state information.

4. The method according to claim 3, characterized in that Updating the vehicle trajectory set specifically includes deleting vehicle trajectories whose multi-dimensional feature states cannot be updated for a long time and adding newly identified vehicle trajectories.

5. The method according to claim 1, characterized in that: Acquiring the long-term trajectory of the vehicle according to the edge integral feature specifically includes matching the edge integral feature of the vehicle and merging the trajectory sets to obtain the long-term trajectory set of the vehicle.

6. A berth entry and exit event analysis device based on long-term multi-target tracking includes: The acquisition and analysis device is used to obtain multiple consecutive frames of images in the surveillance video and extract the vehicle and license plate position information in each frame; Determine the subordinate relationship between the license plate and the vehicle according to the vehicle and license plate position information; Pretreatment device; Used to pre-process the vehicle and license plate images to obtain multi-dimensional features of the vehicle and comprehensive confidence of the license plate recognition result; Acquisition device; associating the multi-dimensional features of the vehicle and the comprehensive confidence of the license plate recognition result to obtain a vehicle trajectory set; The acquisition device is specifically used to select vehicles corresponding to license plates with confidence greater than a preset threshold according to the comprehensive confidence of the license plates, and associate the vehicles corresponding to the license plates with the vehicles in the trajectory set; update the license plate information in the trajectory set to the highest confidence; for vehicles corresponding to license plates with a license plate confidence lower than the preset threshold or vehicles with blocked license plates, obtain vehicle appearance characteristics and motion state characteristics, and perform license plate feature matching, vehicle motion state matching, vehicle appearance feature similarity description, and license plate feature similarity description; construct a cost matrix according to the vehicle detection result of the current frame image, and use the data association method to obtain the optimal allocation result; according to the optimal allocation result, update the trajectory state of each vehicle, wherein the horizontal coordinate of the cost matrix is ​​the number of vehicle trajectories retained in the previous frame, and the vertical coordinate is the number of vehicles obtained by the detection algorithm in the current frame, and its value is obtained by performing a difference operation and normalization on the multidimensional feature description corresponding to the vehicle trajectory of the previous frame and the multidimensional feature description of the vehicle obtained by the detection algorithm in the current frame; A computing device for segmenting a vehicle image, extracting edge features of a vehicle instance region, and calculating edge integral features; The computing device is specifically used to merge the trajectory sets according to the multi-dimensional features of the vehicle and the license plate recognition results; perform secondary calculation on the unmerged trajectory sets using the edge integral features to obtain the long-term trajectory of the vehicle; Matching device; Match the vehicle trajectory set and edge integral features to obtain the vehicle long-term trajectory set; A judgment device, which obtains the parking position of the vehicle and confirms the entry and exit event of the vehicle based on the relationship between the parking position and the long-term trajectory of the vehicle; The device also includes an extraction device, which is used to pre-process each frame of the image, extract the vehicle area and the license plate area in the image, determine the positional relationship between the license plate area and the vehicle area, and if the license plate area is located in the vehicle area, determine that the license plate belongs to the vehicle, and the pre-processing includes one or more of the vehicle and license plate color channel conversion, scaling, and filtering; The device further comprises: The correction device is used to correct the license plate area image through a correction algorithm and determine the license plate category according to the correction result. A segmentation device, used to segment the license plate characters by a segmentation algorithm to obtain the number of license plate characters, and determine the complete confidence level of the license plate according to the number of license plate characters; The segmentation device is also used to segment the vehicle image through a segmentation algorithm to extract the edge features of the vehicle instance area; the calculation device is also used to calculate the edge integral features of the moving vehicle and the edge integral features of the stationary vehicle; The recognition device is used to recognize the license plate characters through a recognition algorithm and confirm the character recognition confidence; and determine the comprehensive confidence of the license plate recognition result according to the license plate integrity confidence and the character recognition confidence.

7. The device according to claim 6, characterized in that: The acquisition device is also used to update the vehicle trajectory set. Updating the vehicle trajectory set specifically includes selecting a license plate with a high threshold to associate with the vehicle in the trajectory set, and adding one or more of new vehicle appearance features and vehicle status information.

8. The device according to claim 6, characterized in that: The matching device is also used to match the edge integral features of the vehicle through an algorithm, and perform trajectory set merging to obtain the long-term trajectory of the vehicle.