Cross-camera vehicle tracking and twin data generation scheme
Patent Information
- Application Number
- CN202410784814.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2024-11-12
AI Technical Summary
[0006]一、车辆跟踪验证只通过特征匹配,只依赖ReID特征匹配进行相似度验证,对于光线变化和遮挡,跨摄像头跟踪的准确率较低;
[0039] First: the present invention uses the ReID feature extraction module and the license plate recognition model to bind the target lost by the previous camera to realize the solution of tracking vehicles and other targets across cameras. At the same time, the camera is calibrated, and the prediction module is based on the tracking results of the previous step. Generate the position of the target in the twin coordinate system and generate the corresponding twin target data;
Smart Images

Figure CN118942049A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of road security technology, and specifically to a cross-camera vehicle tracking and twin data generation solution. Background Art
[0002] As the application of deep learning in the field of vision becomes more mature, the video analysis-related business in the security field (mainly highways and tunnels) is becoming increasingly heavy, and user needs are also increasing dramatically. Users are no longer satisfied with the target detection and event analysis business of a single camera. Usually, more advanced needs are to be able to make final business judgments through the linkage of multiple cameras and comprehensive specific actual conditions.
[0003] After searching, the patent document with the publication number of "CN117218580A" discloses a highway cross-camera multi-vehicle tracking method and application combined with multiple models, performs vehicle target detection of multi-scale features of moving window self-attention on the target vehicle image of the tracking video, extracts multi-fusion ReID features for the detected vehicle, and performs a single-camera vehicle tracking trajectory of a temporary storage-fusion strategy based on the target detection box and feature extracted ReID, uses multiple post-processing technologies to filter the single-camera vehicle tracking trajectory and associate and match the single-camera vehicle tracking trajectories between multiple cameras to form the final cross-camera vehicle tracking trajectory. The highway cross-camera multi-vehicle tracking method combined with multiple models can realize accurate cross-camera vehicle tracking on the highway, and reduce the occurrence of tracking failures caused by the change of ID identities of multiple vehicles tracked across cameras;
[0004] This method mainly uses moving window self-attention to optimize the vehicle target detection model, uses multi-feature ReID fusion to optimize the vehicle target re-identification model, and uses the re-ranking method to optimize the tracked vehicle trajectory to achieve multi-vehicle tracking on the highway using multiple cameras.
[0005] The above prior art still has the following defects:
[0006] First, vehicle tracking verification is only done through feature matching, and only relies on ReID feature matching for similarity verification. For light changes and occlusion, the accuracy of cross-camera tracking is low;
[0007] 2. Without camera calibration data, twin data prediction and trajectory prediction cannot be achieved. Summary of the invention
[0008] The purpose of the present invention is to provide a cross-camera vehicle tracking and twin data generation solution to solve the problems raised in the above background technology.
[0009] The technical solution of the present invention is: a cross-camera vehicle tracking and twin data generation solution, including:
[0010] Video reading module, the video reading module is used to read image frames from the video for subsequent processing;
[0011] The calibration module is used to select points within the visible range of each camera and calibrate the image of each camera;
[0012] Multi-target tracking module: The multi-target tracking module uses Kalman filtering and Hungarian matching algorithms to track multiple targets. For a new target that appears in a single camera, it is necessary to perform feature matching and license plate recognition verification with the target lost by the previous camera. After weighted similarity matrix calculation, if it is determined to be the same vehicle, ID inheritance is performed to ensure the uniqueness of the ID in the vehicle tracking process. If the license plate recognition and feature similarity calculation do not match the vehicle information cached by the previous camera, the target is tracked as a new target.
[0013] Cache target module: The cache target module mainly caches the features and license plate data of each target. The target feature calculation method is the 1*256 dimension feature value output by the target image through the ReID model. Each target caches the feature values of three frames. At the same time, the license plate recognition results of each vehicle are cached.
[0014] Twin data generation module: The twin data generation module directly maps the target position of the twin coordinate system to the target within the visible range of the camera through the calibration matrix. In the invisible range, the Kalman filter is used to predict the position of the lost target. If the predicted moving speed of the current target is too fast in the twin coordinate system, it is necessary to wait for the target of the ID to appear at a slower speed in the next camera. On the contrary, when the target of the ID has appeared in the next camera, the estimation module needs to accelerate the moving speed of the target of the ID in the current camera to catch up with the actual twin coordinate position of the ID in the twin coordinate system.
[0015] Preferably, when the calibration module calibrates each camera screen image, the construction personnel mark no less than four points in the road surface area within the visible range of the camera, denoted as Gp. In the video screen, the pixel coordinates corresponding to Gp are denoted as Pp. The pixel pairs composed of (Gp, Pp) are used to calculate the homography matrix for the conversion between the physical coordinate system and the pixel coordinate system.
[0016] Preferably, the multi-target tracking module first calls the detection model to detect all target frames in the current frame image, and calculates the cost matrix with all tracker objects in the previous frame image. The matched detection frame is responsible for updating the position and status of the tracker. The unmatched detection frame needs to calculate the similarity with the tracker object lost by the previous camera to verify the matching relationship. For matches with a large similarity, the ID in the tracker corresponding to the previous matched camera is used for tracking, and the tracker is updated at the same time. For unmatched targets, the tracker ID is globally updated.
[0017] Preferably, the cache target module is also responsible for storing the targets lost by the current camera movement, and transmitting them to the next camera for feature matching and license plate recognition result verification. The cached data includes the features of the target and the results of license plate recognition; the operations of the multi-target tracking module and the cache target module are repeated in a loop to keep the target tracked and updated.
[0018] Preferably, the spacing distance between the cameras is x, the visible range of the camera is m, and the invisible range of the camera is n. According to the calculation of the multi-target tracking module and the cache target module, the ID will not jump when the vehicle moves between the cameras.
[0019] Preferably, the specific operation process of the multi-target tracking module and the cache target module includes:
[0020] Step 201: Perform target detection on the video frame image, and each target has a corresponding detection frame and category.
[0021] Step 202: The target detection frame and the tracker object of the current camera are matched using the Hungarian matching algorithm to calculate the cost matrix.
[0022] Step 203: Calculate the binding relationship between the detection target and the tracker according to the cost matrix.
[0023] Step 204: For the bound tracker object, the tracker data is updated by the Kalman filter prediction module and the like through module 208.
[0024] Step 205: The detection target that is not bound to the tracker needs to calculate the similarity matrix with the vehicle features and license plate recognition data cached by the previous camera to match the vehicle target lost in the previous camera.
[0025] Step 206: If there is no target that matches the target lost by the previous camera, a new ID is created for tracking.
[0026] Step 207: If the target lost by the previous camera is matched, the matched target ID is inherited and tracking is continued.
[0027] Step 208: Perform Kalman prediction and position update on the target currently tracked by the camera.
[0028] Step 209: Determine whether the target tracked by the tracker is lost. If lost, it is necessary to cache the vehicle features and license plate result data of the lost tracker. If not lost, continue tracking the next frame.
[0029] Step 210: Cache vehicle features, and cache 1*256 feature data output by the ReID model of three different sizes of vehicles: large, medium and small.
[0030] Step 211: Cache crnn license plate recognition result data.
[0031] Preferably, the specific operation process of the twin data generation module includes:
[0032] Step 301: Calibrate each camera, mark no less than four points at four actual positions within the visible range of each camera, and use the calculated calibration matrix to map the target of the pixel point coordinate system to the actual space coordinate system; each camera has its own calibration matrix, and the visible range is m, the blind area is n, and the distance between cameras is x, that is, x=m+n.
[0033] Step 302: The current camera tracking target module mainly performs coordinate transformation on the target within the current visible range and calculates the actual twin coordinate position by using calibration operation;
[0034] Step 303: The Kalman prediction module mainly predicts the position of the current camera blind spot. Step 302 calculates the actual position of the twin coordinate system. Within the field of view, the position of the Kalman prediction module will be updated. At the same time, the Kalman prediction module is responsible for predicting the actual position of the blind spot.
[0035] Step 304: the next camera calculates the target within the visual range of the next camera according to the calibration matrix of the next camera;
[0036] Step 305: When the target predicted by the Kalman algorithm of the current camera exceeds the total distance x of the current camera, the target has been predicted to appear in the next camera, but the target has not yet appeared in the actual next camera. At this time, the prediction speed needs to be slowed down to wait for the target to appear in the next camera.
[0037] Step 306: When the target predicted by the Kalman algorithm of the current camera is still in the blind spot, the target of the next camera has appeared. At this time, the prediction module of the target needs to be informed and the target prediction object needs to be accelerated to catch up with the position of the target of the next camera.
[0038] The present invention provides a cross-camera vehicle tracking and twin data generation solution through improvement, which has the following improvements and advantages compared with the prior art:
[0039] First, the present invention uses the ReID feature extraction module and the license plate recognition model to bind the target lost by the previous camera, realizes the solution of tracking vehicles and other targets across cameras, and performs camera calibration at the same time. The estimation module generates the position of the target in the twin coordinate system according to the tracking result of the previous step, and generates the corresponding twin target data;
[0040] Second: The present invention solves the problem in the prior art that vehicle tracking verification is only carried out through feature matching and only relies on ReID feature matching for similarity verification, resulting in low accuracy of cross-camera tracking due to light changes and occlusion; and the problem that there is no camera calibration data and twin data prediction and trajectory prediction cannot be achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments:
[0042] Figure 1 It is a structural block diagram of the present invention;
[0043] Figure 2 It is a flowchart of the operation of the multi-target tracking module and the cache target module of the present invention;
[0044] Figure 3 It is a coordinate system estimation flow chart of the twin data generation module of the present invention;
[0045] Figure 4 This is a schematic diagram of twin data under adjacent cameras of the present invention.
[0046] Description of reference numerals:
[0047] 101. Video reading module; 102. Calibration module; 103. Multi-target tracking module; 104. Cache target module; 105. Twin data generation module. DETAILED DESCRIPTION
[0048] The present invention is described in detail below, and the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] The present invention provides a cross-camera vehicle tracking and twin data generation solution through improvement. The technical solution of the present invention is:
[0050] like Figure 1 As shown, a cross-camera vehicle tracking and twin data generation solution includes:
[0051] The video reading module 101 is used to read image frames from the video for subsequent processing;
[0052] The calibration module 102 is used to select points within the visible range of each camera and calibrate each camera image;
[0053] The multi-target tracking module 103 uses Kalman filtering and Hungarian matching algorithms to track multiple targets. For a target newly appearing in a single camera, it is necessary to perform feature matching and license plate recognition verification with the target lost by the previous camera. After weighted similarity matrix calculation, if it is determined to be the same vehicle, ID inheritance is performed to ensure the uniqueness of the ID in the vehicle tracking process. If the license plate recognition and feature similarity calculation do not match the vehicle information cached by the previous camera, the target is tracked as a new target.
[0054] The target cache module 104 mainly caches the features and license plate data of each target. The target feature calculation method is the 1*256 dimension feature value output by the target image through the ReID model. Each target caches the feature values of three frames (large, medium and small). At the same time, the license plate recognition result of each vehicle is cached.
[0055] Twin data generation module 105, the twin data generation module 105 directly maps the target position of the twin coordinate system to the target within the visible range of the camera through the calibration matrix, and uses the Kalman filter to predict the position of the lost target in the invisible range. If the predicted moving speed of the current target in the twin coordinate system is too fast, it is necessary to wait for the target of the ID to appear at a slower moving speed in the next camera. On the contrary, when the target of the ID has appeared in the next camera, the estimation module needs to accelerate the moving speed of the target of the ID in the current camera to catch up with the actual twin coordinate position of the ID in the twin coordinate system.
[0056] Among them, when the calibration module 102 calibrates each camera screen image, within the visible range of the camera, the construction personnel mark no less than four points in the road area, denoted as Gp. In the video screen, the pixel point coordinates corresponding to Gp are denoted as Pp. The pixel pairs composed of (Gp, Pp) are used to calculate the homography matrix for the conversion between the physical coordinate system and the pixel coordinate system.
[0057] Furthermore, the multi-target tracking module 103 first calls the detection model to detect all target frames in the current frame image, and calculates the cost matrix with all tracker objects in the previous frame image. The matched detection frame is responsible for updating the position and status of the tracker. The unmatched detection frame needs to calculate the similarity with the tracker object lost by the previous camera to verify the matching relationship. For matches with a large similarity, the ID in the tracker corresponding to the previous matched camera is used for tracking, and the tracker is updated at the same time. For unmatched targets, the tracker ID is globally updated.
[0058] Furthermore, the cache target module 104 is also responsible for storing the targets lost by the current camera movement, and transmitting them to the next camera for feature matching and license plate recognition result verification. The cached data includes the features of the target and the results of license plate recognition; the operations of the multi-target tracking module 103 and the cache target module 104 are repeated in a loop to keep the target tracked and updated.
[0059] Furthermore, the spacing distance between the cameras is x, the visible range of the camera is m, and the invisible range of the camera is n (ie, x=m+n). According to the calculation of the multi-target tracking module 103 and the cache target module 104, the ID will not jump when the vehicle moves between the cameras.
[0060] Based on the above scheme, the present invention uses the ReID feature extraction module and the license plate recognition model to bind the target lost by the previous camera, realizes the scheme of tracking vehicles and other targets across cameras, and performs camera calibration at the same time. The estimation module generates the position of the target in the twin coordinate system according to the tracking result of the previous step, and generates the corresponding twin target data;
[0061] It solves the problem in the existing technology that vehicle tracking verification is only through feature matching, and only relies on ReID feature matching for similarity verification, resulting in low accuracy of cross-camera tracking due to light changes and occlusion; and the problem that there is no camera calibration data and twin data prediction and trajectory prediction cannot be achieved.
[0062] like Figure 2 As shown, the specific operation process of the multi-target tracking module 103 and the cache target module 104 includes:
[0063] Step 201: Perform target detection on the video frame image, and each target has a corresponding detection frame and category.
[0064] Step 202: The target detection frame and the tracker object of the current camera are matched using the Hungarian matching algorithm to calculate the cost matrix.
[0065] Step 203: Calculate the binding relationship between the detection target and the tracker according to the cost matrix.
[0066] Step 204: For the bound tracker object, the tracker data is updated by the Kalman filter prediction module and the like through module 208.
[0067] Step 205: The detection target that is not bound to the tracker needs to calculate the similarity matrix with the vehicle features and license plate recognition data cached by the previous camera to match the vehicle target lost in the previous camera.
[0068] Step 206: If there is no target that matches the target lost by the previous camera, a new ID is created for tracking.
[0069] Step 207: If the target lost by the previous camera is matched, the matched target ID is inherited and tracking is continued.
[0070] Step 208: Perform Kalman prediction and position update on the target currently tracked by the camera.
[0071] Step 209: Determine whether the target tracked by the tracker is lost. If lost, it is necessary to cache the vehicle features and license plate result data of the lost tracker. If not lost, continue tracking the next frame.
[0072] Step 210: Cache vehicle features, and cache 1*256 feature data output by the ReID model of three different sizes of vehicles: large, medium and small.
[0073] Step 211: Cache crnn license plate recognition result data.
[0074] like Figure 3 As shown, the specific operation process of the twin data generation module 105 includes
[0075] Step 301: Calibrate each camera, mark no less than four points at four actual positions within the visible range of each camera, and use the calculated calibration matrix to map the target of the pixel point coordinate system to the actual space coordinate system; each camera has its own calibration matrix, and the visible range is m, the blind area is n, and the distance between cameras is x, that is, x=m+n.
[0076] Step 302: The current camera tracking target module mainly performs coordinate transformation on the target within the current visible range and calculates the actual twin coordinate position by using calibration operation;
[0077] Step 303: The Kalman prediction module mainly predicts the position of the current camera blind spot. Step 302 calculates the actual position of the twin coordinate system. Within the field of view, the position of the Kalman prediction module will be updated. At the same time, the Kalman prediction module is responsible for predicting the actual position of the blind spot.
[0078] Step 304: the next camera calculates the target within the visual range of the next camera according to the calibration matrix of the next camera;
[0079] Step 305: When the target predicted by the Kalman algorithm of the current camera exceeds the total distance x of the current camera, the target has been predicted to appear in the next camera, but the target has not yet appeared in the actual next camera. At this time, the prediction speed needs to be slowed down to wait for the target to appear in the next camera.
[0080] Step 306: When the target predicted by the Kalman algorithm of the current camera is still in the blind spot, the target of the next camera has appeared. At this time, the prediction module of the target needs to be informed and the target prediction object needs to be accelerated to catch up with the position of the target of the next camera.
[0081] To further demonstrate the target tracking results under two adjacent cameras, please refer to Figure 4 It can be seen that there are three actual targets of the twins below, namely 46, 53, and 56. Among them, targets 46 and 56 are targets within the visible range of different cameras, and target 53 exists in the blind spot of the previous camera.
[0082] The above description enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A cross-camera vehicle tracking and twin data generation solution, characterized by: include: A video reading module (101), the video reading module (101) is used to read image frames from the video for subsequent processing; A calibration module (102), the calibration module (102) being used to select points within the visible range of each camera and calibrate each camera screen image; A multi-target tracking module (103), wherein the multi-target tracking module (103) uses algorithms such as Kalman filtering and Hungarian matching to track multiple targets. For a target newly appearing in a single camera, feature matching and license plate recognition verification need to be performed with the target lost by the previous camera. After weighted similarity matrix calculation, if it is determined to be the same vehicle, ID inheritance is performed to ensure the uniqueness of the ID in the vehicle tracking process. If the license plate recognition and feature similarity calculation do not match the vehicle information cached by the previous camera, the target is tracked as a new target; A target caching module (104), wherein the target caching module (104) mainly caches the features and license plate data of each target, wherein the target features are calculated by the feature values of the target image in 1*256 dimensions output by the ReID model, and each target caches the feature values of three frames; and at the same time, the license plate recognition result of each vehicle is cached; A twin data generation module (105), wherein the twin data generation module (105) directly maps the target position of the twin coordinate system to the target within the visible range of the camera through a calibration matrix, and uses a Kalman filter to predict the position of the lost target in the invisible range. If the predicted moving speed of the current target in the twin coordinate system is too fast, it is necessary to wait for the target of the ID to appear in the next camera at a slower moving speed. On the contrary, when the target of the ID has appeared in the next camera, the estimation module needs to accelerate the moving speed of the target of the ID in the current camera to catch up with the actual twin coordinate position of the ID in the twin coordinate system.
2. A cross-camera vehicle tracking and twin data generation solution according to claim 1, characterized in that: When the calibration module (102) calibrates each camera screen image, the construction personnel mark no less than four points in the road surface area within the visible range of the camera, which are denoted as Gp. In the video screen, the pixel point coordinates corresponding to Gp are denoted as Pp. The pixel pairs composed of (Gp, Pp) are used to calculate the homography matrix for the conversion between the physical coordinate system and the pixel coordinate system.
3. A cross-camera vehicle tracking and twin data generation solution according to claim 2, characterized in that: The multi-target tracking module (103) first calls the detection model to detect all target frames in the current frame image, and calculates the cost matrix with all tracker objects in the previous frame image. The matched detection frame is responsible for updating the position and state of the tracker. The unmatched detection frame needs to calculate the similarity with the tracker object lost by the previous camera to verify the matching relationship. For matches with a large similarity, the ID in the tracker corresponding to the previous camera that matched is used for tracking, and the tracker is updated at the same time. For unmatched targets, the tracker ID is globally updated.
4. The cross-camera vehicle tracking and twin data generation solution according to claim 3 is characterized in that: The target cache module (104) is also responsible for storing the target lost by the current camera movement, and transmitting it to the next camera for feature matching and license plate recognition result verification. The cached data includes the target features and the license plate recognition result. The operations of the multi-target tracking module (103) and the target cache module (104) are repeated cyclically to keep the target tracked and updated.
5. The cross-camera vehicle tracking and twin data generation solution according to claim 1, characterized in that: The distance between the cameras is x, the visible range of the cameras is m, and the invisible range of the cameras is n. According to the calculation of the multi-target tracking module (103) and the cache target module (104), the ID of the vehicle will not jump when it moves between the cameras.
6. The cross-camera vehicle tracking and twin data generation solution according to claim 4 is characterized in that: The specific operation process of the multi-target tracking module (103) and the cache target module (104) includes: Step 201: Perform target detection on the video frame image, and each target has a corresponding detection frame and category. Step 202: The target detection frame and the tracker object of the current camera are matched using the Hungarian matching algorithm to calculate the cost matrix. Step 203: Calculate the binding relationship between the detection target and the tracker according to the cost matrix. Step 204: For the bound tracker object, the tracker data is updated by the Kalman filter prediction module and the like through module 208. Step 205: The detection target that is not bound to the tracker needs to calculate the similarity matrix with the vehicle features and license plate recognition data cached by the previous camera to match the vehicle target lost in the previous camera. Step 206: If there is no target that matches the target lost by the previous camera, a new ID is created for tracking. Step 207: If the target lost by the previous camera is matched, the matched target ID is inherited and tracking is continued. Step 208: Perform Kalman prediction and position update on the target currently tracked by the camera. Step 209: Determine whether the target tracked by the tracker is lost. If lost, it is necessary to cache the vehicle features and license plate result data of the lost tracker. If not lost, continue tracking the next frame. Step 210: Cache vehicle features, and cache 1*256 feature data output by the ReID model of three different sizes of vehicles: large, medium and small. Step 211: Cache crnn license plate recognition result data.
7. The cross-camera vehicle tracking and twin data generation solution according to claim 5 is characterized in that: The specific operation process of the twin data generation module (105) includes: Step 301: Calibrate each camera, mark no less than four points at four actual positions within the visible range of each camera, and use the calculated calibration matrix to map the target of the pixel point coordinate system to the actual space coordinate system; each camera has its own calibration matrix, and the visible range is m, the blind area is n, and the distance between cameras is x, that is, x=m+n. Step 302: The current camera tracking target module mainly performs coordinate transformation on the target within the current visible range and calculates the actual twin coordinate position by using calibration operation; Step 303: The Kalman prediction module mainly predicts the position of the current camera blind spot. Step 302 calculates the actual position of the twin coordinate system. Within the field of view, the position of the Kalman prediction module will be updated. At the same time, the Kalman prediction module is responsible for predicting the actual position of the blind spot. Step 304: the next camera calculates the target within the visual range of the next camera according to the calibration matrix of the next camera; Step 305: When the target predicted by the Kalman algorithm of the current camera exceeds the total distance x of the current camera, the target has been predicted to appear in the next camera, but the target has not yet appeared in the actual next camera. At this time, the prediction speed needs to be slowed down to wait for the target to appear in the next camera. Step 306: When the target predicted by the Kalman algorithm of the current camera is still in the blind spot, the target of the next camera has appeared. At this time, the prediction module of the target needs to be informed and the target prediction object needs to be accelerated to catch up with the position of the target of the next camera.
Citation Information
Cited By
Tunnel global tracking method based on old camera re-identification technology
CN120726091A
