A visible light video multi-target tracking method and system based on DeepSORT
By introducing GIoU matching and adaptive Kalman filtering into the DeepSORT algorithm, combined with camera motion compensation technology, the problems of inaccurate matching and insufficient stability in multi-target tracking are solved, achieving higher tracking accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2023-12-28
- Publication Date
- 2026-07-07
AI Technical Summary
The existing DeepSORT multi-target tracking algorithm ignores the difference between mismatched trajectories and unconfirmed trajectories during the matching stage, uses a uniform measurement noise scale, cannot cope with tracking instability caused by camera motion, and does not effectively utilize motion information, resulting in insufficient tracking accuracy and robustness.
By introducing the GIoU matching algorithm to distinguish between mismatched trajectories and unconfirmed trajectories, adaptively adjusting the measurement noise scale of the Kalman filter, and combining it with camera motion compensation technology to incorporate motion information into the cost matrix calculation, the tracking accuracy and stability are improved.
It improves the accuracy of multi-target tracking, reduces the interference of uncertainty on the tracking system, enhances the robustness of the system, reduces the impact of camera shake, and reduces false matching caused by changes in appearance features or occlusion.
Smart Images
Figure CN117788514B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method and system for multi-target tracking of visible light video based on DeepSORT. Background Technology
[0002] Visible light images are digital images acquired through devices such as cameras and video cameras. They are widely used in photography and video recording in daily life, driving, and traffic management, and have a high degree of ubiquity. Therefore, the analysis and processing of visible light images, especially technologies for multi-target detection and tracking, also have broad application scenarios.
[0003] Early target tracking algorithms primarily relied on target modeling or tracking target features. These algorithms suffered from two fatal flaws: they failed to consider background information, making them prone to tracking failures under interference such as target occlusion, lighting changes, and motion blur; and their slow execution speed made them unable to meet real-time requirements.
[0004] With the rapid development of deep learning technology, deep learning-based tracking algorithms have been widely used in multi-object tracking. Network models trained using deep learning, especially in the context of big data, possess more powerful convolutional feature output representation capabilities. Even in complex image scenes, these models still exhibit good recognition and tracking capabilities. However, most current deep learning-based tracking algorithms still face some challenges in multi-object tracking. For example, when an object is briefly occluded and then reappears, the model often misidentifies it as a new object; during long-term tracking, the tracker struggles to maintain stable tracking of the target. Therefore, researching a visible light video multi-object detection and tracking system capable of accurate and coherent tracking has significant practical value.
[0005] The existing classic target tracking model DeepSORT has the following problems:
[0006] Question 1: In the matching phase, DeepSORT performs IoU (Intersection over Union) matching between the mismatched detection target after cascaded matching and the mismatched trajectory and the unconfirmed trajectory. However, this ignores the difference between the mismatched trajectory and the unconfirmed trajectory, that is, the mismatched trajectory is more likely to be the correct trajectory than the unconfirmed trajectory.
[0007] Question 2: The linear Kalman filter algorithm used in DeepSORT simply sets a uniform measurement noise scale for all detected targets. However, the quality of target detection directly affects the accuracy of tracking. When the target detection quality is low, the corresponding detection results may have high uncertainty, interfering with the multi-target tracking system.
[0008] Question 3: The DeepSORT model exhibits tracking instability when the camera moves or shakes, which may lead to incorrect matching due to changes in the appearance features of the target or target occlusion during camera movement.
[0009] Question 4: The DeepSORT model only uses motion information to filter out non-compliant matches, and does not incorporate motion information into the calculation of the cost matrix. This can lead to incorrect matches when the target is partially occluded due to the lack of motion information. Summary of the Invention
[0010] To address the aforementioned problems, this invention provides a visible light video multi-target tracking method and system based on DeepSORT. By reconstructing the matching stage of DeepSORT, after cascaded matching, GIoU (Generalized Intersection over Union) matching is first performed between the mismatched trajectory and the mismatched target, and then IoU matching is performed between the mismatched target and the unconfirmed trajectory and the GIoU mismatched trajectory. This improves the accuracy of multi-target tracking. By adaptively adjusting the corresponding measurement noise scale of the Kalman filter, the interference of uncertainty on the tracking system is reduced. By introducing motion camera compensation technology, the stability of the tracked target is maintained, the impact of camera shake on tracking is reduced, and the robustness of the target tracking system is improved. By incorporating motion information into the cost matrix calculation process, it helps to reduce erroneous matching caused by changes in appearance features or occlusion.
[0011] To achieve the above objectives, this invention provides a visible light video multi-target tracking method based on DeepSORT, comprising:
[0012] Input a visible light video, and use the YOLOv5 target detector to detect the target in the current frame of the visible light video to obtain target information;
[0013] A camera motion compensation method is introduced to correct the position information of the target based on the camera's motion state, and a Kalman filter algorithm is used to predict the trajectory of all detected targets to determine whether the trajectory is in a confirmed or unconfirmed state.
[0014] The detected target and the confirmed trajectory are combined with the target's appearance features, distance, and motion information to calculate a cost matrix for cascade matching;
[0015] For the mismatched trajectory and mismatched target in cascaded matching, the GIoU matching algorithm is used for GIoU matching, and for the mismatched trajectory and mismatched target in GIoU matching, the IoU matching algorithm is used for IoU matching.
[0016] Using the Kalman filter algorithm combined with an adaptive noise scale, the matching trajectories obtained by the cascaded matching, the GIoU matching, and the IoU matching are updated in state. Unconfirmed trajectories and confirmed trajectories that exceed the maximum number of mismatches in the mismatch trajectories obtained by the IoU matching are deleted, and the target information after state update is output.
[0017] In the above technical solution, preferably, the method of introducing camera motion compensation to correct the position information of the target according to the motion state of the camera includes the following specific process:
[0018] For the visible light video, the motion state of the camera is determined before processing each frame;
[0019] If the camera is detected to be in motion, a camera motion compensation method is introduced. Based on the ECC (Enhanced Correlation Coefficient) method, image registration or alignment is performed on the offset or distortion caused by the camera motion to correct the target position information.
[0020] The ECC method is expressed as follows:
[0021]
[0022] i w (p)=φ(i r ;p)
[0023] in, This indicates the untransformed reference image with a mean of 0. Indicates passing through φ(i) r The mean of the distorted image after transformation is 0, and φ represents an image transformation function controlled by parameter p.
[0024] In the above technical solution, preferably, the step of using the Kalman filter algorithm to predict the trajectory of all detected targets and determine the state of the trajectory as either confirmed or unconfirmed includes the following specific steps:
[0025] Using the posterior estimate of k-1 from the previous time step The prior estimated state at time k is obtained by transforming the state transition matrix F.
[0026]
[0027] Estimate the covariance P using the posterior estimate of the previous time step k-1. k-1 Calculate the prior estimate covariance of the current time k.
[0028]
[0029] Where Q represents the process noise covariance matrix;
[0030] The state transition matrix F is:
[0031]
[0032] Based on the calculation of the prior estimated state and the prior estimated covariance, the estimated state and position of the trajectory at the next time step are determined;
[0033] By comparing the matching results of the trajectory for a preset number of consecutive times, the trajectory that matches for the preset number of consecutive times is determined to be a confirmed trajectory; otherwise, it is determined to be an unconfirmed trajectory.
[0034] In the above technical solution, preferably, the step of combining the detected target with the confirmed trajectory and the target's appearance feature distance and motion information to calculate the cost matrix for cascade matching specifically includes:
[0035] The cost matrix is calculated using the target's appearance feature distance and motion information. The specific calculation method is as follows:
[0036] C=λA a +(1-λ)A m
[0037] Where C is the appearance feature distance cost A a And the cost of motion information A m The weighted sum, where λ is the weighting factor;
[0038] The cost matrix is combined with the detected target and the confirmed trajectory, and the Hungarian algorithm is used for matching to obtain the matching trajectory, the mismatched trajectory, and the mismatched target.
[0039] In the above technical solution, preferably, the GIoU matching algorithm is expressed as:
[0040]
[0041] Where IoU represents the ratio of the intersection area to the union area of the two bounding boxes, A c U represents the area of the smallest bounding box that encloses the two bounding boxes, and U represents the total area covered by the two bounding boxes, which is the area of each bounding box plus the area of their intersection.
[0042] The IoU matching algorithm is expressed as follows:
[0043]
[0044] Where A∩B represents the area of the overlapping portion of bounding box A and bounding box B, and A∪B represents the total coverage area of bounding box A and bounding box B.
[0045] In the above technical solution, preferably, the process of using the Kalman filter algorithm combined with an adaptive noise scale to update the state of the matching trajectories obtained by the cascaded matching, the GIoU matching, and the IoU matching specifically includes:
[0046] Based on the confidence level of the detection results, the measurement noise scale in the Kalman filter algorithm is adjusted. The noise covariance is calculated as follows:
[0047]
[0048] Among them, R k For the preset constant measurement noise covariance, C k The detection confidence value at state k;
[0049] The covariance matrix P and the observation matrix H, as well as the noise covariance matrix R, are estimated using prior estimation. k Calculate the Kalman gain matrix K k :
[0050]
[0051] The prior estimate x of the Kalman filter is projected onto the measurement space through the observation matrix H, and the residual y between the measured value z and the observed value z is calculated:
[0052]
[0053] Converted to matrix form:
[0054]
[0055] The predicted and measured values of the Kalman filter are calculated according to the Kalman gain K. k By combining the proportions, we obtain the posterior estimated state:
[0056]
[0057] Calculate the posterior covariance of the Kalman filter:
[0058]
[0059] Based on the calculation of the posterior estimated state and the posterior estimated covariance, the trajectory state at the current moment is determined, and the trajectory state is updated.
[0060] In the above technical solution, preferably, the specific process of deleting the mismatch trajectory obtained from the IoU matching includes:
[0061] For the IoU matching mismatch target, create a corresponding new trajectory;
[0062] For the mismatch trajectory of the IoU matching, the unconfirmed trajectory is deleted, the confirmed trajectory where the mismatch reaches the preset maximum mismatch number threshold is deleted, and the confirmed trajectory where the mismatch does not reach the preset maximum mismatch number threshold is retained, and matching is continued in subsequent frames.
[0063] This invention also proposes a visible light video multi-target tracking system based on DeepSORT, characterized in that it applies the visible light video multi-target tracking method based on DeepSORT disclosed in any of the above technical solutions, including:
[0064] The target detection module is used to acquire visible light video and use the YOLOv5 target detector to detect targets in the current frame of the visible light video to obtain target information;
[0065] The target prediction module is used to introduce a camera motion compensation method to correct the position information of the target according to the motion state of the camera, and to use the Kalman filter algorithm to predict the trajectory of all detected targets and determine the state of the trajectory as confirmed or unconfirmed.
[0066] The cascaded matching module is used to combine the detected target with the confirmed trajectory and the target's appearance feature distance and motion information to calculate a cost matrix for cascaded matching;
[0067] The association matching module is used to perform GIoU matching for mismatched trajectories and mismatched targets in cascaded matching, and to perform IoU matching for mismatched trajectories and mismatched targets in GIoU matching.
[0068] The trajectory update module is used to update the state of the matching trajectories obtained by the cascaded matching, the GIoU matching, and the IoU matching by using the Kalman filter algorithm combined with an adaptive noise scale. It deletes the unconfirmed trajectories and confirmed trajectories that exceed the maximum number of mismatches in the mismatch trajectories obtained by the IoU matching, and outputs the target information after the state update.
[0069] In the above technical solution, preferably, the target prediction module is specifically used for:
[0070] For the visible light video, the motion state of the camera is determined before processing each frame;
[0071] If the camera is detected to be in motion, a camera motion compensation method is introduced. Based on the enhanced correlation coefficient (ECC) method, image registration or alignment is performed on the offset or distortion caused by the camera motion to correct the target position information.
[0072] The ECC method is expressed as follows:
[0073]
[0074] i w (p)=φ(i r ;p)
[0075] in, This indicates the untransformed reference image with a mean of 0. Indicates passing through φ(i) r The mean of the distorted image after transformation is 0, and φ represents an image transformation function controlled by parameter p.
[0076] Using the posterior estimate of k-1 from the previous time step The prior estimated state at time k is obtained by transforming the state transition matrix F.
[0077]
[0078] Estimate the covariance P using the posterior estimate of the previous time step k-1. k-1 Calculate the prior estimate covariance of the current time k.
[0079]
[0080] Where Q represents the process noise covariance matrix;
[0081] The state transition matrix F is:
[0082]
[0083] Based on the calculation of the prior estimated state and the prior estimated covariance, the estimated state and position of the trajectory at the next time step are determined;
[0084] By comparing the matching results of the trajectory for a preset number of consecutive times, the trajectory that matches for the preset number of consecutive times is determined to be a confirmed trajectory; otherwise, it is determined to be an unconfirmed trajectory.
[0085] In the above technical solution, preferably, the trajectory update module is specifically used for:
[0086] Based on the confidence level of the detection results, the measurement noise scale in the Kalman filter algorithm is adjusted. The noise covariance is calculated as follows:
[0087]
[0088] Among them, R k For the preset constant measurement noise covariance, C k The detection confidence value at state k;
[0089] The covariance matrix P and the observation matrix H, as well as the noise covariance matrix R, are estimated using prior estimation. k Calculate the Kalman gain matrix K k :
[0090]
[0091] The prior estimate x of the Kalman filter is projected onto the measurement space through the observation matrix H, and the residual y between the measured value z and the observed value z is calculated:
[0092]
[0093] Converted to matrix form:
[0094]
[0095] The predicted and measured values of the Kalman filter are calculated according to the Kalman gain K. k By combining the proportions, we obtain the posterior estimated state:
[0096]
[0097] Calculate the posterior covariance of the Kalman filter:
[0098]
[0099] Based on the calculation of the posterior estimated state and the posterior estimated covariance, the trajectory state at the current moment is determined, and the trajectory state is updated.
[0100] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0101] (1) By reconstructing the matching stage of DeepSORT, after cascaded matching, GIoU matching is first performed between the mismatched trajectory and the mismatched target, and then IoU matching is performed between the mismatched target and the unconfirmed trajectory and the GIoU mismatched trajectory, which improves the accuracy of multi-target tracking.
[0102] (2) By adaptively adjusting the corresponding measurement noise scale of the Kalman filter, the interference of uncertainty on the tracking system is reduced;
[0103] (3) By introducing motion camera compensation technology, the stability of the tracked target is maintained, the impact of camera shake on tracking is reduced, and the robustness of the target tracking system is improved.
[0104] (4) By incorporating motion information into the cost matrix calculation process, it helps to reduce incorrect matching caused by changes in appearance features or occlusion. Attached Figure Description
[0105] Figure 1 This is a flowchart illustrating a visible light video multi-target tracking method based on DeepSORT, as disclosed in one embodiment of the present invention.
[0106] Figure 2 Images (a) and (b) are schematic diagrams of the visible light video multi-target tracking method disclosed in an embodiment of the present invention before and after labeling in the frame image;
[0107] Figure 3 This is a schematic diagram of the cascaded matching process of a visible light video multi-target tracking method based on DeepSORT disclosed in one embodiment of the present invention;
[0108] Figure 4 (a) and (b) are schematic diagrams before and after multi-target tracking tests on frame images in a test dataset disclosed in an embodiment of the present invention;
[0109] Figure 5 (a) and (b) are schematic diagrams before and after multi-target tracking tests on the test dataset frame images disclosed in an embodiment of the present invention. Detailed Implementation
[0110] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0111] The present invention will now be described in further detail with reference to the accompanying drawings:
[0112] like Figure 1 As shown, a visible light video multi-target tracking method based on DeepSORT provided by the present invention includes:
[0113] Input a visible light video and use the YOLOv5 target detector to detect targets in the current frame of the visible light video to obtain target information;
[0114] A camera motion compensation method is introduced to correct the target's position information based on the camera's motion state, and the Kalman filter algorithm is used to predict the trajectory of all detected targets to determine whether the trajectory is in a confirmed or unconfirmed state.
[0115] The detected target and the confirmed trajectory are combined with the target's appearance features, distance, and motion information to calculate the cost matrix for cascaded matching;
[0116] For the mismatched trajectory and mismatched target in cascaded matching, the GIoU matching algorithm is used for GIoU matching, and for the mismatched trajectory and mismatched target in GIoU matching, the IoU matching algorithm is used for IoU matching.
[0117] The Kalman filter algorithm combined with an adaptive noise scale is used to update the state of the matching trajectories obtained by cascade matching, GIoU matching and IoU matching. Unconfirmed trajectories and confirmed trajectories that exceed the maximum number of mismatches in the mismatch trajectories obtained by IoU matching are deleted, and the target information after state update is output.
[0118] In this implementation, by reconstructing the matching stage of DeepSORT, after cascaded matching, GIoU matching is first performed between the mismatched trajectory and the mismatched target, and then IoU matching is performed between the mismatched target and the unconfirmed trajectory and the GIoU mismatched trajectory. This improves the accuracy of multi-target tracking. By adaptively adjusting the corresponding measurement noise scale of the Kalman filter, the interference of uncertainty on the tracking system is reduced. By introducing motion camera compensation technology, the stability of the tracked target is maintained, the impact of camera shake on tracking is reduced, and the robustness of the target tracking system is improved. By incorporating motion information into the cost matrix calculation process, it helps to reduce erroneous matching caused by changes in appearance features or occlusion.
[0119] like Figure 2As shown in (a) and (b), YOLOv5 specifically extracts features from the input image through multiple layers of convolutional and pooling layers. These layers progressively reduce the image resolution and transform the image into a series of feature maps, on which object detection is performed. YOLOv5 employs a single-stage object detection approach. The model divides the feature map into a grid, with each grid responsible for detecting objects in the image. Each grid predicts multiple bounding boxes, each containing the object's location and category information. Prediction processing is performed on each bounding box, including its location coordinates and object category. The location coordinates typically include the bounding box's center coordinates, width, and height. The category information is the model's prediction of the object's category. Each bounding box is also accompanied by a confidence score, representing the confidence level that the bounding box contains an object. The output information of YOLOv5 (bounding box location, category, and confidence score) is input into DeepSORT for multi-object tracking.
[0120] Traditional tracking-by-detection trackers, such as DeepSORT, rely primarily on the overlap between predicted and detected bounding boxes during the matching phase. However, with camera movement, the bounding box positions change, increasing the likelihood of ID switching. This invention introduces a camera motion compensation method to address the offset or distortion between images caused by camera movement, achieving image registration or alignment, thereby correcting image deviations caused by camera movement.
[0121] In the cascaded matching process, to improve matching accuracy, this invention prioritizes mismatched trajectories, considering them more likely to be correct. Therefore, this invention employs the GIoU matching algorithm to match mismatched trajectories with the detection target. Finally, the mismatched detection target, mismatched trajectory, and unconfirmed trajectory are all subjected to IoU matching to obtain the matched trajectory and detection target, as well as the mismatched trajectory and mismatched detection target.
[0122] The linear Kalman filter algorithm used in traditional DeepSort sets a uniform measurement noise scale for all detected targets. However, the quality of target detection directly affects the accuracy of tracking. If the target detection algorithm can accurately and reliably identify and locate targets, the corresponding detection results will have high credibility, i.e., low uncertainty. Therefore, for high-quality detections, the measurement noise scale can be reasonably reduced to ensure that these detection results have a relatively large weight in state updates. When the target detection quality is low, the corresponding detection results may have high uncertainty. In this case, the system should give these detection results a larger measurement noise scale to reduce their impact on state updates, thereby reducing the interference of uncertainty on the tracking system. Therefore, for detection results of different qualities, this invention uses an adaptive noise scale combined with Kalman filtering for state updates.
[0123] Finally, based on the updated target and trajectory information, the tracker's output is saved in MOT data format, recording multiple pieces of information about the tracked target, including frame number, target ID, top-left X coordinate, top-left Y coordinate, width, height, confidence level, and position coordinates (X, Y, Z).
[0124] In the above embodiments, preferably, a camera motion compensation method is introduced to correct the target's position information based on the camera's motion state, ensuring the target's stability. The specific process includes:
[0125] For visible light video, the camera's motion state is determined before processing each frame;
[0126] If camera motion is detected, a camera motion compensation method is introduced. Based on the enhanced correlation coefficient (ECC) method, image registration or alignment is performed on the offset or distortion caused by camera motion to correct the target position information.
[0127] The ECC method is expressed as follows:
[0128]
[0129] i w (p)=φ(i r ;p)
[0130] in, This indicates the untransformed reference image with a mean of 0. Indicates passing through φ(i) r The mean of the distorted image after transformation is 0, and φ represents an image transformation function controlled by parameter p.
[0131] In the above embodiment, preferably, the Kalman filter algorithm is used to predict the trajectory of all detected targets to determine whether the trajectory is in a confirmed or unconfirmed state. The specific process includes:
[0132] Using the posterior estimate of k-1 from the previous time step The prior estimated state at time k is obtained by transforming the state transition matrix F.
[0133]
[0134] Estimate the covariance P using the posterior estimate of the previous time step k-1. k-1 Calculate the prior estimate covariance of the current time k.
[0135]
[0136] Where Q represents the process noise covariance matrix;
[0137] The state transition matrix F is:
[0138]
[0139] Based on the calculation of the prior estimated state and the prior estimated covariance, the estimated state and position of the trajectory at the next time step are determined.
[0140] Compare the matching results of the trajectory for a preset number of consecutive times. The trajectory that matches for the preset number of consecutive times is determined to be a confirmed trajectory; otherwise, it is determined to be an unconfirmed trajectory.
[0141] like Figure 3 As shown, in the above embodiment, preferably, the detected target and the confirmed trajectory are combined with the target's appearance feature distance and motion information to calculate a cost matrix for cascade matching. The specific process includes:
[0142] The cost matrix is calculated using the target's appearance features, distance, and motion information. The specific calculation method is as follows:
[0143] C=λA a +(1-λ)A m
[0144] Where C is the appearance feature distance cost A a And the cost of motion information A m The weighted sum, where λ is the weighting factor, and preferably λ is set to 0.98;
[0145] By combining the cost matrix with the detected target and the confirmed trajectory, the Hungarian algorithm is used for matching to obtain the matching trajectory, the mismatched trajectory, and the mismatched target.
[0146] In the above embodiments, preferably, the GIoU matching algorithm is expressed as:
[0147]
[0148] Where IoU represents the ratio of the intersection area to the union area of the two bounding boxes, A c U represents the area of the smallest bounding box that encloses the two bounding boxes, and U represents the total area covered by the two bounding boxes, which is the area of each bounding box plus the area of their intersection.
[0149] The IoU matching algorithm is expressed as:
[0150]
[0151] Where A∩B represents the area of the overlapping portion of bounding box A and bounding box B, and A∪B represents the total coverage area of bounding box A and bounding box B.
[0152] In the above embodiments, preferably, the Kalman filter algorithm combined with an adaptive noise scale is used to update the state of the matching trajectories obtained by cascaded matching, GIoU matching, and IoU matching. The specific process includes:
[0153] Based on the confidence level of the detection results, the measurement noise scale in the Kalman filter algorithm is adjusted. The noise covariance is calculated as follows:
[0154]
[0155] Among them, R k For the preset constant measurement noise covariance, C k The detection confidence value at state k;
[0156] The covariance matrix P and the observation matrix H, as well as the noise covariance matrix R, are estimated using prior estimation. k Calculate the Kalman gain matrix K k :
[0157]
[0158] The prior estimate x of the Kalman filter is projected onto the measurement space through the observation matrix H, and the residual y between the measured value z and the observed value z is calculated:
[0159]
[0160] Converted to matrix form:
[0161]
[0162] The predicted and measured values of the Kalman filter are calculated according to the Kalman gain K. k By combining the proportions, we obtain the posterior estimated state:
[0163]
[0164] Calculate the posterior covariance of the Kalman filter:
[0165]
[0166] Based on the calculation of the posterior estimated state and the posterior estimated covariance, the trajectory state at the current moment is determined, and the trajectory state is updated.
[0167] In the above embodiments, preferably, the specific process of deleting the mismatch trajectory obtained from IoU matching includes:
[0168] For the mismatch target in IoU matching, create a corresponding new trajectory;
[0169] For mismatch trajectories in IoU matching, delete the unconfirmed trajectories, delete the confirmed trajectories where the mismatch reaches the preset maximum mismatch count threshold, and retain the confirmed trajectories where the mismatch does not reach the preset maximum mismatch count threshold, and continue to try matching in subsequent frames.
[0170] like Figure 4 and Figure 5 As shown, according to the DeepSORT-based visible light video multi-target tracking method disclosed in the above embodiments, in order to evaluate and improve the multi-target tracking system, this invention utilizes the publicly available MOT17 dataset. MOT17 is a dataset for pedestrian tracking, containing 7 sequences, with 5316 frames used for training and 5919 frames used for testing. This dataset is widely accepted and used, providing researchers with a standard benchmark for evaluating the performance of multi-target tracking systems. Secondly, a visible light vehicle dataset is constructed using DarkLabel annotation software, focusing on cars traveling on highways. Further, DarkLabel annotation software is used to annotate the bounding boxes and category information of the targets. Next, regarding improvements to the DeepSORT model, this invention makes the following improvements and produces the following effects: A layer of GIoU matching is introduced, which improves the accuracy of multi-target tracking by addressing mismatched trajectories after cascaded matching. The Kalman filter algorithm is improved so that it can adaptively adjust the noise scale according to the target detection quality, reducing the interference of low-quality target detection results on the tracking system. Camera motion compensation technology was introduced to ensure the stability of the tracked target and reduce the impact of camera shake, thereby improving the robustness of the tracking system. When calculating the cost matrix, appearance feature distance and motion information were combined to reduce false matches caused by changes in appearance features or occlusion.
[0171] Finally, based on the data, the model testing and performance evaluation of the visible light video multi-target tracking method based on DeepSORT proposed in this invention can be performed. Specifically, experiments were conducted on the MOT17 validation set, and the tracking performance was evaluated using HOTA, MOTA, IDF1, IDs, and DetRe metrics. The evaluation results are shown in Table 1 below:
[0172] Table 1
[0173]
[0174] This invention also proposes a visible light video multi-target tracking system based on DeepSORT, characterized in that it applies the visible light video multi-target tracking method based on DeepSORT disclosed in any of the above embodiments, including:
[0175] The target detection module is used to acquire visible light video and use the YOLOv5 target detector to detect targets in the current frame of the visible light video and obtain target information.
[0176] The target prediction module is used to introduce a camera motion compensation method to correct the target's position information based on the camera's motion state, and to use a Kalman filter algorithm to predict the trajectory of all detected targets and determine whether the trajectory is in a confirmed or unconfirmed state.
[0177] The cascaded matching module is used to combine the detected target with the confirmed trajectory and the target's appearance features, distance, and motion information to calculate the cost matrix for cascaded matching.
[0178] The association matching module is used to perform GIoU matching for mismatched trajectories and mismatched targets in cascaded matching, and to perform IoU matching for mismatched trajectories and mismatched targets in GIoU matching.
[0179] The trajectory update module is used to update the state of the matched trajectories obtained by cascaded matching, GIoU matching and IoU matching by using the Kalman filter algorithm combined with an adaptive noise scale. It deletes the unconfirmed trajectories and confirmed trajectories that exceed the maximum number of mismatches in the mismatched trajectories obtained by IoU matching, and outputs the target information after the state update.
[0180] In the above embodiments, preferably, the target prediction module is specifically used for:
[0181] For visible light video, the camera's motion state is determined before processing each frame;
[0182] If camera motion is detected, a camera motion compensation method is introduced. Based on the enhanced correlation coefficient (ECC) method, image registration or alignment is performed on the offset or distortion caused by camera motion to correct the target position information.
[0183] The ECC method is expressed as follows:
[0184]
[0185] i w (p)=φ(i r ;p)
[0186] in, This indicates the untransformed reference image with a mean of 0. Indicates passing through φ(i) r The mean of the distorted image after transformation is 0, and φ represents an image transformation function controlled by parameter p.
[0187] Using the posterior estimate of k-1 from the previous time step The prior estimated state at time k is obtained by transforming the state transition matrix F.
[0188]
[0189] Estimate the covariance P using the posterior estimate of the previous time step k-1. k-1 Calculate the prior estimate covariance of the current time k.
[0190]
[0191] Where Q represents the process noise covariance matrix;
[0192] The state transition matrix F is:
[0193]
[0194] Based on the calculation of the prior estimated state and the prior estimated covariance, the estimated state and position of the trajectory at the next time step are determined.
[0195] Compare the matching results of the trajectory for a preset number of consecutive times. The trajectory that matches for the preset number of consecutive times is determined to be a confirmed trajectory; otherwise, it is determined to be an unconfirmed trajectory.
[0196] In the above embodiments, preferably, the trajectory update module is specifically used for:
[0197] Based on the confidence level of the detection results, the measurement noise scale in the Kalman filter algorithm is adjusted. The noise covariance is calculated as follows:
[0198]
[0199] Among them, R k For the preset constant measurement noise covariance, C k The detection confidence value at state k;
[0200] The covariance matrix P and the observation matrix H, as well as the noise covariance matrix R, are estimated using prior estimation. k Calculate the Kalman gain matrix K k :
[0201]
[0202] The prior estimate x of the Kalman filter is projected onto the measurement space through the observation matrix H, and the residual y between the measured value z and the observed value z is calculated:
[0203]
[0204] Converted to matrix form:
[0205]
[0206] The predicted and measured values of the Kalman filter are calculated according to the Kalman gain K. k By combining the proportions, we obtain the posterior estimated state:
[0207]
[0208] Calculate the posterior covariance of the Kalman filter:
[0209]
[0210] Based on the calculation of the posterior estimated state and the posterior estimated covariance, the trajectory state at the current moment is determined, and the trajectory state is updated.
[0211] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A visible light video multi-target tracking method based on DeepSORT, characterized in that, include: Input a visible light video, and use the YOLOv5 target detector to detect the target in the current frame of the visible light video to obtain target information; A camera motion compensation method is introduced to correct the position information of the target based on the camera's motion state, and a Kalman filter algorithm is used to predict the trajectory of all detected targets to determine whether the trajectory is in a confirmed or unconfirmed state. The detected target and the confirmed trajectory are combined with the target's appearance feature distance and motion information to calculate a cost matrix for cascade matching; For the mismatched trajectory and mismatched target in cascaded matching, the GIoU matching algorithm is used for GIoU matching, and for the mismatched trajectory and mismatched target in GIoU matching, the IoU matching algorithm is used for IoU matching. Using the Kalman filter algorithm combined with an adaptive noise scale, the matching trajectories obtained by the cascaded matching, the GIoU matching, and the IoU matching are updated in state. Unconfirmed trajectories and confirmed trajectories that exceed the maximum number of mismatches in the mismatch trajectories obtained by the IoU matching are deleted, and the target information after state update is output. The process of using the Kalman filter algorithm to predict the trajectory of all detected targets and determine the trajectory state as either confirmed or unconfirmed includes: Using the posterior estimate of k-1 from the previous time step By transforming the state transition matrix F, the prior estimated state at the current time k is obtained. : Estimate the covariance using the posterior of the previous time step k-1. Calculate the prior estimate covariance of the current time k. : in, Q Represents the process noise covariance matrix; The state transition matrix F is: Based on the calculation of the prior estimated state and the prior estimated covariance, the estimated state and position of the trajectory at the next time step are determined; Compare the matching results of the trajectory for a preset number of consecutive times, and determine that the trajectory that matches for the preset number of consecutive times is a confirmed trajectory; otherwise, it is determined to be an unconfirmed trajectory. The process of updating the state of the matching trajectories obtained from the cascaded matching, the GIoU matching, and the IoU matching using the Kalman filter algorithm combined with an adaptive noise scale includes: Based on the confidence level of the detection results, the measurement noise scale in the Kalman filter algorithm is adjusted. The noise covariance is calculated as follows: in, R k The noise covariance is measured as a preset constant. C k The detection confidence value at state k; Estimating the covariance matrix using priors And the observation matrix H, and the noise covariance matrix R k Calculate the Kalman gain matrix K k : The prior estimates of the Kalman filter The observation matrix H is projected onto the measurement space, and the result is calculated to correspond to the measured value. residual y : Converted to matrix form: The predicted and measured values of the Kalman filter are calculated according to the Kalman gain matrix. K k By combining the proportions, we obtain the posterior estimated state: Calculate the posterior covariance of the Kalman filter: Based on the calculation of the posterior estimated state and the posterior estimated covariance, the trajectory state at the current moment is determined, and the trajectory state is updated.
2. The visible light video multi-target tracking method based on DeepSORT according to claim 1, characterized in that, The introduced camera motion compensation method corrects the target's position information based on the camera's motion state. The specific process includes: For the visible light video, the motion state of the camera is determined before processing each frame; If the camera is detected to be in motion, a camera motion compensation method is introduced. Based on the enhanced correlation coefficient (ECC) method, image registration is performed on the offset or distortion caused by the camera motion to correct the position information of the target. The ECC method is expressed as follows: in, This indicates the untransformed reference image with a mean of 0. Indicates the process The transformed distorted image has a mean of 0. Represents a parameter p Controlled image transformation function.
3. The visible light video multi-target tracking method based on DeepSORT according to claim 1, characterized in that, The process of combining the detected target with the confirmed trajectory and the target's appearance feature distance and motion information to calculate the cost matrix for cascade matching includes: The cost matrix is calculated using the target's appearance feature distance and motion information. The specific calculation method is as follows: Where C is the appearance feature distance cost A a And the cost of motion information A m The weighted sum, where λ is the weighting factor; The cost matrix is combined with the detected target and the confirmed trajectory, and the Hungarian algorithm is used for matching to obtain the matching trajectory, the mismatched trajectory, and the mismatched target.
4. The visible light video multi-target tracking method based on DeepSORT according to claim 1, characterized in that, The GIoU matching algorithm is expressed as follows: in, IoU This represents the ratio of the intersection area to the union area of two bounding boxes. A c U represents the area of the smallest bounding box that encloses the two bounding boxes, and U represents the total area covered by the two bounding boxes, which is the area of each bounding box plus the area of their intersection. The IoU matching algorithm is expressed as follows: Where A∩B represents the area of the overlapping portion of bounding box A and bounding box B, and A∪B represents the total coverage area of bounding box A and bounding box B.
5. The visible light video multi-target tracking method based on DeepSORT according to claim 1, characterized in that, The specific process of deleting the mismatch trajectory obtained from the IoU matching includes: For the IoU matching mismatch target, create a corresponding new trajectory; For the mismatch trajectory of the IoU matching, the unconfirmed trajectory is deleted, the confirmed trajectory where the mismatch reaches the preset maximum mismatch number threshold is deleted, and the confirmed trajectory where the mismatch does not reach the preset maximum mismatch number threshold is retained, and matching is continued in subsequent frames.
6. A visible light video multi-target tracking system based on DeepSORT, characterized in that, The visible light video multi-target tracking method based on DeepSORT as described in any one of claims 1 to 5 includes: The target detection module is used to acquire visible light video and use the YOLOv5 target detector to detect targets in the current frame of the visible light video to obtain target information; The target prediction module is used to introduce a camera motion compensation method to correct the position information of the target according to the motion state of the camera, and to use the Kalman filter algorithm to predict the trajectory of all detected targets and determine the state of the trajectory as confirmed or unconfirmed. The cascaded matching module is used to combine the detected target with the confirmed trajectory and the target's appearance feature distance and motion information to calculate a cost matrix for cascaded matching; The association matching module is used to perform GIoU matching for mismatched trajectories and mismatched targets in cascaded matching, and to perform IoU matching for mismatched trajectories and mismatched targets in GIoU matching. The trajectory update module is used to update the state of the matching trajectories obtained by the cascaded matching, the GIoU matching, and the IoU matching using the Kalman filter algorithm combined with an adaptive noise scale. It deletes the unconfirmed trajectories and confirmed trajectories that exceed the maximum number of mismatches in the mismatch trajectories obtained by the IoU matching, and outputs the target information after the state update.
7. The visible light video multi-target tracking system based on DeepSORT according to claim 6, characterized in that, The target prediction module is specifically used for: For the visible light video, the motion state of the camera is determined before processing each frame; If the camera is detected to be in motion, a camera motion compensation method is introduced. Based on the enhanced correlation coefficient (ECC) method, image registration is performed on the offset or distortion caused by the camera motion to correct the position information of the target. The ECC method is expressed as follows: in, This indicates the untransformed reference image with a mean of 0. Indicates the process The transformed distorted image has a mean of 0. Represents a parameter p Controlled image transformation function; Using the posterior estimate of k-1 from the previous time step By transforming the state transition matrix F, the prior estimated state at the current time k is obtained. : Estimate the covariance using the posterior of the previous time step k-1. Calculate the prior estimate covariance of the current time k. : in, Q Represents the process noise covariance matrix; The state transition matrix F is: Based on the calculation of the prior estimated state and the prior estimated covariance, the estimated state and position of the trajectory at the next time step are determined; By comparing the matching results of the trajectory for a preset number of consecutive times, the trajectory that matches for the preset number of consecutive times is determined to be a confirmed trajectory; otherwise, it is determined to be an unconfirmed trajectory.
8. The visible light video multi-target tracking system based on DeepSORT according to claim 6, characterized in that, The trajectory update module is specifically used for: Based on the confidence level of the detection results, the measurement noise scale in the Kalman filter algorithm is adjusted. The noise covariance is calculated as follows: in, R k The noise covariance is measured as a preset constant. C k The detection confidence value at state k; Estimating the covariance matrix using priors And the observation matrix H, and the noise covariance matrix R k Calculate the Kalman gain matrix K k : The prior estimates of the Kalman filter The observation matrix H is projected onto the measurement space, and the result is calculated to correspond to the measured value. The residual y: Converted to matrix form: The predicted and measured values of the Kalman filter are calculated according to the Kalman gain matrix. K k By combining the proportions, we obtain the posterior estimated state: Calculate the posterior covariance of the Kalman filter: Based on the calculation of the posterior estimated state and the posterior estimated covariance, the trajectory state at the current moment is determined, and the trajectory state is updated.