A traffic multi-target tracking method and system based on MF-DeepSORT

Abstract

The present invention discloses a traffic multi-target tracking method and system based on MF-DeepSORT, including: using the YOLOv5 detection model to perform target detection and obtain the target area; extracting the deep convolution features and directional gradient histogram features of the target area, and inputting them into the cascade matching stage for tracking and matching; performing intersection-and-union correlation matching on the target area that failed to match successfully and the tracking target. Compared with the current DeepSORT tracking algorithm, the present invention improves the tracking accuracy performance of the multi-target tracking algorithm while increasing a small amount of calculation, and simultaneously obtains higher multi-target tracking accuracy and lower tracking ID switching times.

Description

Technical Field

[0001] This invention relates to the technical field of multi-target tracking, and in particular to a traffic multi-target tracking method and system based on MF-DeepSORT. Background Technology

[0002] Multi-object tracking technology is an important research direction in the field of computer vision, with wide applications in pedestrian detection, intelligent traffic monitoring, and autonomous driving. Traffic multi-object tracking can match spatial vehicle target information across video sequences to track vehicles in the video, record their motion trajectories, and subsequently analyze vehicle behavior. Therefore, research on traffic multi-object tracking is of great significance and value.

[0003] Early traffic multi-target tracking algorithms mainly involved template matching using pixel features, drawing vehicle outlines for tracking calculations, or calculating the overlap area of ​​vehicles between two frames. Traditional vehicle tracking methods are greatly limited by the simplicity of feature semantics and the insufficient computational power of matching algorithms. They can only be used in ideal and limited tracking scenarios, are easily affected by environmental factors such as occlusion and lighting, have poor tracking performance, are prone to tracking errors, and do not have high practical application value.

[0004] With the significant success of deep learning and object detection technologies in computer vision, numerous studies have proposed mainstream two-stage multi-object tracking frameworks based on object detection and tracking-matching. These frameworks include a detection stage using object detection models and a tracking-matching stage based on Kalman filtering and Hungarian matching algorithms. Researchers typically begin by detecting objects in each frame of a video sequence using object detection algorithms to obtain target regions and their location information. Then, they measure the motion features and depth appearance features of the target regions and the predicted motion features of the tracked targets to construct a similarity matrix. Finally, they use an association matching algorithm to solve the target association between consecutive frames, achieving matching and tracking of the target regions. This type of method is often used for pedestrian multi-object tracking under complex tracking conditions and achieves good tracking results. However, existing two-stage multi-object tracking frameworks based on object detection and tracking-matching have low tracking accuracy in traffic multi-object tracking tasks. Measuring motion features based on Mahalanobis distance is prone to tracking matching errors, and relying solely on the appearance similarity matrix constructed from depth convolution features cannot achieve accurate matching. These object tracking algorithms have limitations when facing complex traffic multi-object tracking scenarios. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a traffic multi-target tracking method and system based on MF-DeepSORT to solve the technical problems mentioned in the background. In the target detection stage, this method uses YOLOv5 to detect the target region of each input image frame, extracting the depth convolutional features and HOG features of the target region. In the tracking and matching stage, the IOU between the target region and the tracked target is used to measure motion features, and the depth feature distance and HOG feature distance are fused to measure appearance features. The two form an association matrix for association matching. This method outperforms the DeepSORT algorithm on traffic multi-target datasets in terms of MOTA and IDsw, effectively improving multi-target tracking performance.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A traffic multi-object tracking method based on MF-DeepSORT, the method comprising:

[0008] Step S1: For a video sequence, target detection is performed using a target detection model to obtain its target region;

[0009] Step S2: For the target region obtained in step S1, extract its depth convolution features and HOG features, and then perform a cascaded matching operation on the target region, its features, and the tracking target in the confirmed state to achieve tracking matching. The cascaded matching operation is to perform cascaded tracking matching on the target region and the tracking target.

[0010] Step S3: Perform IOU association matching operation on the target areas that failed to match in step S2 and the remaining tracking targets that failed to match.

[0011] Step S4: Input the target region and the target after tracking and matching into the tracker for tracking, and update the tracker, including: updating the Kalman filter matrix of the matched target and making predictions, creating tracking targets for the unmatched target regions, and deleting the unmatched tracking targets.

[0012] Furthermore, in step S1, a pre-trained YOLOv5s is used as the object detection model.

[0013] Furthermore, step S2 specifically includes:

[0014] Step S201: For the target region obtained in step S1, extract its HOG features, perform image preprocessing on the target region, and then input the image patch after normalization into the convolutional neural network to extract convolutional depth features.

[0015] Step S202: Perform a cascaded matching operation, matching the unupdated frame count attribute of the tracked target with the target region in ascending order.

[0016] Furthermore, step S202 specifically includes:

[0017] Step S2021: First, for the target region and the tracking target, calculate the minimum cosine distance between the corresponding deep convolutional features and HOG features. Then, based on the corresponding minimum cosine distance, obtain the deep convolutional feature distance matrix and the HOG feature distance matrix. Then, fuse the two distance matrices to obtain the fused appearance feature distance matrix. Finally, obtain the appearance feature threshold matrix from the fused appearance feature distance matrix.

[0018] Step S2022: Use IOU distance to measure the motion characteristics of the target area and the tracked target to obtain the motion feature distance matrix, and obtain the motion feature threshold matrix from the threshold of the motion feature distance matrix;

[0019] Step S2023: Combine the appearance feature threshold matrix obtained in step S2021 and the motion feature threshold matrix obtained in step S2022 to obtain a threshold matrix. Then, apply this threshold matrix to the appearance feature distance matrix to obtain the correlation matrix. Finally, use the Hungarian matching rule to match the correlation matrix to obtain the correct matching correspondence between the tracking target and the target area.

[0020] Furthermore, step S3 includes:

[0021] For target regions and tracking targets that failed to match in the cascade matching phase, as well as unconfirmed tracking targets, perform IOU association matching on the target regions and tracking targets to obtain the IOU distance matrix.

[0022] An IOU threshold is set to eliminate incorrect matching relationships. Then, the Hungarian matching rule is used to match the IOU distance matrix to obtain the correct matching correspondence between the remaining tracked targets and the remaining target regions.

[0023] Furthermore, the IOU threshold is specifically 0.7.

[0024] Furthermore, step S4 includes:

[0025] Step S401: For tracked targets whose status has been confirmed but have not been matched, increment their unupdated frame count attribute by 1, and then determine whether the unupdated frame count after the increment is greater than the threshold. If it is greater, delete the tracked targets whose status has been confirmed but have not been matched.

[0026] Step S402: Update the Kalman filter for the successfully matched tracking target and predict the position of the tracking target in the next frame image;

[0027] Step S403: For a tracked target that has been successfully matched but is in an unconfirmed state, increment its matching frame count attribute by 1, and then determine whether the number of matching frames after the increment is greater than the initial matching frame count threshold. If it is greater, mark the tracked target as confirmed.

[0028] Step S404: For the target areas that failed to match, create tracking targets and mark them as unconfirmed.

[0029] Step S405: Write the frame number of the current frame, the tracking target ID, and the Kalman filter prediction region location into the tracking result.

[0030] Furthermore, in step S1, when performing detection, the target region confidence screening threshold output by the detection model is 0.3; the screened target regions are then subjected to non-maximum suppression screening to exclude redundant target regions with IOU greater than 0.5.

[0031] A traffic multi-object tracking system based on MF-DeepSORT, the system comprising:

[0032] The target detection module performs target detection on a video sequence using a target detection model to obtain the target region.

[0033] The cascaded matching module extracts the deep convolutional features and HOG features of the target region input by the target detection module, and then performs a cascaded matching operation on the target region, its features, and the tracking target in the confirmed state to achieve tracking matching. The cascaded matching operation is to perform cascaded tracking matching on the target region and the tracking target.

[0034] The IOU association matching module performs IOU association matching operations on the target regions that failed to match and the remaining tracking targets that failed to match, based on the output of the cascade matching module.

[0035] The tracker update module takes the target region and the target after tracking matching and inputs them into the tracker for tracking, and updates the tracker, including: updating the Kalman filter matrix of the matched target and making predictions, creating tracking targets for the unmatched target regions, and deleting the unmatched tracking targets.

[0036] The beneficial effects of this invention are:

[0037] This invention, based on the DeepSORT multi-target tracking framework, introduces HOG features to characterize target information, while MF-DeepSORT uses IOU to measure spatial location information. Compared to the original DeepSORT multi-target tracking algorithm, this invention improves the tracking accuracy of multi-target tracking algorithms while maintaining a slight increase in computational cost, achieving higher multi-target tracking accuracy (MOTA) and lower tracking ID switching count (IDsw). In the field of deep learning, this invention demonstrates that the improved MF-DeepSORT tracking algorithm can enhance the accuracy of multi-target tracking in the transportation sector, providing valuable reference for other researchers. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart illustrating a traffic multi-target tracking method based on MF-DeepSORT provided in Example 1;

[0039] Figure 2 This is a flowchart illustrating the target detection stage provided in Example 1;

[0040] Figure 3 This is a flowchart illustrating the cascading matching stage provided in Example 1;

[0041] Figure 4 This is a schematic diagram of the IOU association matching process provided in Example 1;

[0042] Figure 5 This is a flowchart illustrating the tracker update phase provided in Example 1;

[0043] Figure 6 This is a comparison of the multi-target tracking performance of the method of the present invention and the DeepSORT algorithm in Example 1;

[0044] Figure 7 This is a comparison of the multi-target tracking performance of the method of the present invention and the DeepSORT algorithm in Example 1. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1

[0047] See also Figures 1-7This embodiment provides a traffic multi-target tracking method based on MF-DeepSORT. The specific process of this method is as follows: Figure 1 As shown, the method specifically includes:

[0048] Step S1: For a video sequence, target detection is performed using a target detection model to obtain the target region;

[0049] Specifically, in this embodiment, step S1 includes:

[0050] The specific process of object detection is as follows: Figure 2 As shown, this embodiment uses the YOLOv5 detection model, and it is a pre-trained YOLOv5s. The image size of the input model is 640×640.

[0051] The target region confidence screening threshold output by the detection model is 0.3, which can ensure a sufficient number of target regions in each frame of image, while also eliminating most falsely detected target regions.

[0052] The selected target regions are then subjected to non-maximum suppression filtering to exclude redundant target regions with an IOU greater than 0.5.

[0053] Finally, the detection results of each frame are output and input into the tracking and matching stage along with the existing tracking target.

[0054] Step S2: For the target region obtained in step S1, extract its depth convolution features and HOG features, and then perform a cascaded matching operation on the target region, its features, and the tracking target in the confirmed state to achieve tracking matching. The cascaded matching operation is to perform cascaded tracking matching on the target region and the tracking target.

[0055] Specifically, in this embodiment, step S2 is the cascade matching stage, and the specific process of this stage is as follows: Figure 3 As shown; step S2 above specifically includes the following sub-steps:

[0056] Step S201: For the target region obtained in step S1, extract its HOG features and perform image preprocessing on the target region. Change the size of the target region image block to 256×128, and then input the image block into the convolutional neural network after normalization to extract convolutional depth features.

[0057] Step S202, Cascaded Matching Stage: The unupdated frame count attribute of the tracked target is matched with the target region in ascending order;

[0058] Step S203: First, for the target region and the tracking target, calculate the minimum cosine distance between the corresponding deep convolutional features and HOG features. Then, based on the corresponding minimum cosine distance, obtain the deep convolutional feature distance matrix and the HOG feature distance matrix. Then, fuse the two distance matrices to obtain the fused appearance feature distance matrix. Finally, obtain the appearance feature threshold matrix from the fused appearance feature distance matrix.

[0059] Step S204: Use IOU distance to measure the motion characteristics of the target area and the tracked target to obtain the motion feature distance matrix, and obtain the motion feature threshold matrix from the threshold of the motion feature distance matrix;

[0060] Step S205: Combine the appearance feature threshold matrix obtained in step S203 and the motion feature threshold matrix obtained in step S204 to obtain a threshold matrix. Then, apply this threshold matrix to the appearance feature distance matrix to obtain the correlation matrix. Finally, use the Hungarian matching rule to match the correlation matrix to obtain the correct matching correspondence between the tracking target and the target area.

[0061] Step S3: Perform IOU association matching on the target areas that failed to match in step S2 and the remaining tracking targets that failed to match.

[0062] Specifically, in this embodiment, the specific process of the IOU association and matching stage is as follows: Figure 4 As shown, specifically including:

[0063] For target regions and tracking targets that failed to match in the cascade matching stage, as well as unconfirmed tracking targets, IOU association matching is performed between the target regions and tracking targets to obtain the IOU distance matrix. Incorrect matching relationships are eliminated by setting the IOU threshold to 0.7. The Hungarian matching rule is then used to match the IOU distance matrix to obtain the correct matching correspondence between the remaining tracking targets and the remaining target regions.

[0064] Step S4: Input the target region and the tracking target after the tracking matching in step S3 into the tracker for tracking, and update the tracker, including: updating the Kalman filter matrix of the matched tracking target and making predictions, creating tracking targets for the unmatched target regions, and deleting tracking targets that have not been matched for a long time.

[0065] Specifically, in this embodiment, step S4 is the tracker update stage, and its specific process is as follows: Figure 5 As shown, specifically including:

[0066] Step S401: For tracked targets whose status has been confirmed but have not been matched, increment their unupdated frame count attribute by 1, and then determine whether the unupdated frame count after the increment is greater than the threshold. If it is greater, delete the tracked targets whose status has been confirmed but have not been matched.

[0067] Step S402: Update the Kalman filter for the successfully matched tracking target and predict the position of the tracking target in the next frame image;

[0068] Step S403: For a tracked target that has been successfully matched but is in an unconfirmed state, increment its matching frame count attribute by 1, and then determine whether the number of matching frames after the increment is greater than the initial matching frame count threshold. If it is greater, mark the tracked target as confirmed.

[0069] Step S404: For the target areas that failed to match, create tracking targets and mark them as unconfirmed.

[0070] Step S405: Write the frame number of the current frame, the tracking target ID, and the Kalman filter prediction region location into the tracking result.

[0071] It should be noted that the multi-target tracking framework provided in this embodiment is not limited to the PyTorch framework, as long as it can track multi-target tracking datasets and perform performance evaluation.

[0072] Finally, the tracking performance of the multi-target tracking algorithm was evaluated on a traffic multi-target tracking dataset. Specifically, based on the tracking results output by the multi-target tracking algorithm and the labeled data in the dataset, the multi-target tracking accuracy (MOTA), multi-target tracking precision (MOTP), ID switching count (IDsw), and the number of tracks where most targets were tracked (MT) were calculated to evaluate the tracking performance of the multi-target tracking algorithm. Higher values ​​for MOTA, MOTP, and MT, and lower values ​​for IDsw, all indicate better performance of the multi-target tracking algorithm.

[0073] like Figure 6 The figure shows a comparison of the multi-target tracking performance of the method of this invention with the current DeepSORT multi-target tracking algorithm. From... Figure 6 The test results show that the method of this invention achieves better MOTA and MOTP values ​​than the DeepSORT algorithm on the traffic multi-target tracking dataset, indicating that the method of this invention achieves better tracking accuracy and precision. The MOTA is 4.839% higher than DeepSORT, reaching 62.071%, and the IDsw value also decreased from 34 to 2, indicating that the algorithm of this invention has achieved a significant improvement in the accuracy of multi-target tracking and the ability to maintain tracking.

[0074] like Figure 7The figure shown is a comparison of the multi-target tracking performance of the method of this invention and the DeepSORT algorithm. MF-DeepSORT has better tracking and holding capabilities. As can be seen from frames 42 to 43 of Task 1, in the DeepSORT algorithm, the tracking ID of the same vehicle (pointed by the blue arrow, ID=30) changes to 22 in the next frame, while in the MF-DeepSORT algorithm, the tracking ID of the same vehicle remains unchanged at 24.

[0075] MF-DeepSORT exhibits better environmental robustness. In tracking task 6, from frame 30 to frame 31, the DeepSORT algorithm was affected by changes in tree shadows and lighting. The same vehicle, indicated by the red arrow, was assigned two tracking IDs, and both tracking regions differed significantly from the vehicle's actual outline. MF-DeepSORT, however, maintained normal tracking for the same vehicle without being affected. This demonstrates that MF-DeepSORT, by measuring motion features through IOU, can significantly avoid spatial mismatches, maintain tracking status, and reduce tracking ID switching. Furthermore, fusing appearance feature measurement with HOG feature distance improves the accuracy of appearance feature measurement and enhances tracking precision.

[0076] Example 2

[0077] This embodiment provides a traffic multi-target tracking system based on MF-DeepSORT, the system comprising:

[0078] The target detection module performs target detection on a video sequence using a target detection model to obtain the target region.

[0079] The cascaded matching module extracts the deep convolutional features and HOG features of the target region input by the target detection module, and then performs a cascaded matching operation on the target region, its features, and the tracking target in the confirmed state to achieve tracking matching. The cascaded matching operation is to perform cascaded tracking matching on the target region and the tracking target.

[0080] The IOU association matching module performs IOU association matching operations on the target regions that failed to match and the remaining tracking targets that failed to match, based on the output of the cascade matching module.

[0081] The tracker update module takes the target region and the target after tracking matching and inputs them into the tracker for tracking, and updates the tracker, including: updating the Kalman filter matrix of the matched target and making predictions, creating tracking targets for the unmatched target regions, and deleting the unmatched tracking targets.

[0082] In summary, this invention provides a traffic multi-object tracking method and system based on MF-DeepSORT. Compared with the current DeepSORT tracking algorithm, it improves the tracking accuracy performance of multi-object tracking algorithms with a small increase in computation, while achieving higher multi-object tracking accuracy (MOTA) and lower tracking ID switching count (ID Switch, IDsw).

[0083] Any aspects of this invention not described in detail are well-known to those skilled in the art.

[0084] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A traffic multi-target tracking method based on MF-DeepSORT, characterized in that: The method comprises: Step S1, performing target detection on a video sequence by using a target detection model to obtain its target area; Step S2: extracting the deep convolution features and HOG features of the target area obtained in step S1, and then performing a cascade matching operation on the target area and its features and the tracking target in the confirmed state to achieve tracking matching, wherein the cascade matching operation is to perform cascade tracking matching on the target area and the tracking target; Step S3, performing an IOU association matching operation on the target area that failed to match successfully in step S2 and the remaining tracking targets that failed to match successfully; Step S4, inputting the target area and the tracking target after the tracking matching is performed into the tracker for tracking, and updating the tracker, which includes: updating the Kalman filter matrix of the matching tracking target, and making a prediction, creating a tracking target for the target area that fails to match, and deleting the tracking target that fails to match; The step S2 specifically includes: Step S201: extract the HOG features of the target area obtained in step S1, perform image preprocessing on the target area, and then normalize the image block and input it into the convolutional neural network to extract convolution depth features; Step S202, performing a cascade matching operation, matching the unupdated frame number attributes of the tracked target with the target area in order from small to large; The step S202 specifically includes: Step S2021: First, for the target area and the tracked target, the minimum cosine distances of the corresponding deep convolution features and HOG features are calculated, and then the deep convolution feature distance matrix and the HOG feature distance matrix are obtained according to the corresponding minimum cosine distances. Then, the two distance matrices are fused to obtain a fused appearance feature distance matrix, and finally, the appearance feature threshold matrix is ​​obtained from the fused appearance feature distance matrix. Step S2022: Use the IOU distance to measure the motion features of the target area and the tracked target to obtain a motion feature distance matrix, and obtain a motion feature threshold matrix from the motion feature distance matrix threshold; Step S2023, combining the appearance feature threshold matrix obtained in step S2021 and the motion feature threshold matrix obtained in step S2022 to obtain a threshold matrix, and then using the threshold matrix to act on the appearance feature distance matrix to obtain a correlation matrix, and finally using the Hungarian matching rule to match the correlation matrix to obtain the correct matching correspondence between the tracking target and the target area.

2. The traffic multi-target tracking method based on MF-DeepSORT according to claim 1, characterized in that: In step S1, the pre-trained YOLOv5s is used as the target detection model.

3. The traffic multi-target tracking method based on MF-DeepSORT according to claim 1, characterized in that: The step S3 comprises: Perform IOU association matching on the target area and tracking target that failed to match successfully in the cascade matching stage and the unconfirmed tracking target, and obtain the IOU distance matrix. The IOU threshold is set to exclude incorrect matching relationships, and then the Hungarian matching rule is used to match the IOU distance matrix to obtain the correct matching correspondence between the remaining tracked targets and the remaining target areas.

4. The traffic multi-target tracking method based on MF-DeepSORT according to claim 3 is characterized in that: The IOU threshold is specifically 0.

7.

5. The traffic multi-target tracking method based on MF-DeepSORT according to claim 4 is characterized in that: The step S4 comprises: Step S401: for the tracking targets that have been confirmed and failed to match, the unupdated frame number attribute is increased by 1, and then it is determined whether the unupdated frame number is greater than the threshold after the increase. If it is greater, the tracking targets that are greater than the loss time threshold are deleted; the tracking targets that are unconfirmed and failed to match are deleted; Step S402, updating the Kalman filter of the successfully matched tracking target to predict the position of the tracking target in the next frame image; Step S403: for the tracking target that is successfully matched but in an unconfirmed state, its matching frame number attribute is increased by 1, and then it is determined whether its matching frame number is greater than the initial matching frame number threshold after the increase. If so, the tracking target is marked as a confirmed state; Step S404: create a tracking target for the target area that cannot be matched, and mark it as unconfirmed; Step S405: write the frame number of the current frame, the tracking target ID, and the Kalman filter prediction area position into the tracking result.

6. The method for tracking multiple traffic targets based on MF-DeepSORT according to claim 5, characterized in that: In step S1, when performing detection, the confidence screening threshold of the target area output by the detection model is 0.3; the screened target area is then screened by non-maximum suppression to exclude redundant target areas with IOU greater than 0.

5.

7. A traffic multi-target tracking system based on MF-DeepSORT for implementing the traffic multi-target tracking method according to any one of claims 1 to 6, characterized in that: The system includes: An object detection module performs object detection on a video sequence by using an object detection model to obtain an object area; A cascade matching module extracts deep convolution features and HOG features of the target area passed in by the target detection module, and then performs a cascade matching operation on the target area and its features as well as the tracking target in the confirmed state to achieve tracking matching. The cascade matching operation is to adopt cascade tracking matching for the target area and the tracking target; An IOU association matching module performs an IOU association matching operation on the target area that failed to match successfully and the remaining tracking targets that failed to match successfully based on the output of the cascade matching module; The tracker update module inputs the target area and tracking target after tracking matching into the tracker for tracking, and updates the tracker, which includes: updating the Kalman filter matrix of the matching tracking target and making predictions, creating tracking targets for the unmatched target areas, and deleting the unmatched tracking targets.

Images