DeepSORT pedestrian tracking method based on multi-feature space-time cooperative interaction

Abstract

The invention discloses a DeepSORT pedestrian tracking method based on multi-feature space-time cooperative interaction, and belongs to the field of computer vision and intelligent video analysis. The method comprises the following steps: acquiring and processing pedestrian data, constructing a target detection and feature extraction model, detecting a test set after training to generate a candidate box, extracting appearance features to construct a cost matrix, matching and updating a trajectory by using a Hungary algorithm, and finally outputting a visual tracking result. In the detection stage, a small target feature enhancement pyramid is designed to improve the small target detection precision, PSConv, Triplet Attention and DyHead are fused to construct a multi-dimensional feature interaction mechanism, and the scale adaptability and the anti-shielding capability are enhanced; in the tracking stage, an IAU module is embedded into an Re-ID branch of DeepSORT, feature discrimination is enhanced through space-time and channel feature dynamic modeling, and ID Switch is reduced. The method effectively improves the perception recognition capability of a multi-scale and strong-shielding target, guarantees the detection accuracy and tracking robustness in a complex environment, and has a good application deployment value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and intelligent video analysis, and particularly relates to a DeepSORT pedestrian tracking method based on multi-feature space-time collaborative interaction. BACKGROUND

[0002] Multi-object tracking is a key task in video analysis, which aims to detect and associate the continuous trajectories of the same object in a video sequence. Multi-object tracking is widely used in automatic driving, intelligent monitoring, sports analysis and other scenarios, and is of great significance for pedestrian behavior analysis and environmental understanding. However, there are still many challenges in dense crowd scenes: frequent target occlusion, small-scale target detection problem and similar appearance target identity confusion phenomenon, combined with detection noise introduced by fast target motion and complex background, which jointly restrict the tracking accuracy and robustness of MOT algorithm. Specifically, small targets are difficult to be fully perceived on low-resolution feature maps, resulting in missed detection, while occlusion superposition exacerbates ID Switch, ultimately affecting the continuity and integrity of the trajectory. Therefore, improving the representation ability of the detection model for multi-scale targets and the cross-frame feature discrimination robustness of the Re-ID module are the key to breaking through the current performance bottleneck.

[0003] In recent years, the tracking-by-detection (TBD) paradigm has made significant progress, but the performance improvement of tracking under this paradigm largely depends on large detectors such as YOLOX-X, which brings the problem of large model and weak migration ability while improving accuracy, in addition, most mainstream TBD methods use independent pedestrian re-identification (Re-ID) modules for target association. Typical such as DeepSORT introduces appearance features extracted by CNN for data association, combined with motion prediction, significantly improves the matching robustness in occlusion scenes. However, this type of method still has two common problems: first, the appearance encoder is mostly a static CNN model, which is difficult to model the space-time context and body part dependency between targets; second, the feature expression ability is insufficient, and ID confusion and drift still easily occur under the conditions of long-time occlusion or multi-target interaction. SUMMARY

[0004] The purpose of the present application is to provide a DeepSORT pedestrian tracking method based on multi-feature space-time collaborative interaction, which solves the technical problem of low tracking accuracy caused by small target missed detection and target occlusion in existing multi-object tracking.

[0005] The technical solution adopted by the present application is a DeepSORT pedestrian tracking method based on multi-feature space-time collaborative interaction, comprising the following steps: S1: Obtain pedestrian data, clean and standardize the data, and obtain a data set; S2: preprocessing the data set, dividing the training set, the validation set and the test set; S3: constructing a target detection model; S4: constructing a feature extraction model; S5: training and optimizing the target detection model and the feature extraction model with the training set and the validation set; S6: inputting the test set into the target detection model to generate a candidate frame; S7: the feature extraction model extracts the appearance features of the candidate region, calculates the cosine similarity and the Mahalanobis distance to construct a cost matrix; S8: matching the cost matrix with the Hungarian algorithm and updating the trajectory; S9: outputting the trajectory information and generating a visual tracking video.

[0006] The application also has the characteristics of: S1 is specifically: Obtaining target pedestrian image data and video sequence data thereof, cleaning and standardizing the image, if it is unmarked data, using Labelme or other marking tools to complete the marking of the boundary box and the identity information, if it is marked data, converting it into YOLO format and MOT format.

[0007] S2 is specifically: S2.1: scaling the images in the data set to a uniform size and performing image enhancement; S2.2: dividing the preprocessed data set into a training set, a validation set and a test set according to the ratio of 8:1:1.

[0008] S3 is specifically: S31: introducing a small target feature enhancement module SOEP in the YOLO detector: through the SPDConv module, the P2 layer features are down-sampled and rearranged, and the processed P2 features are fused into the P3 layer; a multi-branch module with a fusion CSP structure and an OmniKernel mechanism is constructed, and the fused features are divided into a global branch, a large branch and a local branch; S32: constructing a multi-dimensional feature interaction enhancement mechanism DMFI and deploying it in the Backbone and Neck structures in turn: integrating the PSConv module in the backbone network to perform multi-directional asymmetric convolution; introducing the Triplet Attention module to capture the interaction dependence between the channel and the spatial dimension; replacing the detection head with the DyHead module, which integrates scale perception, spatial perception and task perception mechanisms, to realize accurate perception of multi-scale targets and task adaptation; S33: integrate the SOEP and DMFI modules to construct a YOLO-MFIE target detection model, optimize the configuration of the number of channels and the size of the convolution kernel, and output the candidate target bounding box, class and confidence score in each frame of image.

[0009] S4 specifically is: In the Re-ID branch of DeepSORT, a space-time channel interaction enhancement module IAU is introduced and embedded into the feature extraction network: IAU is composed of STIAU and CIAU modules in series, wherein STIAU realizes intra-frame and inter-frame consistency modeling by dividing the key parts of the human body, and CIAU models the semantic dependence relationship of channel dimension, IAU is embedded into the last bottleneck structure of ResNet50 in a residual manner.

[0010] S5 specifically is: S51: After data preprocessing, the YOLO-MFIE detection model and the IAU enhanced Re-ID model are initially trained using the training set: the detection model is jointly trained using classification loss, bounding box regression loss and DFL loss, the Re-ID model is jointly trained using cross-entropy loss, triplet loss and mask loss, the model parameters are iteratively updated through the back propagation algorithm, the training loss and validation accuracy are recorded, if the loss decreases and tends to be stable in multiple consecutive training periods, and the validation set performance does not fluctuate significantly, it is determined that the model is preliminarily converged; S52: On the basis of initial training convergence, the batch size, initial learning rate and training rounds are dynamically adjusted: the batch size is set according to the calculation resources and training stability, the initial learning rate adopts a phased decay strategy, and the training rounds are set according to the validation set index trend; After each training round, the precision, recall, AP and mAP of the detection model, and the Rank-1 and mAP of the Re-ID model are calculated; S53: continuously train and evaluate until the main evaluation indicators are stable, when the mAP of the detection model and the mAP and Rank-1 indicators of the Re-ID model fluctuate less than ±0.1% in multiple consecutive validation periods, and the index improvement amplitude is less than the preset threshold after adjusting the hyperparameters, the training is terminated, and the weight files of the YOLO-MFIE model and the IAU enhanced Re-ID model are finally saved.

[0011] S6 specifically is: The video stream in the actual application scenario is input frame by frame into the YOLO-MFIE detection model to generate the bounding box position, class label and corresponding confidence score of the candidate target in each frame of image; according to the pre-set confidence threshold, the inference output is filtered, only the detection box higher than the threshold is retained, and the filtered bounding box information and confidence are passed to the subsequent tracking processing.

[0012] S7 specifically is: After completing the target detection, the candidate bounding box region reserved in each frame is cropped, and the processed image region is input into the IAU enhanced Re-ID model trained in S5 to extract the appearance embedding vector of each candidate target; the embedding vector and the historical trajectory feature are calculated by cosine similarity, the motion state of the historical trajectory is predicted by using the Kalman filter, and the Mahalanobis distance is calculated based on the predicted state and the current detection box position.

[0013] S8 is specifically: The Hungarian algorithm is used to perform optimal allocation on the joint cost matrix generated in S7 to realize the association of the current frame detection result and the historical trajectory; for the matched trajectory, the state estimation, the covariance matrix and the appearance embedding feature are updated; for the un-matched detection target, if it is a new appearing object, a new trajectory identifier is assigned; for the historical trajectory with the number of consecutive mismatched frames exceeding the preset threshold, it is marked as inactivated and the tracking is terminated; the Kalman filter is used to predict the position of each active trajectory in the next frame.

[0014] S9 is specifically: After completing the target association and trajectory management, all active state trajectories are traversed, the identifier, the current frame bounding box coordinates and the confidence score of each trajectory are output, and written into a tracking result file; the trajectory identifier and the bounding box are drawn on the original video frame to generate a visual tracking video stream; based on the bounding box and the trajectory ID sequence output by each frame, the target motion trajectory data is summarized to record the appearance time and the disappearance time of each trajectory; the summary result generates a final report to provide the appearance frame and the disappearance frame of the trajectory, the motion path data and the overall tracking performance evaluation.

[0015] The beneficial effects of the present application are: In the detection stage, a small target feature enhancement pyramid (SOEP) is designed to enhance the expression ability of shallow features to improve the detection accuracy of small targets; at the same time, a multi-dimensional feature interaction enhancement mechanism is constructed by fusing PSConv, Triplet Attention and DyHead to enhance the adaptability of the model to scale changes and the anti-occlusion ability. In the tracking stage, the IAU module is embedded into the Re-ID branch of DeepSORT, the spatio-temporal and channel features are dynamically modeled to enhance the feature discrimination, and the ID Switch caused by similar appearance is effectively reduced. In summary, the present application effectively improves the perception and recognition ability of multi-scale and strongly occluded targets, ensures the detection accuracy and tracking robustness in complex environments, and has good practical application and deployment value. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the DeepSORT pedestrian tracking method based on multi-feature spatio-temporal collaborative interaction of the present application; Figure 2 Schematic diagram of the DeepSORT framework in the DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction of the present invention; Figure 3 This is a schematic diagram of the network structure of YOLO-MFIE in the DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction of the present invention; Figure 4 This is a schematic diagram of the CSP-OmniKernel module structure in the DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction of the present invention; Figure 5 This is a schematic diagram of the IAU structure of the Re-ID branch in the DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction in the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] Example 1 like Figures 1-5 As shown, the DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction disclosed in this embodiment includes the following steps: S1: Obtain pedestrian data, clean and standardize the data to obtain a dataset; S2: Preprocess the dataset and divide it into training set, validation set and test set; S3: Build target detection model; S4: Build feature extraction model; S5: Use the training set and validation set to train and fine-tune the object detection model and feature extraction model; S6: Input the test set into the target detection model to generate candidate boxes; S7: The feature extraction model extracts the appearance features of the candidate regions, calculates the cosine similarity and Mahalanobis distance to construct the cost matrix; S8: Use the Hungarian algorithm to match the cost matrix and update the trajectory; S9: Output trajectory information and generate visual tracking video.

[0019] Example 2 Based on Example 1, S1 is specifically: Obtain target pedestrian image data and its video sequence data, clean and standardize the image, and if it is unannotated data, use Labelme or other annotation tools to complete the annotation of the boundary box and identity information; if it is annotated data, convert it into YOLO format and MOT format to adapt to subsequent training input.

[0020] Specifically, obtain pedestrian image dataset and video sequence data, and the data sources can include public datasets (such as CrowdHuman, Market1501, MOT17, and MOT20) or self-collected data. The original data is filtered, cleaned, and formatted: the unannotated image data is manually annotated with boundary boxes and identity IDs using Labelme and other tools; the existing annotated XML, COCO or MOT format annotation files are converted into the required format for YOLO or Re-ID training (such as txt / json / .npy); redundant or low-quality samples are removed to ensure annotation accuracy and data balance.

[0021] Embodiment 3 On the basis of embodiment 1, S2 is specifically: S2.1: Scale the images in the dataset to a uniform size and perform image enhancement; S2.2: Divide the preprocessed dataset into a training set, a validation set, and a test set according to a ratio of 8:1:1, and ensure the balance of sample distribution to support model training and generalization ability evaluation.

[0022] Specifically, the processed dataset is divided into a training set, a validation set, and a test set according to a ratio of 8:1:1 to ensure balanced distribution of data in time series and target categories. The following preprocessing operations are performed on the images: scale the image to a uniform size (such as 640x640) while maintaining the aspect ratio; image enhancement (random cropping, flipping, color disturbance, etc.); uniformly adjust the input image of the Re-ID module to 256x128 size; generate training cache (such as.npy index or mosaic mixed image) for the training set to speed up the training process.

[0023] Embodiment 4 On the basis of embodiment 1, S3 is specifically: constructing a target detection model YOLO-MFIE. Design a small target feature enhancement module (SOEP) based on YOLOv10n, combine PSConv, Triplet Attention and DyHead to build a multi-dimensional feature interaction mechanism, and improve the adaptability of the model to scale changes, pose changes and occlusion scenarios.

[0024] S31: A small target feature enhancement module SOEP is introduced in the YOLO detector to improve the response ability of shallow features to small targets. First, the SPDConv module is used to downsample and rearrange the P2 layer features to extract the edge and structural features of small targets, and the processed P2 features are fused into the P3 layer. On this basis, a multi-branch module with CSP structure and OmniKernel mechanism is constructed, which divides the fused features into global branch, large branch and local branch, and realizes multi-granularity feature decoupling and expression through different receptive field feature paths, thereby enhancing the representation ability of small target features in complex background and improving the detection performance while keeping the computational complexity controllable.

[0025] S32: A multi-dimensional feature interaction enhancement mechanism DMFI is proposed to enhance the modeling ability of the model for multi-scale and multi-pose targets. The DMFI module is deployed in the Backbone and Neck structures in sequence, and the overall feature optimization link follows the "feature enhancement-context modeling-task adaptation" principle. First, the PSConv module is integrated in the backbone network to enhance the feature direction perception ability through multi-direction asymmetric convolution; second, the Triplet Attention module is introduced to model the long-distance dependence between channel and spatial dimensions, and to improve the context consistency of the target region; finally, the DyHead module is replaced in the detection head to fuse scale perception, spatial perception and task perception mechanisms, dynamically adjust the importance weight of multi-task features, and improve the consistency and robustness of the detection results.

[0026] S33: After integrating the SOEP and DMFI modules described above, the overall structure of the YOLO-MFIE target detection model is constructed, and the channel number and convolution kernel size are optimized according to the lightweight design principle. The model parameter quantity and computational complexity are controlled to meet the real-time and resource constraint requirements in actual deployment while ensuring the detection accuracy. The model output is the candidate target bounding box, class and confidence score in each frame of image, which provides high-quality detection input for the subsequent tracking module.

[0027] Embodiment 5 On the basis of embodiment 4, S4 is specifically: introducing a space-time channel interaction enhancement module IAU into the Re-ID branch of the DeepSORT tracking framework, and embedding it into the feature extraction network to improve the target appearance modeling capability and the robustness of cross-frame matching. The IAU module is composed of a STIAU module and a CIAU module in series, the STIAU is used to model the context relationship of the target in the time dimension and the spatial structure, and the intra-frame and inter-frame consistency modeling is realized by dividing the key parts of the human body; the CIAU is used to model the semantic dependency relationship in the channel dimension, and the expression ability of the weak part features is improved. The IAU is embedded in the last bottleneck structure of ResNet50 in a residual manner, which effectively enhances the discriminability of the Re-ID features while ensuring the computational efficiency of the model, and improves the continuous tracking performance of the target under complex occlusion and pose change conditions.

[0028] Embodiment 6 On the basis of embodiment 5, S5 is specifically: training the training set input detector and Re-ID branch respectively, using classification loss, bounding box regression loss, triplet loss and other indicators for supervised training, and monitoring the loss curve to determine whether it converges. The learning rate, batch size and iteration number are dynamically adjusted during the training process until the accuracy on the validation set is stable.

[0029] S51: After data preprocessing is completed, the training set is used to initially train the YOLO-MFIE detection model and the IAU enhanced Re-ID model respectively. The detection model uses a target detection loss function, including classification loss, bounding box regression loss and DFL loss; the Re-ID model uses a joint training method of cross-entropy loss, triplet loss and mask loss. In the training process, the model parameters are iteratively updated through the back propagation algorithm, and the training loss and validation accuracy changes are continuously recorded. If the loss decreases and tends to be stable in multiple consecutive training periods, and the validation set performance has no obvious fluctuation, it is considered that the model has been preliminarily converged; S52: On the basis of initial training convergence, the training hyperparameters are dynamically adjusted, including batch size, initial learning rate and training rounds. The batch size is set according to the computing resources and training stability; the initial learning rate uses a phased decay strategy, which is gradually reduced in the middle and late stages of training to improve the generalization ability; the training rounds are set according to the trend of the validation set indicators. After each training round is completed, the precision, recall, average precision (AP) and mAP of the detection model, and the recognition rate (Rank-1) and average precision (mAP) of the Re-ID model are calculated, which are used as reference for hyperparameter adjustment; S53: Continue model training and evaluation until the main evaluation index is stable, and terminate the training when the improvement of the index by continuously adjusting the hyperparameters is less than the preset threshold. The stability criterion is that the mAP of the detection model and the mAP and Rank-1 index of the Re-ID model fluctuate by less than ±0.1% in consecutive multiple validation periods. Finally, the weights of the trained YOLO-MFIE model and the IAU enhanced Re-ID model are saved for subsequent video target tracking process loading. In this embodiment, the batch size during training of the detection model is 8, the number of training rounds is 200, the initial learning rate is 0.0001, and the input image size is 640x640. The batch size for training the Re-ID model is 64, the number of training rounds is 150, and the initial learning rate is 0.00035.

[0030] Embodiment 7 On the basis of embodiment 6, S6 is specifically: The video stream in the actual application scenario is input frame by frame into the system, and the YOLO-MFIE detection model saved in S5 is loaded for forward inference to generate the bounding box position, class label and corresponding confidence score of the candidate target in each frame of image. The method can filter the inference output according to the preset confidence threshold (for example, 0.5), and only keep the detection boxes higher than the threshold, and pass the filtered bounding box information and confidence to the subsequent tracking processing.

[0031] Embodiment 8 On the basis of embodiment 7, S7 is specifically: After completing target detection, crop and adjust all retained candidate bounding box regions in each frame to 256x128, and then input the processed image region into the IAU enhanced Re-ID model trained in S5 to extract the appearance embedding vector of each candidate target. The cosine similarity between the embedding vector and the target feature stored in the historical tracking trajectory is calculated, and the Kalman filter is used to predict the motion state of the historical trajectory, and the Mahalanobis distance is calculated based on the predicted state and the current detection box position. The cosine similarity and the Mahalanobis distance are combined according to the preset weight to generate a joint cost matrix for data association, which is called by the Hungarian matching algorithm.

[0032] Embodiment 9 On the basis of embodiment 8, S8 is specifically: The Hungarian algorithm is used to optimally distribute the joint cost matrix generated in S7 to realize the association of the current frame detection result and the historical trajectory. For the matched trajectory, the state estimation, the covariance matrix and the appearance embedding feature are updated to maintain the continuity of the trajectory. For the un-matched detection target, it is judged whether it belongs to a new object. If so, a new trajectory identifier is assigned to it. For the historical trajectory which has not been matched for a long time, when the number of consecutive mismatched frames exceeds a preset threshold (for example, 30 frames), the trajectory is marked as inactive and the tracking is terminated. Meanwhile, the Kalman filter is used to predict the position of each active trajectory in the next frame to improve the smoothness and robustness of the tracking.

[0033] Embodiment 10 Based on the embodiment 9, S9 is specifically: After completing the target association and trajectory management, the system traverses all trajectories in the active state, and for each trajectory, the trajectory identifier (ID), the bounding box coordinates of the current frame and the confidence score are output in turn, and the above information is written into the tracking result file (for example, MOT format). At the same time, the trajectory identifier and the corresponding bounding box are optionally drawn on the original video frame in the form of superimposed text and graphics to generate a visual tracking video stream with real-time labeling. Finally, based on the output bounding box and trajectory ID sequence of each frame, the complete target motion trajectory data is summarized and formed, and the appearance time and disappearance time of each trajectory are recorded, providing structured data support for subsequent behavior analysis, event detection or trajectory statistics. The above results are summarized to generate a final report, which provides the appearance frame and disappearance frame of each trajectory, motion path data, and outputs the overall tracking performance evaluation for subsequent behavior analysis, event detection or system optimization.

[0034] To verify the effectiveness of the SOEP module and the DMFI structure proposed in the application in the target detection task, an ablation experiment is constructed based on the commonly used detection evaluation indicators, and each sub-module is introduced under the unified model framework to evaluate its influence on Precision (P), Recall (R) and average precision (AP). The experimental results are shown in Tables 1-3.

[0035] In Table 1, Baseline refers to the unimproved YOLOv10 detector structure; after gradually introducing SOEP and DMFI, YOLO-MFIE detector is finally formed, in which P, R and AP are increased to 83.7%, 67.2% and 77.7% respectively. Table 2 verifies the gain effect of SPDConv and CSP-OmniKernel in the SOEP module; Table 3 analyzes the combination mode of PSConv, Triplet Attention and DyHead in DMFI, and the results show that the cooperation of the three modules in DMFI can significantly improve the detection stability and task adaptation ability, and finally realize the efficient detection of small targets and multi-scale targets in complex scenes.

[0036] Table 1

[0037] Table 2

[0038] Table 3

[0039] To verify the contribution of the proposed YOLO-MFIE detector and IAU enhanced Re-ID module to the multi-target tracking performance, module-level ablation experiments were conducted on the MOT17 (see Table 4) and MOT20 (see Table 5) datasets. The following three groups of models were constructed in the DeepSORT framework: 1) the original structure without introducing MFIE and IAU; 2) only introducing the MFIE detection module; and 3) introducing the complete structure of MFIE and IAU module. The evaluation indicators of this experiment include MOTA, HOTA, AssA, and IDF1, and the results are shown in the table.

[0040] On the MOT17 dataset, to further verify the effectiveness of the proposed module, the authors conducted a system comparison experiment on DeepSORT under different combination configurations, as shown in Table 4.

[0041] The original DeepSORT model on MOT17 has an IDF1 of 56.72, a MOTA of 48.90, an ID Switch number of 461, and an AssA of 52.13. The overall performance is average, but there are certain matching errors and identity drift phenomena in scenes with frequent target interactions.

[0042] When only the IAU module is introduced, the identity discrimination ability of the model is marginally improved, with IDF1 increasing to 56.85 and AssA increasing to 52.70, and ID Switch number decreasing to 435. This indicates that the module plays a positive role in modeling the fine-grained appearance between targets, especially in the case of occlusion and similar appearance targets, and can enhance discrimination.

[0043] Further introduction of the MFIE module slightly improves the detection accuracy of the model, with MOTP reaching 85.92. Although IDF1 decreases slightly to 55.98, ID Switch number is significantly reduced to 364, indicating that MFIE has certain contribution in small target detection and appearance feature enhancement, which helps to reduce matching errors caused by inaccurate detection.

[0044] When the MFIE and IAU modules are introduced simultaneously, the model achieves optimal performance in all indicators: IDF1 is improved to 62.16, an increase of 5.44 over the original model; MOTA is improved to 50.71; AssA is significantly improved to 58.54, and IDSwitch is reduced to 218, a decrease of more than 52%. HOTA is also improved to 51.36, reflecting the optimization of overall matching quality.

[0045] In summary, the MFIE and IAU modules complement each other in detection enhancement and spatiotemporal feature modeling, significantly improving the multi-target tracking performance of the model on the MOT17 dataset, especially in identity consistency and occlusion robustness.

[0046] On the MOT20 dataset, the original DeepSORT model performs poorly in dense occlusion scenarios, with a MOTA of 54.46%, an IDF1 of 47.78%, an ID Switch number of 10377, and an AssA indicator of only 29.73%, reflecting its limited target matching accuracy and identity preservation ability in complex scenarios.

[0047] After introducing the IAU module, the model's identity discrimination performance is significantly improved, with IDF1 increasing to 52.67% and AssA increasing to 35.03%, and ID Switch number decreasing to 7695, a reduction of 25.88%. The above results show that the IAU module effectively enhances the ability to distinguish similar local features by introducing cross-channel and spatiotemporal cross-perception mechanisms, improving the stability of identity preservation in occlusion and target interaction scenarios.

[0048] After further introducing the MFIE module, the model's detection accuracy is optimized, with MOTP increasing to 76.08, AssA further increasing to 35.28%, and ID Switch number decreasing to 7577, indicating that the module has a positive effect on strengthening small target feature expression and complex pose modeling. However, due to the lack of improvement in appearance modeling ability, the overall identity recognition performance (such as IDF1) has limited improvement.

[0049] When the MFIE and IAU modules are integrated at the same time, the model achieves the optimal performance in various indicators: the IDF1 is improved to 54.11%, which is improved by 6.33% compared with the original model; the AssA is improved to 40.02%, and the improvement amplitude reaches 10.29%; the IDSwitch number is reduced to 7422, and the reduction amplitude reaches 28.47%; in addition, the MOTA is improved to 59.98%, and the HOTA reaches 40.33%. The results fully verify the complementary advantages of MFIE and IAU in the two dimensions of detection enhancement and feature modeling, and the synergistic optimization significantly improves the overall tracking performance and identity discrimination robustness of the model in the high-density occlusion environment.

[0050] The above results show that the detection and Re-ID double-branch mechanism proposed in the application still has high target detection and continuous tracking ability under complex background and multiple occlusion conditions, and has good engineering adaptability and application prospect.

[0051] Table 4

[0052] Table 5

[0053] Finally, it should be noted that in this document, relational terms such as first and second and the like can merely be used to distinguish one entity or action from another, without necessarily requiring or implying that there are any such actual relationship or order between or among the entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0054] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0055] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction, characterized by: The following steps are involved: S1: Obtain pedestrian data, clean and standardize the data to obtain a dataset; S2: Preprocess the dataset and divide it into training set, validation set and test set; S3: Build target detection model; S4: Build feature extraction model; S5: Use the training set and validation set to train and fine-tune the object detection model and feature extraction model; S6: Input the test set into the target detection model to generate candidate boxes; S7: The feature extraction model extracts the appearance features of the candidate regions, calculates the cosine similarity and Mahalanobis distance to construct the cost matrix; S8: Use the Hungarian algorithm to match the cost matrix and update the trajectory; S9: Output trajectory information and generate visual tracking video.

2. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 1 is characterized in that: S1 is specifically: Obtain target pedestrian image data and video sequence data, clean and standardize the images, and use Labelme or other annotation tools to label the bounding box and identity information if the data is unlabeled. If the data is labeled, convert it to YOLO format and MOT format.

3. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 1 is characterized in that: S2 is specifically: S2.1: Scale the images in the dataset to a uniform size and perform image augmentation; S2.2: Divide the preprocessed dataset into training set, validation set and test set in a ratio of 8:1:

1.

4. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 1 is characterized in that: S3 specifically: S31: Introducing the Small Object Feature Enhancement Module (SOEP) into the YOLO detector: Downsampling and rearranging the P2 layer features through the SPDConv module, and fusing the processed P2 features into the P3 layer; Constructing a multi-branch module that fuses the CSP structure and the OmniKernel mechanism, dividing the fused features into global branches, large branches, and local branches; S32: Construct a multi-dimensional feature interaction enhancement mechanism (DMFI) and deploy it sequentially in the Backbone and Neck structures: Integrate the PSConv module in the backbone network to perform multi-directional asymmetric convolution; introduce the Triplet Attention module to capture the interactive dependencies between channels and spatial dimensions; replace the detection head with the DyHead module, integrating scale-aware, spatial-aware, and task-aware mechanisms to achieve accurate perception of multi-scale objects and task adaptation; S33: Integrates the SOEP and DMFI modules to build the YOLO-MFIE object detection model, optimizes the number of channels and convolution kernel size, and outputs the candidate object bounding box, category, and confidence score in each frame.

5. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 4 is characterized in that: S4 is specifically: The spatiotemporal channel interaction enhancement module IAU is introduced into the Re-ID branch of DeepSORT and embedded in the feature extraction network: IAU is composed of STIAU and CIAU modules in series, where STIAU achieves intra-frame and inter-frame consistency modeling by dividing the key parts of the human body, CIAU models the semantic dependency of the channel dimension, and IAU is embedded in the last bottleneck structure of ResNet50 in a residual manner.

6. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 5 is characterized in that: S5 is specifically: S51: After data preprocessing is completed, the training set is used to perform initial training on the YOLO-MFIE detection model and the IAU-enhanced Re-ID model. The detection model is trained jointly with classification loss, bounding box regression loss, and Descriptive Language (DFL) loss, while the Re-ID model is trained jointly with cross-entropy loss, triplet loss, and mask loss. The model parameters are iteratively updated using the backpropagation algorithm, and the training loss and validation accuracy are recorded. If the loss decreases and stabilizes over multiple consecutive training cycles and the validation set performance does not fluctuate significantly, the model is considered to have initially converged. S52: Based on the initial training convergence, the batch size, initial learning rate, and number of training rounds are dynamically adjusted: the batch size is set based on computing resources and training stability, the initial learning rate adopts a staged decay strategy, and the number of training rounds is set based on the trend of the validation set indicators; after each round of training, the precision, recall, AP, and mAP of the detection model, as well as the Rank-1 and mAP of the Re-ID model, are calculated; S53: Continue training and evaluation until the main evaluation indicators are stable. When the fluctuation of the mAP of the detection model and the mAP and Rank-1 indicators of the Re-ID model in multiple consecutive verification cycles is less than ±0.1%, and the improvement of the indicators after adjusting the hyperparameters is less than the preset threshold, the training is terminated. Finally, the weight files of the YOLO-MFIE model and the IAU enhanced Re-ID model are saved.

7. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 6 is characterized in that: S6 specifically: The video stream in the actual application scenario is input into the YOLO-MFIE detection model frame by frame to generate the bounding box position, category label and corresponding confidence score of the candidate target in each frame image; the inference output is filtered according to the preset confidence threshold, only the detection box above the threshold is retained, and the filtered bounding box information and confidence score are passed to the subsequent tracking processing.

8. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 7 is characterized in that: S7 specifically: After completing the object detection, the candidate bounding box area retained in each frame is cropped, and the processed image area is input into the IAU enhanced Re-ID model trained in S5 to extract the appearance embedding vector of each candidate object; The cosine similarity between the embedding vector and the historical trajectory features is calculated, the motion state of the historical trajectory is predicted using a Kalman filter, and the Mahalanobis distance is calculated based on the predicted state and the current detection box position; the cosine similarity and Mahalanobis distance are combined according to preset weights to generate a joint cost matrix.

9. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 8 is characterized in that: S8 specifically: The Hungarian algorithm is used to optimally allocate the joint cost matrix generated in S7 to achieve the association between the current frame detection results and historical trajectories. For the successfully matched trajectories, their state estimation, covariance matrix and appearance embedding features are updated. For unmatched detection targets, if they are newly appeared objects, new trajectory identifiers are assigned. Historical trajectories with a number of consecutive mismatch frames exceeding a preset threshold are marked as inactive and tracking is terminated. The Kalman filter is used to predict the position of each activated trajectory in the next frame.

10. The DeepSORT pedestrian tracking method based on multi-feature spatiotemporal collaborative interaction according to claim 9 is characterized in that: S9 specifically: After completing target association and trajectory management, it traverses all active trajectories, outputs the identifier of each trajectory, the coordinates of the current frame's bounding box, and the confidence score, and writes them into the tracking result file; the trajectory identifier and bounding box are drawn on the original video frame to generate a visual tracking video stream; Based on the bounding box and trajectory ID sequence output for each frame, the target motion trajectory data is aggregated and the appearance and disappearance time of each trajectory is recorded; The results are summarized into a final report that provides the track's appearance and disappearance frames, motion path data, and overall tracking performance evaluation.

Images