A pedestrian multi-target tracking method and system with improved appearance feature processing

Abstract

The present invention discloses a pedestrian multi-target tracking method and system with improved appearance feature processing, which belongs to the fields of computer vision technology and image processing technology, and includes the following steps: S101, constructing a multi-target tracking model: building a CATrack tracker, which includes a detector model and an association model; wherein the association model processes the output of the detector model in four stages, namely: feature acquisition of the target appearing in the current frame, similarity calculation between the target appearing in the current frame and the target historical trajectory, bipartite graph matching between the target appearing in the current frame and the target historical trajectory, and updating the trajectory feature using the matching result; S102, training of the detector model; S103, training of the association model and hyperparameter adjustment; S104, execution of the multi-target tracking task. Through the above-mentioned method, the present invention has excellent tracking performance in complex scenes with dense targets and has good robustness.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and image processing, and in particular to a pedestrian multi-target tracking method and system with improved appearance feature processing. Background Art

[0002] Multi-target tracking (MOT) is an important basic task in the field of computer vision and has a wide range of applications in many fields, such as autonomous driving, video surveillance, and smart transportation. Therefore, achieving high-precision and fast multi-target tracking has important application value. However, the application scenarios in the real world are relatively complex, with characteristics such as target appearance feature degradation, target density, and frequent occlusion, which brings challenges to the research and practical application of multi-target tracking methods.

[0003] At present, a mainstream paradigm for implementing multi-target tracking methods is detection-based tracking. This type of method divides multi-target tracking into two steps: target detection and data association. Among them, target detection aims to locate potential targets of interest in the current frame, which is usually implemented by a neural network target detection model; the association task is achieved by calculating the similarity between the detection results and the target's historical trajectory, and performing bipartite graph matching based on the similarity. Due to the rapid progress in the level of detectors, research in the field of pedestrian multi-target tracking mainly focuses on how to achieve high-quality and robust association.

[0004] In order to achieve high-precision association between detection results and target historical trajectories, two types of association bases are usually considered: target motion state and target appearance features. For example, in terms of motion state, Kalman filtering is usually used to model target motion, and the predicted motion state is used to calculate the motion similarity between the detection result (for example, position relationship and speed direction consistency using IoU as an indicator). In terms of appearance features, neural networks are usually used to extract ReID features from the bounding box corresponding to the detection result, and the feature distance (usually using cosine distance) is calculated with the appearance features stored in the trajectory as the basis for association. At the same time, after successful association, the corresponding motion state and appearance features are detected, which are used to update the Kalman filter parameters of the trajectory and update the appearance features of the trajectory according to the exponential moving average (EMA) state transition formula. However, there are many challenging and complex scenes in real scenes.

[0005] Among them, tracking in dense scenes containing a large number of targets is currently a difficulty in multi-target tracking:

[0006] (1) On the one hand, since the movements of objects in the scene will affect each other, the interaction between a large number of objects also makes the motion pattern more complex.

[0007] (2) On the other hand, interference between highly dense objects will cause the extracted appearance features to have more noise and deteriorate in the accumulation of time series.

[0008] (3) In addition, long-tail situations such as occlusion will be more frequent in this scenario, resulting in uneven quality of the extracted features, which will interfere with tracking.

[0009] (4) Insufficient tracking accuracy: Traditional models do not perform well in tracking in complex scenarios because they do not fully consider the differences in feature distribution and feature quality of targets in different situations.

[0010] (5) Real-time problem: Many high-accuracy tracking algorithms are computationally complex and cannot meet the needs of real-time tracking.

[0011] (6) Poor environmental adaptability: Traditional algorithms have limited generalization capabilities in different scenarios, especially in complex environments with dense targets.

[0012] Due to the above characteristics, highly dense scenes will cause the performance of the tracker to degrade more easily. Therefore, it is an urgent practical problem to improve the target feature model and the association strategy to achieve a higher quality and more robust pedestrian target tracking method.

[0013] Based on this, the present invention designs a pedestrian multi-target tracking method and system with improved appearance feature processing to solve the above problems. Summary of the invention

[0014] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method and system for multi-target pedestrian tracking with improved appearance feature processing.

[0015] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0016] A pedestrian multi-target tracking method with improved appearance feature processing comprises the following steps:

[0017] S101. Build a multi-target tracking model: Use the deep learning framework PyTorch to build a CATrack tracker, which consists of a detector model and an association model, and is used to process the input video frame by frame.

[0018] The association model processes the output of the detector model in four stages, namely: feature acquisition of the target appearing in the current frame, similarity calculation between the target appearing in the current frame and the target's historical trajectory, bipartite graph matching between the target appearing in the current frame and the target's historical trajectory, and trajectory feature update using the matching results;

[0019] S102, training of the detector model: using a video dataset of the target scene to train the detector model;

[0020] S103, training and hyperparameter adjustment of the association model: using the video dataset of the target scene to train the association model, and adjusting the hyperparameters of the association model in the video of the target scene according to the scene characteristics;

[0021] S104, execution of the multi-target tracking task: input the target scene test video data into the trained and hyperparameter-adjusted CATrack tracker, perform the pedestrian multi-target tracking task, calculate the multi-target tracking performance index and save the tracking result video containing the target ID and bounding box.

[0022] Furthermore, in step S101, the detector model adopts the deep neural network target detection model YOLOX; the association model includes an appearance feature extraction sub-model, and the appearance feature extraction sub-model adopts the SBS-50 model.

[0023] Furthermore, in step S101, the feature acquisition of the target includes: acquiring the target motion state, confidence and appearance features; the similarity calculation between the target appearing in the current frame and the target historical trajectory includes: motion similarity calculation, using the settings in the baseline method Hybrid-SORT; appearance similarity calculation, extracting the appearance features of the potential target in the current frame and calculating the cosine similarity with the trajectory appearance features.

[0024] Furthermore, in step S101, when the association model processes the appearance features, the state of the trajectory is divided according to the association situation and the position situation: for the trajectory that has been successfully associated in the previous frame and has just been initialized, its association state is set to activated, and for other trajectories that have not been successfully associated, the association state is set to inactivated; when processing the detection of association with the trajectory in the inactivated state, the detected potential targets are divided according to the position of the target: for any target existing in the frame, the left and right sides of the image are respectively shrunk inward by a distance w of the width of the target bounding box, and the upper and lower sides of the image are respectively shrunk inward by a distance h of the height of the target bounding box to obtain a potential occlusion area, and the other parts are regarded as normal areas; for targets whose target bounding boxes completely fall into the potential occlusion area, they are judged as occluded, and for targets whose target bounding boxes overlap with the normal area, they are judged as non-occluded.

[0025] Furthermore, the association model adopts a confidence-based selective feature update module, namely, a CASFUM module, which includes a selective appearance state update strategy for filtering ambiguous trajectories and a confidence-based adaptive EMA state transition formula.

[0026] Furthermore, the selective appearance state update strategy performs differentiated processing according to the trajectory situation: for trajectories that are in an active state or in an inactive state but not completely in the potential occlusion area, normal appearance feature updates are performed after their association is successful, while for inactive occlusion area trajectories, that is, trajectories that were not successfully associated in the previous frame and are completely in the potential occlusion area, they are not used for the update of trajectory appearance features;

[0027] For the successfully matched trajectory, the adaptive EMA state transition formula that considers both the trajectory confidence and the potential target confidence is used for update. The calculation formula is:

[0028] ;

[0029] in, is the updated trajectory appearance feature, is the trajectory appearance feature before updating, is the appearance feature extracted from the potential target detection box matching the trajectory, is the updated trajectory confidence, is the trajectory confidence before updating; α d is the adaptive inertia weight, which is calculated as:

[0030] ;

[0031] in, α is the initial inertia weight parameter set manually, c i and c det are the confidence of trajectory i and the confidence of the detection result matching it, β is the acceptance threshold parameter used to filter noise, σ is a c i and c det The degree of adjustment is linearly related to the average c i and c det If they are equal, they are zero.

[0032] Furthermore, the association model uses an AAAM correction module when calculating appearance similarity;

[0033] First, two additional appearance features of the trajectory are stored: the appearance feature corresponding to the last successful detection of the target and the average appearance feature of all historical observations of the target;

[0034] Next, the cosine distance between the appearance features corresponding to the last successful detection of the target and the appearance features of the potential target in the current frame is calculated. and the absolute value of the difference between the corresponding confidence levels ; At the same time, calculate the cosine distance between the average appearance features of all historical observations of the target and the appearance features of the potential target in the current frame and the absolute value of the difference between the corresponding confidence levels ;

[0035] For each trajectory-potential target pair (i, j), the trajectory feature with the smallest confidence difference between the last associated feature and the historical average feature is used to construct the candidate template confidence distance matrix And the candidate template appearance feature distance matrix , the calculation formula is:

[0036]

[0037] Adaptive EMA state transition formula is used to update the appearance features and confidence of the trajectory to obtain the confidence distance matrix And the appearance feature distance matrix ;

[0038] Compare candidate template confidence distance matrix Distance matrix with trajectory confidence The size of each element in , by selecting the appearance feature distance corresponding to the smaller confidence distance to construct the final appearance distance matrix , the calculation formula is:

[0039] .

[0040] Furthermore, the association model uses the motion model in the baseline method Hybrid-SORT to calculate motion similarity;

[0041] Use Kalman filtering to model and predict the motion state of the target. The state vector of Kalman filtering is:

[0042] ;

[0043] Where (u, v) is the coordinate of the center point of the target, a is the area of ​​the bounding box, c is the confidence of the target box, and r is the aspect ratio of the bounding box; They are The velocity component of ; Calculate the motion similarity distance of the trajectory-potential target pair (i, j) , using the following formula:

[0044] ;

[0045] in, is the HMIoU spatial position intersection-over-union distance in Hybrid-SORT, is the velocity direction consistency distance in Hybrid-SORT, is the absolute value of the difference in confidence distances, and λ1, λ2, and λ3 are three manually set weight parameters.

[0046] Furthermore, λ1, λ2, and λ3 are set to 1.0, 0.2, and 1.0, respectively.

[0047] In order to better achieve the purpose of the present invention, the present invention also provides a pedestrian multi-target tracking system with improved appearance feature processing, comprising:

[0048] A multi-target tracking model building module, which is used to build a CATrack tracker using the deep learning framework PyTorch. The CATrack tracker consists of a detector model and an association model, which is used to process the input video frame by frame.

[0049] The association model processes the output of the detector model in four stages, namely: feature acquisition of the target appearing in the current frame, similarity calculation between the target appearing in the current frame and the target's historical trajectory, bipartite graph matching between the target appearing in the current frame and the target's historical trajectory, and trajectory feature update using the matching results;

[0050] A detector model training module for training a detector model using a video dataset of a target scene;

[0051] An association model training and hyperparameter adjustment module, used to train an association model using a video dataset of a target scene, and to adjust hyperparameters of the association model in the video of the target scene according to scene characteristics;

[0052] The multi-target tracking execution module is used to input the target scene test video data into the trained and hyperparameter-adjusted CATrack tracker to perform the pedestrian multi-target tracking task, calculate the multi-target tracking performance indicators and save the tracking result video containing the target ID and bounding box.

[0053] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention has higher performance than the baseline method (Hybrid-SORT-ReID) and other advanced methods in multiple evaluation indicators on the MOT20 dataset with highly dense targets and frequent occlusions, indicating that it has excellent tracking performance in complex scenes with dense targets.

[0054] 2. The method of the present invention has a good ability to maintain the target ID on the MOT20 dataset, and has good robustness in scenes with dense targets and frequent occlusions.

[0055] 3. The method of the present invention also shows higher performance advantages than the baseline method (Hybrid-SORT-ReID) and other advanced methods in multiple evaluation indicators on the MOT17 dataset with low target density, demonstrating the good generalization ability and universality of the method of the present invention in different application scenarios.

[0056] 4. The present invention verifies the effectiveness of the proposed selective appearance state update strategy (SU), adaptive EMA state transfer formula (AEMA), and AAAM correction module (AAAM) through ablation experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A flowchart of a pedestrian multi-target tracking method with improved appearance feature processing according to the present invention;

[0059] Figure 2 A schematic diagram of dividing detected potential targets in the present invention;

[0060] Figure 3 A schematic diagram of a pedestrian multi-target tracking method with improved appearance feature processing according to the present invention;

[0061] Figure 4 This is a visualization diagram of the effect of the present invention when processing scenes with dense targets and frequent occlusions. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Embodiment 1: In some embodiments, please refer to the drawings of the specification Figure 1, a pedestrian multi-target tracking method with improved appearance feature processing, comprising the following steps:

[0064] S101. Build a multi-target tracking model: Use the deep learning framework PyTorch to build a CATrack tracker (CATrack for short). The CATrack tracker consists of a detector model and an association model, which is used to process the input video frame by frame.

[0065] Among them, the detector model adopts the deep neural network target detection model YOLOX, which is used to output the bounding boxes of all potential targets appearing in the current frame;

[0066] The association model processes the output of the detector model in four stages: feature acquisition of the target in the current frame, similarity calculation between the target in the current frame and the target's historical trajectory, bipartite graph matching between the target in the current frame and the target's historical trajectory, and trajectory feature update using the matching results. The association model includes an appearance feature extraction sub-model (appearance feature extractor), which uses the SBS-50 model from FastReID (a Python toolkit from the paper (Fastreid: A pytorch toolbox for general instance re-identification)).

[0067] S102, training of detector model: use the video dataset of the target scene to train the detector model of CATrack, and iteratively update the parameters of the network according to the back-propagation principle to enable the model to more accurately identify and locate the target.

[0068] S103, training and hyperparameter adjustment of the association model: using the video dataset of the target scene to train the appearance feature extraction sub-model, so that the model can obtain more discriminative target features, iteratively updating the parameters of the network according to the back-propagation principle, and adjusting the hyperparameters of the association model of CATrack in the video of the target scene according to the scene characteristics, so as to improve the tracking accuracy and adaptability;

[0069] S104, execution of the multi-target tracking task: input the target scene test video data into the trained and hyperparameter-adjusted CATrack tracker, perform the pedestrian multi-target tracking task, calculate the multi-target tracking performance index and save the tracking result video containing the target ID (unique identifier of the target in digital form) and the bounding box.

[0070] Embodiment 2: In some embodiments, as a preferred embodiment of the present invention, the feature acquisition of the target includes: acquiring the target motion state, confidence and appearance features; the similarity calculation between the target appearing in the current frame and the target historical trajectory includes: motion similarity calculation, using the settings in the baseline method Hybrid-SORT (combining the Kalman filter prediction results to calculate the highly corrected intersection-over-combination ratio, speed direction consistency, and confidence consistency), and the similarity calculation also includes appearance similarity calculation: extracting the appearance features of the potential target in the current frame and the trajectory appearance features to calculate the cosine similarity;

[0071] Taking Hybrid-SORT as the basic method as an example, the similarity calculation between the target and the target's historical trajectory includes three types of motion consistency information: the highly corrected intersection-over-union ratio HMIoU, the velocity direction consistency ROCM, and the confidence method consistency; and one type of appearance consistency information: the consistency information obtained by correcting the cosine similarity between the trajectory history appearance features updated by the CASFUM module and the new target appearance features through the AAAM correction module.

[0072] like Figure 2 As shown, when the association model processes the appearance features, the state of the track is divided according to the association situation and the position situation: for the track that has been successfully associated in the previous frame and has just been initialized, its association state is set to activated, and for other tracks that have not been successfully associated, the association state is set to inactivated. When processing the detection associated with the track in the inactivated state (potential targets in the new frame), these detected potential targets are divided according to the position of the target: for any target existing in the frame, the left and right sides of the image are respectively shrunk inward by a distance w of the width of the target bounding box, and the upper and lower sides of the image are respectively shrunk inward by a distance h of the height of the target bounding box to obtain a potential occlusion area, and the other parts are regarded as normal areas; for targets whose target bounding boxes completely fall into the potential occlusion area, they are judged as occluded, and for targets whose target bounding boxes overlap with the normal area, they are judged as non-occluded; according to the different activation and occlusion states of the targets, the appearance branch and feature update module of the present invention perform differential processing on the targets;

[0073] The association model adopts a confidence-based selective feature update module (CASFUM module). The CASFUM module includes a selective appearance state update strategy (SU) for filtering ambiguous trajectories, which is used to filter the noise caused by low-quality features, and a confidence-based adaptive EMA state transfer formula (AEMA), which considers both the detection results and the confidence of the trajectory when updating the trajectory appearance embedding.

[0074] The selective appearance state update strategy (SU) performs differentiated processing according to the trajectory situation: for trajectories that are in an active state or in an inactive state but do not completely fall in the potential occlusion area, normal appearance feature updates are performed after their association is successful, while for inactive occlusion area trajectories, that is, trajectories that have not been successfully associated in the previous frame and are completely in the potential occlusion area, they are not used for updating the appearance features of the trajectory.

[0075] For the successfully matched trajectories, a confidence-based adaptive EMA state transfer formula (AEMA) that considers both the trajectory confidence and the potential target confidence is used for update. The calculation formula is:

[0076] ;

[0077] in, is the updated trajectory appearance feature, is the trajectory appearance feature before updating, is the appearance feature extracted from the potential target detection box matching the trajectory, is the updated trajectory confidence, is the trajectory confidence before updating; α d is the adaptive inertia weight, which is calculated as:

[0078] ;

[0079] in, α is the initial inertia weight parameter set manually, c i and c det are the confidence of trajectory i and the confidence of the detection result matching it, β is the manually set acceptance threshold parameter for filtering noise, σ is a c i and c det The degree of adjustment is linearly related to the average c i and c det When they are equal, they are taken as zero. At the same time, σ Will also be β The present invention is tested on MOT20 and MOT17. β Set to 0.3 and 0.8. When the detection confidence is greater than the trajectory confidence, α d will decrease, and the influence of the original appearance features of the trajectory on the updated features will decrease, and vice versa.

[0080] The association model uses an AAAM correction module when calculating appearance similarity, and adaptively selects a suitable trajectory appearance feature template according to the state of the potential target; the AAAM correction module updates multiple appearance state templates and corresponding confidence levels of the trajectory online, and selects the template closest to the potential target according to the detection confidence level to calculate the appearance similarity. Specifically, two additional appearance features of the trajectory are stored: the appearance feature corresponding to the last successful detection of the target and the average of the appearance features of all historical observations of the target. Next, the cosine distance between the appearance feature corresponding to the last successful detection of the target and the appearance feature of the potential target in the current frame is calculated. and the absolute value of the difference between the corresponding confidence levels At the same time, the cosine distance between the average appearance features of all historical observations of the target and the appearance features of the potential target in the current frame is calculated and the absolute value of the difference between the corresponding confidence levels For each trajectory-potential target pair (i, j), the trajectory feature with the smallest confidence difference between the last associated feature and the historical average feature is used to construct the candidate template confidence distance matrix And the candidate template appearance feature distance matrix , the calculation formula is:

[0081] ;

[0082] Similarly, the adaptive EMA state transfer formula (AEMA) is used to update the appearance features and confidence of the trajectory to obtain the confidence distance matrix And the appearance feature distance matrix On this basis, the confidence distance matrix of the candidate templates is compared Distance matrix with trajectory confidence The size of each element in , by selecting the appearance feature distance corresponding to the smaller confidence distance to construct the final appearance distance matrix , the calculation formula is:

[0083] ;

[0084] The association model follows the motion model in the baseline method Hybrid-SORT and uses Kalman filtering to model and predict the motion state of the target. The state vector of the Kalman filter is:

[0085]

[0086] Among them, (u, v) is the coordinate of the center point of the target, a is the area of ​​the bounding box, c is the confidence of the target box, and r is the aspect ratio of the bounding box. They are Using this motion model, we can get the motion similarity distance of the trajectory-potential target pair (i, j) , using the following formula:

[0087] ;

[0088] in, is the HMIoU spatial position intersection-over-union distance in Hybrid-SORT, is the velocity direction consistency distance in Hybrid-SORT, is the absolute value of the difference in confidence distances, and λ1, λ2, and λ3 are three manually set weight parameters.

[0089] When calculating position similarity, the traditional intersection-and-union ratio based on the area scale is multiplied by the intersection-and-union ratio based on the height scale, which also implies depth information to a certain extent. The radian value differences of the velocity direction vectors of the four vertices of the target bounding box are calculated respectively and the average value is taken. The velocity direction vector is obtained by calculating the normalized difference between the two observation points in the x and y coordinates using historical observations. It is obtained by calculating the absolute value of the difference between the prediction confidence and the detection confidence of the trajectory. In terms of the weights of each factor, the present invention follows the setting of the baseline method and sets λ1, λ2, and λ3 to 1.0, 0.2, and 1.0 respectively.

[0090] Embodiment 3: In some embodiments, Figure 3 As shown in the figure, the principle of a pedestrian multi-target tracking method with improved appearance feature processing is as follows: first, input frame t, extract the bounding box of its potential target and the corresponding confidence through the detector model, and obtain the detection result Dt. At the same time, the target historical trajectory Tt-1 is predicted by Kalman filtering to obtain the motion state, and the motion state and Dt are input into the motion branch to calculate the three types of similarities: the height-corrected intersection-over-union ratio HMIoU, the speed direction consistency ROCM, and the confidence method consistency, and obtain the motion similarity distance In the appearance branch, the appearance features of the current frame target are extracted according to Dt and the appearance feature library of Tt-1 is used for similarity calculation. The similarity calculation result is corrected by the AAAM correction module to obtain the appearance distance matrix Then, combined with the motion similarity distance and the appearance distance matrix The weighted result uses the Hungarian association algorithm to associate Dt and Tt-1 in three stages (matching Tt-1 and high confidence Dt, matching Tt-1 and low confidence Dt, matching Tt-1 and the remaining high confidence Dt). Finally, in the trajectory feature management module, the CASFUM module is used to update the features based on the association results, and the high confidence detection results that have not been successfully associated are initialized as new trajectories, and the trajectories that leave the target scene are deleted to obtain the tracking result Tt.

[0091] Embodiment 4: In some embodiments, a pedestrian multi-target tracking system with improved appearance feature processing includes:

[0092] A multi-target tracking model building module, which is used to build a CATrack tracker using the deep learning framework PyTorch. The CATrack tracker consists of a detector model and an association model, which is used to process the input video frame by frame.

[0093] Among them, the detector model adopts the deep neural network target detection model YOLOX, which is used to output the bounding boxes of all potential targets appearing in the current frame;

[0094] The association model processes the output of the detector model in four stages: feature acquisition of the target in the current frame, similarity calculation between the target in the current frame and the target’s historical trajectory, bipartite graph matching between the target in the current frame and the target’s historical trajectory, and trajectory feature update using the matching results. The association model includes an appearance feature extraction sub-model, which uses the SBS-50 model in FastReID.

[0095] The detector model training module is used to train the detector model of CATrack using the video dataset of the target scene, and iteratively updates the parameters of the network according to the back-propagation principle, so that the model can more accurately identify and locate the target.

[0096] The association model training and hyperparameter adjustment module is used to train the appearance feature extraction sub-model using the target scene video dataset, so that the model can obtain more discriminative target features, iteratively update the parameters of the network according to the back-propagation principle, and adjust the hyperparameters of CATrack's association model in the target scene video according to the scene characteristics to improve tracking accuracy and adaptability;

[0097] The multi-target tracking execution module is used to input the target scene test video data into the trained and hyperparameter-adjusted CATrack tracker to perform pedestrian multi-target tracking tasks, calculate the multi-target tracking performance indicators and save the tracking result video containing the target ID (the unique identifier of the target in digital form) and the bounding box.

[0098] The pedestrian multi-target tracking system with improved appearance feature processing performs the following process:

[0099] First, the input frame t is passed through the detector model to obtain the detection result Dt, which contains the bounding box of the potential target and the corresponding confidence.

[0100] At the same time, the motion branch combines the target historical trajectory T t-1 The motion state predicted by Kalman filtering is used to obtain the motion similarity distance At the same time, the bounding box information is also input into the appearance branch of the AAAM correction module to extract appearance features and combined with T t-1 The feature library is used to calculate the similarity and obtain the appearance distance matrix .

[0101] Then, by combining the motion similarity distance and the appearance distance matrix , use the Hungarian correlation algorithm to correlate T t-1 and Dt.

[0102] Specifically, by calculating the motion similarity distance and the appearance distance matrix The weighted summation is used to obtain a distance matrix combining motion and appearance information, and the distance matrix combining motion and appearance information is input into the Hungarian algorithm (the Hungarian algorithm is a classic algorithm for bipartite graph matching by inputting a distance matrix. The present invention uses it for the following three-stage association: matching T t-1 and high confidence Dt, matching T t-1 and low confidence Dt, matching T t-1 and the remaining high confidence Dt) to obtain T t-1 The correlation results between and Dt.

[0103] Finally, the trajectory feature management module uses the CASFUM module to update the trajectory features based on the association results, initialize new trajectories and delete the trajectories that leave the target scene to obtain the tracking result T t .

[0104] Application example: MOT20 and MOT17 are widely recognized baseline datasets in the field of multi-target tracking. Among them, MOT17 provides pedestrian tracking data captured indoors and outdoors, as well as high-quality annotations. MOT20 also provides high-quality pedestrian data captured naturally, but the scenes involved are more crowded. In MOT17, the average pedestrian density per frame is 21.1 and 31.8 on the training set and test set, respectively. In MOT20, the average pedestrian density per frame reaches 149.7 and 170.9 on the training set and test set, respectively, which is more challenging.

[0105] The present invention is compared with other advanced methods in recent years on the commonly used multi-target tracking indicators on the MOT20 dataset. The experimental results are shown in Table 1.

[0106] The multi-target tracking index used in the present invention is:

[0107] HOTA: High-order tracking accuracy, an indicator that comprehensively reflects detection accuracy and association accuracy. The larger the better.

[0108] IDF1: The accuracy of target ID, calculated by the ratio of the number of correctly identified targets to the average number of true targets and calculated targets. The larger the better.

[0109] AssA: association accuracy, the larger the better;

[0110] AssR: Associative recall rate, the larger the better.

[0111] Table 1 Performance comparison of the present invention and various advanced multi-target tracking methods on the MOT20 dataset

[0112]

[0113] As can be seen from Table 1, the proposed method has higher performance than the baseline method (Hybrid-SORT-ReID) and other advanced methods in multiple evaluation indicators on the MOT20 dataset with highly dense targets and frequent occlusions, indicating that it has excellent tracking performance in complex scenes with dense targets.

[0114] In addition, the tracking results obtained by the method of the present invention in a dense occlusion scene are visualized on the MOT20 dataset. Figure 4 As shown, it can be seen that the method of the present invention has a good ability to maintain the target ID on the MOT20 dataset, and the method of the present invention has good robustness in scenes with dense targets and frequent occlusions.

[0115] To further verify the generalization ability of the method of the present invention, the present invention conducted experiments on the test set of the MOT17 dataset with relatively low target density. The experimental results are shown in Table 2.

[0116] Table 2 Performance comparison of the proposed method and several advanced multi-target tracking methods on the MOT17 dataset

[0117]

[0118] As can be seen from Table 2, the method of the present invention also shows higher performance advantages than the baseline method (Hybrid-SORT-ReID) and other advanced methods in multiple evaluation indicators on the MOT17 dataset with low target density, demonstrating the good generalization ability and universality of the method of the present invention in different application scenarios.

[0119] At the same time, on the MOT20 dataset, ablation experiments are carried out with Hybrid-SORT-ReID as the baseline method to verify the effectiveness of multiple parts of the present invention (selective appearance state update strategy (SU), adaptive EMA state transfer formula (AEMA), AAAM correction module (AAAM)). The results are shown in Table 3.

[0120] Table 3 Ablation experiments on the effectiveness of multiple parts of the present invention

[0121]

[0122] It can be seen from Table 3 that the selective appearance state update strategy (SU), adaptive EMA state transfer formula (AEMA), and AAAM correction module (AAAM) proposed in the present invention are effective.

[0123] In summary, the present invention can achieve performance comparable to that of current advanced methods and demonstrates superior performance in complex dense pedestrian scenes. The present invention has important implications for the practical application of multi-target tracking methods.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pedestrian multi-target tracking method with improved appearance feature processing, characterized in that: The following steps are involved: S101. Build a multi-target tracking model: Use the deep learning framework PyTorch to build a CATrack tracker, which consists of a detector model and an association model, and is used to process the input video frame by frame. The association model processes the output of the detector model in four stages, namely: feature acquisition of the target appearing in the current frame, similarity calculation between the target appearing in the current frame and the target's historical trajectory, bipartite graph matching between the target appearing in the current frame and the target's historical trajectory, and trajectory feature update using the matching results; S102, training of the detector model: using a video dataset of the target scene to train the detector model; S103, training and hyperparameter adjustment of the association model: using the video dataset of the target scene to train the association model, and adjusting the hyperparameters of the association model in the video of the target scene according to the scene characteristics; S104, execution of the multi-target tracking task: inputting the target scene test video data into the trained and hyperparameter-adjusted CATrack tracker, performing the pedestrian multi-target tracking task, calculating the multi-target tracking performance index and saving the tracking result video including the target ID and the bounding box; The association model adopts a confidence-based selective feature update module, namely, a CASFUM module, which includes a selective appearance state update strategy for filtering ambiguous trajectories and a confidence-based adaptive EMA state transition formula; The selective appearance state update strategy performs differentiated processing according to the trajectory situation: for trajectories that are in an active state or in an inactive state but not completely in the potential occlusion area, normal appearance feature updates are performed after their association is successful, while for inactive occlusion area trajectories, that is, trajectories that were not successfully associated in the previous frame and are completely in the potential occlusion area, they are not used for the update of trajectory appearance features; For the successfully matched trajectory, the adaptive EMA state transition formula that considers both the trajectory confidence and the potential target confidence is used for update. The calculation formula is: ； ； in, is the updated trajectory appearance feature, is the trajectory appearance feature before updating, is the appearance feature extracted from the potential target detection box matching the trajectory, is the updated trajectory confidence, is the trajectory confidence before updating; α d is the adaptive inertia weight, which is calculated as: ； ； Among them, α is the initial inertia weight parameter set manually, c i and c det are the confidence of trajectory i and the confidence of the detection result matching it, β is the acceptance threshold parameter used to filter noise, σ is a c i and c det The degree of adjustment is linearly related to the average c i and c det If they are equal, they are zero.

2. The pedestrian multi-target tracking method with improved appearance feature processing according to claim 1, characterized in that: In step S101, the detector model adopts the deep neural network target detection model YOLOX; the association model includes an appearance feature extraction sub-model, and the appearance feature extraction sub-model adopts the SBS-50 model.

3. The pedestrian multi-target tracking method with improved appearance feature processing according to claim 2, characterized in that: In step S101, the feature acquisition of the target includes: acquiring the target motion state, confidence and appearance features; the similarity calculation between the target appearing in the current frame and the target historical trajectory includes: motion similarity calculation, using the settings in the baseline method Hybrid-SORT; appearance similarity calculation, extracting the appearance features of the potential target in the current frame and calculating the cosine similarity with the trajectory appearance features.

4. The pedestrian multi-target tracking method with improved appearance feature processing according to claim 3 is characterized in that: In step S101, when the association model processes the appearance features, the state of the trajectory is divided according to the association situation and the position situation: for the trajectory that has been successfully associated in the previous frame and has just been initialized, its association state is set to activated, and for other trajectories that have not been successfully associated, the association state is set to inactivated; when processing the detection associated with the trajectory in the inactivated state, the detected potential targets are divided according to the position of the target: for any target existing in the frame, the left and right sides of the image are respectively shrunk inward by a distance w of the width of the target bounding box, and the upper and lower sides of the image are respectively shrunk inward by a distance h of the height of the target bounding box to obtain a potential occlusion area, and the other parts are regarded as normal areas; for targets whose target bounding boxes completely fall into the potential occlusion area, they are judged as occluded, and for targets whose target bounding boxes overlap with the normal area, they are judged as non-occluded.

5. The pedestrian multi-target tracking method with improved appearance feature processing according to claim 4 is characterized in that: The association model uses the AAAM correction module when calculating the appearance similarity, which includes the following steps: First, two additional appearance features of the trajectory are stored: the appearance feature corresponding to the last successful detection of the target and the average appearance feature of all historical observations of the target; Next, the cosine distance between the appearance features corresponding to the last successful detection of the target and the appearance features of the potential target in the current frame is calculated. and the absolute value of the difference between the corresponding confidence levels ; At the same time, calculate the cosine distance between the average appearance features of all historical observations of the target and the appearance features of the potential target in the current frame and the absolute value of the difference between the corresponding confidence levels ; For each trajectory-potential target pair (i, j), the trajectory feature with the smallest confidence difference between the last associated feature and the historical average feature is used to construct the candidate template confidence distance matrix And the candidate template appearance feature distance matrix , the calculation formula is: ； ； Adaptive EMA state transition formula is used to update the appearance features and confidence of the trajectory to obtain the confidence distance matrix And the appearance feature distance matrix ; Compare candidate template confidence distance matrix Distance matrix with trajectory confidence The size of each element in , by selecting the appearance feature distance corresponding to the smaller confidence distance to construct the final appearance distance matrix , the calculation formula is: .

6. The pedestrian multi-target tracking method with improved appearance feature processing according to claim 5, characterized in that: The association model uses the motion model in the baseline method Hybrid-SORT to calculate motion similarity; Use Kalman filtering to model and predict the motion state of the target. The state vector of Kalman filtering is: ； Where (u, v) is the coordinate of the center point of the target, a is the area of ​​the bounding box, c is the confidence of the target box, and r is the aspect ratio of the bounding box; They are The velocity component of ; Calculate the motion similarity distance of the trajectory-potential target pair (i, j) , using the following formula: ； in, is the HMIoU spatial position intersection-over-union distance in Hybrid-SORT, is the velocity direction consistency distance in Hybrid-SORT, is the absolute value of the difference in confidence distances, and λ1, λ2, and λ3 are three manually set weight parameters.

7. The pedestrian multi-target tracking method with improved appearance feature processing according to claim 6, characterized in that: λ1, λ2, and λ3 are set to 1.0, 0.2, and 1.0, respectively.

8. A pedestrian multi-target tracking system with improved appearance feature processing, characterized in that: include: The multi-target tracking model building module is used to build a CATrack tracker using the deep learning framework PyTorch. The CATrack tracker consists of a detector model and an association model, which is used to process the input video frame by frame. The association model processes the output of the detector model in four stages, namely: feature acquisition of the target appearing in the current frame, similarity calculation between the target appearing in the current frame and the target's historical trajectory, bipartite graph matching between the target appearing in the current frame and the target's historical trajectory, and trajectory feature update using the matching results; A detector model training module for training a detector model using a video dataset of a target scene; An association model training and hyperparameter adjustment module, used to train an association model using a video dataset of a target scene, and to adjust hyperparameters of the association model in the video of the target scene according to scene characteristics; The multi-target tracking execution module is used to input the target scene test video data into the trained and hyperparameter-adjusted CATrack tracker to perform the pedestrian multi-target tracking task, calculate the multi-target tracking performance indicators and save the tracking result video containing the target ID and bounding box; The association model adopts a confidence-based selective feature update module, namely, a CASFUM module, which includes a selective appearance state update strategy for filtering ambiguous trajectories and a confidence-based adaptive EMA state transition formula; The selective appearance state update strategy performs differentiated processing according to the trajectory situation: for trajectories that are in an active state or in an inactive state but not completely in the potential occlusion area, normal appearance feature updates are performed after their association is successful, while for inactive occlusion area trajectories, that is, trajectories that were not successfully associated in the previous frame and are completely in the potential occlusion area, they are not used for the update of trajectory appearance features; For the successfully matched trajectory, the adaptive EMA state transition formula that considers both the trajectory confidence and the potential target confidence is used for update. The calculation formula is: ； ； in, is the updated trajectory appearance feature, is the trajectory appearance feature before updating, is the appearance feature extracted from the potential target detection box matching the trajectory, is the updated trajectory confidence, is the trajectory confidence before updating; α d is the adaptive inertia weight, which is calculated as: ； ； Among them, α is the initial inertia weight parameter set manually, c i and c det are the confidence of trajectory i and the confidence of the detection result matching it, β is the acceptance threshold parameter for filtering noise, and σ is a c i and c det The degree of adjustment is linearly related to the average c i and c det If they are equal, they are zero.

Images