A pedestrian multi-target tracking method based on human head detection and body segmentation

Abstract

The application discloses a pedestrian multi-target tracking method and system based on head detection and body segmentation, which comprises the following steps: adopting a head detection model and an instance segmentation model to perform head detection and pedestrian instance segmentation on a current frame of a video to obtain a head detection frame set and a pedestrian segmentation mask set; based on each pedestrian segmentation graph, a feature extraction model is adopted to extract a deep appearance feature vector, the head detection frame is matched with the corresponding mask and feature vector in space, and a head-segmentation feature mapping table is established; Kalman filter motion prediction is carried out based on historical targets, two-stage position matching and one-stage appearance feature matching are sequentially performed, and the Kalman filter posterior state is updated according to the matching result; the historical target count and tracking state are updated in combination with multi-level correlation results, a new target is created for an unmatched detection frame, and an overdue unmatched target is eliminated; and tracking results are output and subsequent frames are processed in a loop until the end of the video. The application realizes robust pedestrian multi-target tracking.

Description

Technical Field

[0001] This application relates to a method and system for multi-target pedestrian tracking based on head detection and body segmentation, belonging to the field of multi-target pedestrian tracking technology. Background Technology

[0002] Multi-object tracking (MOT) is one of the core tasks in computer vision, widely used in scenarios such as intelligent video surveillance, autonomous driving environmental perception, smart city pedestrian flow statistics, sports event analysis, and human-computer interaction. In these practical applications, tracking algorithms need to continuously and accurately identify and associate multiple targets of interest in complex dynamic environments, providing reliable support for subsequent high-level tasks such as behavior analysis and trajectory prediction.

[0003] However, MOT still faces significant challenges in complex scenarios, including: (1) Occlusion causes the target to disappear temporarily and switch identities: When the target is partially or completely occluded by other objects or pedestrians, the tracker is prone to losing the target; after the occlusion is removed, the target has often moved to a new location. If there is no effective re-identification mechanism, it is often misjudged as a newly appeared target, causing identity switching (IDSwitch) error.

[0004] (2) Unstable human detection bounding boxes affect motion prediction accuracy: Existing human detection-based tracking methods first obtain human bounding boxes through a target detector, and then use the geometric properties of the bounding boxes (such as size and position) for motion prediction (such as Kalman filtering). However, in dense crowds, the size of the human bounding boxes detected for the same target varies greatly in different frames. Especially when there is local occlusion, the shape of the detection box changes drastically, which leads to a significant decrease in the accuracy of motion prediction.

[0005] (3) Feature extraction is severely affected by background interference, resulting in poor re-identification performance: When extracting appearance features directly from the human body detection box, a large amount of background noise and local limb information of other pedestrians are inevitably mixed in. Such mixed features cannot accurately represent the unique appearance of the tracked target. When geometry-based motion tracking fails, the accuracy of re-identification based on appearance features is extremely low, further exacerbating the problems of IDSwitch and trajectory fragmentation.

[0006] In summary, the fundamental reasons why existing mainstream multi-object tracking algorithms perform poorly in the above scenarios can be attributed to two points: (1) The detection box size is too large and changes drastically across frames: The human body detection box covers a large area and is very easy to overlap in crowded scenes. Furthermore, the geometric parameters of the detection boxes of the same target fluctuate greatly between frames, which seriously interferes with the position prediction accuracy based on motion models such as Kalman filtering.

[0007] (2) Feature extraction is contaminated by background: Redundant information (background, other people's limbs) in the human body detection box makes the extracted appearance features insufficient in discrimination, resulting in unsatisfactory re-identification effect based on features after the target is lost, and it is difficult to distinguish between the tracked target and the newly appeared target.

[0008] Therefore, overcoming the above-mentioned shortcomings and improving the robustness and accuracy of multi-target tracking in complex scenarios has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] According to one aspect of this application, a method for multi-object pedestrian tracking based on head detection and body segmentation is provided, which achieves robust multi-object pedestrian tracking.

[0010] A method for multi-target pedestrian tracking based on head detection and body segmentation includes the following steps: S1. Using a pre-trained head detection model and instance segmentation model, perform head detection and pedestrian instance segmentation on the current frame of the video respectively to obtain a set of head detection boxes and a set of pedestrian segmentation maps; S2. Based on the pedestrian segmentation maps, a pre-trained feature extraction model is used to extract depth appearance feature vectors from the original images. Each head detection box is spatially matched with the corresponding pedestrian segmentation map and depth appearance feature vector to establish a head-segmentation feature mapping table. S3. Obtain the currently existing historical tracking targets, perform Kalman filter motion prediction based on the historical tracking targets, and sequentially perform the first-level matching based on location, the second-level matching based on location, and the third-level matching based on appearance features in the head-segmentation feature mapping table. Update the Kalman filter posterior state using the detection box of the current frame according to the matching results of each level. S4. Based on the matching results of the multi-level data association, update the continuous unmatched count and tracking status of each historical tracking target, create new targets for the unmatched current frame detection box, and remove overdue unmatched targets. S5. Output the tracking result of the current frame, jump to S1 to process the next frame, until the video stream ends.

[0011] Furthermore, in S2, the depth appearance feature vector is extracted from the original image, including: A segmentation mask is generated using the pedestrian segmentation map. Based on the segmentation mask, the pedestrian region is cropped from the original image to obtain pixels within the human body contour. The pixels within the human body contour are then input into the feature extraction model to obtain a depth appearance feature vector. The spatial matching includes: The ratio of the intersection area of ​​the head detection box and each pedestrian segmentation map to the area of ​​the head detection box is calculated as the partial intersection-union ratio (MIR). The pedestrian segmentation map with the largest MIR is selected as the matching object. When the MIR reaches a preset mapping threshold, a mapping relationship is established and the corresponding feature vector is associated.

[0012] Furthermore, the intersection area between the head detection bounding box and each pedestrian segmentation map is calculated, including: The intersection set between the head detection box and the polygons of the pedestrian segmentation map is calculated using a geometric operation library. The intersection set is a set of one or more intersecting polygons. The sum of the areas of each intersecting polygon in the intersection set is taken as the intersection area. If the maximum intersection-union ratio does not reach the preset mapping threshold, the head detection box will not be associated with segmentation features for the time being, and its corresponding feature vector will be set to empty, waiting for features to be added in subsequent frames.

[0013] Furthermore, the geometry operation library adopts the Boost.Geometry library. The intersection() function of the Boost.Geometry library is used to calculate the intersection of the head detection box and the polygons of the pedestrian segmentation map, and the area() function of the Boost.Geometry library is used to calculate the area of ​​each polygon in the intersection. For pedestrian segmentation maps that do not match any head detection bounding boxes, they are considered missed targets and are not included in the tracking system for the time being; for head detection bounding boxes that do not match any pedestrian segmentation map, they are considered newly appearing targets, and their tracking feature vectors are temporarily empty, waiting to be supplemented in subsequent frames.

[0014] Furthermore, in S3, the first-level matching is as follows: taking the historical target in the active state and the historical target in the temporarily lost state as the first matching objects, the predicted box obtained by Kalman filtering of each of the first matching objects is matched with each of the head detection boxes in the current frame by intersection-union comparison. The second-level matching is as follows: taking the historical target with the state of being lost as the second matching object, taking the head detection box that was last successfully matched in each of the second matching objects as the reference box, and performing cross-union matching with each of the remaining head detection boxes after the first-level matching, and the second-level cross-union threshold is less than the first-level cross-union threshold; The third-level matching is as follows: the remaining head detection boxes after the first two levels of matching are matched with the remaining historical targets after the first two levels of matching, and cosine similarity matching is performed based on the appearance feature vectors in the head-segmentation feature mapping table.

[0015] Furthermore, in S4, the tracking state includes active, temporarily lost, lost, and removed, and each state transitions based on the number of consecutive unmatched frames of the historical target: When the number of consecutive unmatched frames is 0, the target is in the active state; When the number of consecutive unmatched frames is greater than or equal to 1 and less than 3, the target is in the temporary loss state; When the number of consecutive unmatched frames is greater than or equal to 3 and less than 20, the target is in the lost state; When the number of consecutive unmatched frames is greater than or equal to 20, the target is in the removal state and is deleted from the historical tracked target set.

[0016] Furthermore, the head detection employs a YOLOv8-det detector; The instance segmentation uses the YOLOv8n-seg segmenter; The feature extraction employs a ResNet-50 network pre-trained on a pedestrian re-identification dataset.

[0017] According to another aspect of this application, a pedestrian multi-target tracking system based on head detection and body segmentation is provided, comprising: The perception module is used to perform head detection and pedestrian instance segmentation on video frames to obtain head detection boxes and pedestrian segmentation masks; The feature module is used to extract depth appearance features from the pedestrian segmentation mask and establish a mapping relationship between the human head and the segmentation features; The tracking module is used to maintain historical trajectory sets, perform Kalman filter prediction and updates, perform multi-level data association matching based on location and features, and manage the tracking status and lifecycle of the target.

[0018] Furthermore, the tracking module includes: Kalman filter unit, used for target motion state prediction and posterior state update; A multi-level matching unit is used to sequentially perform first-level matching based on intersection-union ratio (IU), second-level matching based on IU, and third-level matching based on cosine similarity. The status management unit is used to update the target tracking status and manage the lifecycle based on the matching results.

[0019] The beneficial effects that this application can produce include: This application provides a method and system for multi-target pedestrian tracking based on head detection and body segmentation. It combines head detection and pedestrian instance segmentation to achieve multi-cue fusion pedestrian tracking. A head-segmentation feature mapping is constructed by matching the head detection bounding box with the pedestrian segmentation mask space. Precise depth and appearance features are extracted based on the pedestrian mask. A multi-level matching strategy is then employed, first relying on Kalman filtering for motion prediction, and then progressively performing position matching and appearance feature matching for data association. This effectively alleviates the problems of identity switching and missed tracking in scenarios with pedestrian occlusion and overlapping targets, improving target association accuracy. Simultaneously, the hierarchical matching mechanism balances computational efficiency and matching reliability, accurately updating the tracking target status and reasonably managing new and invalid targets, achieving stable continuous multi-target pedestrian tracking in video scenarios. Attached Figure Description

[0020] Figure 1 This is a flowchart of a pedestrian multi-target tracking method based on head detection and body segmentation in one embodiment of this application; Figure 2 This is an overall block diagram of a pedestrian multi-target tracking method based on head detection and body segmentation in one embodiment of this application; Figure 3 This is a system block diagram of a pedestrian multi-target tracking system based on head detection and body segmentation in one embodiment of this application. Detailed Implementation

[0021] The present application is described in detail below with reference to the embodiments, but the present application is not limited to these embodiments.

[0022] See Figure 1-2 A method for multi-target pedestrian tracking based on head detection and body segmentation includes the following steps: S1. Using a pre-trained head detection model and instance segmentation model, perform head detection and pedestrian instance segmentation on the current frame of the video respectively to obtain a set of head detection boxes and a set of pedestrian segmentation maps; S2. Based on the pedestrian segmentation maps, a pre-trained feature extraction model is used to extract depth appearance feature vectors from the original images. Each head detection box is spatially matched with the corresponding pedestrian segmentation map and depth appearance feature vector to establish a head-segmentation feature mapping table. S3. Obtain the currently existing historical tracking targets, perform Kalman filter motion prediction based on the historical tracking targets, and sequentially perform the first-level matching based on location, the second-level matching based on location, and the third-level matching based on appearance features in the head-segmentation feature mapping table. Update the Kalman filter posterior state using the detection box of the current frame according to the matching results of each level. S4. Based on the matching results of the multi-level data association, update the continuous unmatched count and tracking status of each historical tracking target, create new targets for the unmatched current frame detection box, and remove overdue unmatched targets. S5. Output the tracking result of the current frame, jump to S1 to process the next frame, until the video stream ends.

[0023] Specifically, the algorithm uses three pre-trained models to parse single-frame images of a video in parallel, extracting pedestrian information from local head and overall body shapes to compensate for the shortcomings of a single detection model. The algorithm offline loads three types of pre-trained models trained on large-scale datasets: a head detection model, a pedestrian instance segmentation model, and a feature extraction model. The head detection model focuses on identifying pedestrian heads in the image, exhibiting strong resistance to partial occlusion and overlapping interference. The pedestrian instance segmentation model accurately distinguishes the body boundaries of multiple overlapping pedestrians in the image, outputting an independent pixel-level mask for each pedestrian. The feature extraction model extracts pedestrian-specific, highly discriminative appearance depth features. For each frame of the input video image, a dual-task algorithm is performed. Parallel inference is employed, scanning the entire frame image using a head detection model to filter out all pedestrian head regions and output a set of head detection bounding boxes containing head coordinates, width, height, and confidence scores. This locks the core area where pedestrians exist and avoids interference from background clutter. Furthermore, a pedestrian instance segmentation model performs pixel-level segmentation on the same frame image, distinguishing pedestrians from the background and different individual pedestrians. It generates a set of pedestrian segmentation masks for each independent pedestrian. The mask is a binary pixel matrix that accurately marks the complete shape range of a single pedestrian, including the torso, limbs, and all other effective areas. Combining the advantages of head detection in resisting occlusion and instance segmentation in accurately distinguishing overlapping targets, the accuracy of subsequent tracking is improved from the source.

[0024] Based on the pre-trained feature extraction model, all pedestrian segmentation masks obtained from S1 are traversed. High-dimensional feature encoding is performed using the pixel region of the complete pedestrian shape corresponding to each mask as input, extracting a depth appearance feature vector that characterizes the color, texture, body shape, and contour of the pedestrian's clothing. This feature vector is unique and can effectively distinguish different pedestrians with similar appearances and close positions. Based on the principle of image spatial coordinate consistency, a one-to-one matching is performed between the head detection box and the pedestrian segmentation mask. It is determined whether the center coordinates and coverage area of ​​a single head detection box fall within the pixel range of a certain pedestrian segmentation mask. Simultaneously, based on the shape range of the mask, a unique corresponding head target is selected, eliminating invalid matches due to misalignment or false detection. The successfully matched head detection boxes, corresponding pedestrian segmentation masks, and exclusive depth appearance feature vectors are bound together and structurally stored in a head-segmentation feature mapping table. This mapping table establishes a unique data file for each valid pedestrian target in each frame, achieving a one-to-one correspondence between head localization, body contour, and appearance features.

[0025] The system retrieves historical motion data of all tracked targets (identified pedestrians) from historical video frames, including historical coordinates, speed, and direction of movement. Using a Kalman filter algorithm, it optimally predicts the position and state of each historical target in the current frame, outputting the predicted motion state of each tracked target. This pre-emptively identifies the high-probability occurrence area of ​​the target in the current frame, narrowing the search range for subsequent matching and improving matching efficiency. A three-layer progressive hierarchical matching process is executed, filtering step by step. The first layer is a coarse matching for close-range positions. Using the target position predicted by the Kalman filter as the center, a fixed spatial threshold is set, and the actual position of the head detection box in the current frame is compared with the predicted position of the historical target. Targets with a spatial distance less than the threshold are initially identified as the same target, completing the basic coarse matching. This is primarily suitable for typical scenarios where pedestrians move at a constant speed and are unobstructed. The second layer is a fault-tolerant position matching, addressing targets that were not matched in the first layer (suitable for pedestrians). (Position deviations caused by sudden speed changes, small displacements, and slight occlusions) The spatial distance threshold is relaxed, and the predicted position is compared with the actual detected position again by combining the overall center position of the pedestrian shape mask. This captures targets with slight motion anomalies, improves the algorithm's fault tolerance, and reduces missed matches. The third layer is fine-grained appearance feature matching. For suspected targets that have not been matched or whose positions overlap in the first two layers of position matching, the depth appearance feature vector in the mapping table constructed by S2 is retrieved, and the similarity between the features of historically tracked targets and the features of unmatched targets in the current frame is calculated. The target with the highest feature similarity is determined to be the same target, solving the problem of indistinguishable multiple pedestrians that are clustered, overlapping, or in close positions. Based on the final result of the three-layer matching, the actual head detection position, shape features, and appearance features of the current frame are used to correct the prediction bias of the Kalman filter, update the filtered posterior state of all tracked targets, and record the current real motion and appearance information of the target to provide a basis for prediction in the next frame.

[0026] For each historical tracking target, based on the multi-level matching results of S3, the consecutive unmatched count is counted and updated: if a match is successful, the count is reset to zero, and the target is marked as being in a normal tracking state; if a match fails, the count is incremented, and the target is marked as being in a suspected lost state. For all head detection boxes in the current frame that have not successfully matched any historical tracking target (corresponding to pedestrians newly entering the field of view or new independent targets that reappear after occlusion), a unique tracking ID is automatically assigned, the corresponding segmentation mask and appearance features are bound, a new tracking target is created, and it is included in the global tracking list. A fixed unmatched threshold number of frames is set. If the consecutive unmatched count of a certain historical target exceeds the threshold, it is determined that the pedestrian has left the video screen or has been completely occluded and disappeared. The target is then directly removed from the tracking list to avoid invalid data consuming computing power and interfering with subsequent tracking.

[0027] After completing the detection, matching, and status update of all targets in the current frame, the final tracking result of the current frame is output uniformly, including complete information such as the unique tracking ID, real-time coordinates, shape segmentation region, and tracking status of each pedestrian. The system automatically jumps to step S1, loads the next frame of the video, and repeats the above full-process reasoning, matching, and update operations, continuously iterating through each frame of the video stream. The entire process is looped until the entire video stream is read and processed, achieving continuous and stable multi-target tracking of all pedestrian targets in the video, effectively avoiding problems such as target ID skipping, missed tracking, mistracking, and overlapping confusion in traditional algorithms.

[0028] In step S2, the depth appearance feature vector is extracted from the original image, including: A segmentation mask is generated using the pedestrian segmentation map. Based on the segmentation mask, the pedestrian region is cropped from the original image to obtain pixels within the human body contour. The pixels within the human body contour are then input into the feature extraction model to obtain a depth appearance feature vector. The spatial matching includes: The ratio of the intersection area of ​​the head detection box and each pedestrian segmentation map to the area of ​​the head detection box is calculated as the partial intersection-union ratio (MIR). The pedestrian segmentation map with the largest MIR is selected as the matching object. When the MIR reaches a preset mapping threshold, a mapping relationship is established and the corresponding feature vector is associated.

[0029] Specifically, when extracting depth appearance feature vectors, a feature extraction scheme using masked pixel filtering and precise pedestrian region cropping is adopted to completely eliminate background interference and ensure the specificity and recognizability of the feature vectors. Based on the pedestrian instance segmentation mask output by S1, the original current frame image of the video is cropped to extract the complete body region corresponding to a single pedestrian. Pixel masking is performed, setting all pixels outside the mask outline within the cropped area to zero, retaining only the effective pixel information inside the pedestrian's body outline, completely filtering out invalid interference information such as background clutter around the pedestrian, overlapping areas of other pedestrian edges, and image noise. The image region purified by masking and retaining only pure human pixels is input into a pre-trained feature extraction model. Through convolutional coding, dimensionality compression, and feature mapping of the model, a fixed-dimensional, highly discriminative pedestrian-specific depth appearance feature vector is output. The feature vectors extracted in this way focus only on the color, texture, body shape, and clothing features of the pedestrian body, without redundant background information, significantly improving the accuracy of subsequent cross-frame feature matching and effectively solving the problems of feature similarity deviation and matching disorder caused by background interference and regional redundancy in traditional feature extraction methods.

[0030] Furthermore, to address issues such as multiple-person overlap, head offset, and mismatches, a partial intersection-union ratio (MIR) matching mechanism is adopted. This mechanism achieves one-to-one matching between head detection boxes and pedestrian segmentation masks. For each head detection box and each pedestrian segmentation mask in the current frame, the area of ​​their intersection region is calculated, and the MIR is further obtained. This metric is defined as (intersection area of ​​head detection box and pedestrian segmentation mask) / total area of ​​the current head detection box, used to characterize the degree of overlap and fit between a single head region and the corresponding pedestrian shape. All pedestrian segmentation masks are traversed, and all MIR values ​​corresponding to the same head detection box are compared. The pedestrian segmentation mask with the highest MIR is then selected. The head is initially identified as the corresponding shape object. A fixed mapping and matching threshold is preset. Only when the maximum intersection-union ratio (IU) of the filtered parts is greater than or equal to the preset mapping threshold is the head detection box and the corresponding pedestrian segmentation mask considered a valid match pair, eliminating invalid matches due to weak association, false detection, or overlapping interference. After the matching verification is passed, the head detection box and pedestrian segmentation mask are strongly associated and bound with the depth appearance feature vector extracted from the corresponding mask, and included in the head-segmentation feature mapping table. This adapts to complex scenarios such as pedestrians looking down, turning to the side, head offset, and multiple people standing closely side by side, accurately distinguishing the individual to which the head belongs, completely avoiding the problem of incorrect binding of the head to misaligned shapes, and ensuring the accuracy of the mapping table data.

[0031] Calculate the intersection area between the head detection bounding box and each pedestrian segmentation map, including: The intersection set between the head detection box and the polygons of the pedestrian segmentation map is calculated using a geometric operation library. The intersection set is a set of one or more intersecting polygons. The sum of the areas of each intersecting polygon in the intersection set is taken as the intersection area. If the maximum intersection-union ratio does not reach the preset mapping threshold, the head detection box will not be associated with segmentation features for the time being, and its corresponding feature vector will be set to empty, waiting for features to be added in subsequent frames.

[0032] The geometry operation library uses the Boost.Geometry library. The intersection() function of the Boost.Geometry library is used to calculate the intersection of the head detection box and the polygons of the pedestrian segmentation map, and the area() function of the Boost.Geometry library is used to calculate the area of ​​each polygon in the intersection. For pedestrian segmentation maps that do not match any head detection bounding boxes, they are considered missed targets and are not included in the tracking system for the time being; for head detection bounding boxes that do not match any pedestrian segmentation map, they are considered newly appearing targets, and their tracking feature vectors are temporarily empty, waiting to be supplemented in subsequent frames.

[0033] Specifically, spatial matching between head detection boxes and pedestrian segmentation maps employs a partial intersection-to-union (PIoU) matching strategy, performed using a geometry library. During matching, the geometry library calculates the intersection set between the head detection box and each polygon in the pedestrian segmentation map. This intersection set may contain one or more separate intersecting polygons (e.g., when the segmentation map partially covers the head region or has multiple intersecting regions). The total intersection area is obtained by summing the areas of each polygon in the intersection set, and then divided by the area of ​​the head detection box itself to obtain the PIO as the matching metric. If the maximum PIO reaches a preset mapping threshold, a mapping relationship is established between the head detection box and the corresponding pedestrian segmentation map, and its depth and appearance feature vectors are associated. If the threshold is not reached, the head detection box is not associated with segmentation features, and its corresponding feature vector is marked as empty, awaiting feature addition in subsequent frames. To ensure computational accuracy, the geometry library specifically uses the Boost.Geometry library, which calls the `intersection()` function to calculate the intersection set of polygons and the `area()` function to accumulate the areas of each intersecting polygon. Furthermore, pedestrian segmentation maps that do not match any head detection bounding boxes are considered missed targets and are temporarily excluded from the tracking system; head detection bounding boxes that do not match any pedestrian segmentation map are considered newly appearing targets without feature information, and their feature vectors are temporarily empty, awaiting supplementation in subsequent frames. This matching mechanism ensures the accuracy of the association between head detection bounding boxes and segmentation features, and provides a reliable data foundation for subsequent feature-based re-identification matching.

[0034] In S3, the first hierarchical matching is as follows: taking the historical target in the active state and the historical target in the temporarily lost state as the first matching objects, the predicted box obtained by Kalman filtering of each of the first matching objects is matched with the head detection boxes of the current frame by intersection-union comparison. The second-level matching is as follows: taking the historical target with the state of being lost as the second matching object, taking the head detection box that was last successfully matched in each of the second matching objects as the reference box, and performing cross-union matching with each of the remaining head detection boxes after the first-level matching, and the second-level cross-union threshold is less than the first-level cross-union threshold; It should be noted that active and temporarily lost targets that fail to match after the first-level matching do not participate in the second-level IoU matching (the second level only performs recovery matching for lost targets). These unmatched active and temporarily lost targets will directly enter the third level and be merged with the remaining unmatched targets from the second level to form the historical target candidate pool for appearance feature matching in the third level.

[0035] The third-level matching is: the remaining head detection boxes after the first two levels of matching are matched with the remaining historical targets after the first two levels of matching, and the cosine similarity matching is performed based on the appearance feature vectors in the head-segmentation feature mapping table. Specifically, the first layer is the basic conventional matching layer, targeting pedestrian targets that are normally tracked and in a stable state in the video. Its function is to quickly and accurately associate most interference-free, continuously moving targets. The matching objects are limited to: valid head detection boxes in the current frame and Kalman prediction boxes corresponding to targets in the historical tracking queue that are in an active state or temporarily lost state. Among them, active targets represent pedestrians who have been successfully matched between frames and are stably tracked throughout; temporarily lost targets represent pedestrians who are slightly occluded for a short time, and who are not matched in a single frame but have not been identified as lost. Both types of targets have continuous motion trajectories and small positional deviations, making them suitable for high-precision position matching. This layer uniformly uses the intersection-union ratio (IOU) of the detection box and the prediction box as the matching metric. The matching similarity is determined by calculating the proportion of the overlapping area between the current detection box and the historical target prediction boxes. To ensure the accuracy of conventional target matching and eliminate close-range mismatches, this layer sets a high IOU threshold. A match is only considered successful when the IOU is greater than the high threshold. This layer is suitable for typical scenarios where pedestrians walk at a constant speed, without obstructions, and without drastic displacement. It efficiently performs cross-frame association of most stable targets, filters out reliable matching targets, and reduces the computational load of subsequent layers. Unmatched detection boxes in the current frame and historical targets automatically flow into the second layer for further matching.

[0036] The second layer is a fault-tolerant fallback matching layer, which solves the problem of target matching failure caused by short-term occlusion, temporary disappearance, and displacement of pedestrians. It effectively supplements the first layer. The matching objects are: the unmatched detection boxes of the current frame that are still matched in the first layer, and the last successfully matched box of the target in the historical state of being lost. Among them, the lost target refers to the pedestrian target that has not been matched for multiple consecutive frames and has temporarily fallen out of tracking. Its Kalman prediction trajectory has failed. Therefore, the real prediction box is no longer used. Instead, the real detection box of the last successfully tracked target is retrieved as the matching benchmark to match the positional features of the lost target reappearance. This level also uses the intersection-union (IoU) matching mechanism, but to adapt to scenarios with target reappearance, position shift, and slight misalignment, an IoU threshold lower than that of the first level is set to relax position matching constraints and improve the recall rate of lost targets. The lower threshold can accommodate the problem of bounding box shift caused by pedestrians reappearing after occlusion, slight movement, and posture changes, effectively preventing targets from being identified as new targets after a brief disappearance and reappearance, and eliminating frequent changes in tracking IDs. After being filtered by this level, successfully matched targets will be reset to active tracking status, and the remaining unmatched targets will automatically flow into the third level.

[0037] The third layer is the final matching layer. Addressing complex and challenging scenarios that the first two layers of location matching cannot resolve, it completely overcomes the limitations of location matching by relying on pedestrian appearance features for accurate differentiation. The matching targets are: the remaining unmatched detection boxes in the current frame after the first two layers of filtering, and the remaining unmatched historical tracking targets. These targets are often difficult targets with dense overlap of multiple people, similar positions and heights, extremely low crossover ratios, and completely failed location matching. This layer switches to a depth appearance feature cosine similarity matching mechanism, retrieving pedestrian-specific depth features bound to the S2 head-segmentation feature mapping table. The algorithm calculates the similarity between the current frame's target feature vector and the historical target feature vectors using the cosine similarity formula. Cosine similarity can accurately represent the directional consistency of high-dimensional feature vectors, is unaffected by feature amplitude, and can accurately distinguish different pedestrians with similar appearances and overlapping positions. The algorithm traverses all remaining target feature pairs and selects the combination with the highest similarity as the final matching result, completing the cross-frame association of difficult targets. It makes up for the shortcomings of position matching from the appearance dimension, completely solves the problems of missed matching and mismatch caused by crowd gathering, target overlap, and position confusion, and maximizes the integrity and stability of multi-target tracking.

[0038] In step S4, the tracking state includes active, temporarily lost, lost, and removed, and each state transitions based on the number of consecutive unmatched frames of the historical target: When the number of consecutive unmatched frames is 0, the target is in the active state; When the number of consecutive unmatched frames is greater than or equal to 1 and less than 3, the target is in the temporary loss state; When the number of consecutive unmatched frames is greater than or equal to 3 and less than 20, the target is in the lost state; When the number of consecutive unmatched frames is greater than or equal to 20, the target is in the removal state and is deleted from the historical tracked target set.

[0039] Specifically, the active state (MISS_COUNT=0) represents the normal tracking state of the target. When a historical pedestrian target successfully matches multiple levels in the current frame, its consecutive unmatched frame count is reset to zero, and the target is determined to be in the active state. Targets in the active state have continuous trajectories, valid features, and normal position updates, making them effective targets for stable inter-frame tracking. They can participate normally in Kalman motion prediction and level matching in the next frame, suitable for typical scenarios where pedestrians are unobstructed and continuously appear in the frame; the temporarily lost state (1≤MISS_COUNT<N, where N is the threshold for the number of temporarily lost frames, in this case...) In the example, N=3 represents the short-term disconnection transition state of the target. When an active pedestrian target fails to match in the current frame, its consecutive unmatched frame count increments by 1. Once MISS_COUNT≥1, the target state changes from active to temporarily disconnected. This adapts to short-term matching failure scenarios caused by slight pedestrian occlusion, side occlusion, instantaneous movement out of the frame edge, or brief noise interference in the image. In this case, the target is not considered lost, its trajectory is not deleted, and a new target is not created. The target's historical position, appearance features, and motion trajectory are temporarily retained, waiting for the target to reappear in subsequent frames to re-match and restore the active state. To minimize ID jumps caused by minor interference; the lost state (N≤MISS_COUNT<MAX_AGE, MAX_AGE is the maximum survival frame threshold, in this embodiment MAX_AGE=20) is the target's deep disconnection cache state. When a target in the temporarily lost state continues to be unmatched, and the cumulative number of consecutive unmatched frames reaches the temporarily lost frame threshold N, the target state changes from temporarily lost to lost. This state corresponds to scenarios such as pedestrian long-term occlusion, long-distance occlusion in the image, and temporary occlusion. The target cannot be detected in the image for a short period of time. For targets in the lost state, the Kalman motion prediction trajectory fails and no longer participates in the regular prediction box intersection-union matching. Only the last successfully matched real detection box and appearance features are retained and included in the second-level matching candidate set, continuously waiting for the target to reappear; the removed state (MISS_COUNT≥MAX_AGE) is the target's complete departure termination state. When a target in the lost state continues to have no matching results, and the cumulative number of consecutive unmatched frames reaches the maximum survival frame threshold MAX_AGE, it is determined that the pedestrian has completely left the video surveillance screen, the occlusion has completely disappeared, the target trajectory has completely failed, and the removal mechanism is triggered. The target is permanently removed from the global historical trajectory set and the tracking target queue, its accumulated frame count, motion trajectory and feature cache are cleared, and the tracking life cycle of the target is terminated to avoid invalid zombie targets occupying computing power and interfering with the matching of new targets.

[0040] It is worth noting that the state transition follows the logic of increasing frame count, unidirectional degradation, and progressive fault tolerance: a successful match immediately returns to the active state (MISS_COUNT returns to zero), while a failed match increments MISS_COUNT frame by frame, and the state is progressively downgraded from active, temporarily lost (MISS_COUNT≥1), lost (MISS_COUNT≥N), and removed (MISS_COUNT≥MAX_AGE), fully covering the entire lifecycle of pedestrians from stable tracking to brief loss of contact, deep occlusion, and complete departure, greatly improving the tracking stability in complex pedestrian flow scenarios.

[0041] In S3, for targets in an inactive state, the altitude change rate in their velocity component is set to zero before Kalman filter prediction.

[0042] Specifically, historical motion data of all tracked targets (identified pedestrians) in historical video frames are retrieved, including historical coordinates, movement speed, and direction of movement. For active, normally tracked targets, their historical position, horizontal movement speed, and altitude change rate are fully preserved. The Kalman filter algorithm is used for normal iterative calculation to predict their position and motion state in the current frame. For all inactive targets (including temporarily lost or missing targets), before the Kalman filter prediction iteration is executed, the altitude change rate in their velocity component is forcibly set to zero, retaining only the horizontal position prediction capability and blocking the vertical motion update. After completing the correction of the state-differentiated parameters, the Kalman filter is used to make the optimal prediction of the position and state of each historical target in the current frame, outputting the predicted motion state of each tracked target. This pre-locks the high-probability occurrence area of ​​the target in the current frame, narrows the search range for subsequent matching, and improves matching efficiency and prediction accuracy.

[0043] It is worth noting that inactive targets mainly include two types of lost targets: temporarily lost and missing targets. These targets fail to match for multiple frames due to issues such as occlusion, temporary movement out of the frame edge, and frame interference. Their continuous motion trajectory is broken, and the height direction (vertical) velocity parameters collected in historical frames are invalid and lack continuity. If the invalid vertical velocity is directly used for prediction, the target prediction box will continuously shift upward or downward, resulting in severe trajectory drift. This causes problems such as subsequent matching misalignment, inability to recall targets, and incorrect creation of new IDs. This application clears inactive targets separately before the Kalman filter iteration update. The height dimension velocity component in the state vector keeps the longitudinal motion increment fixed at 0. Position prediction is completed solely based on the target's last valid position and horizontal velocity, freezing the longitudinal dynamic changes of inactive targets. This ensures that the predicted bounding box of the lost target only fluctuates slightly within a reasonable horizontal range, maintaining a stable position and eliminating unfounded longitudinal offset drift. At the same time, this strategy does not affect normal horizontal prediction and is compatible with real-world scenarios involving lateral re-enactment and slight horizontal displacement after pedestrian occlusion. It avoids predicted trajectory divergence and distortion while maximizing the recall capability of the lost target, significantly improving the tracking stability of occluded targets.

[0044] The cross-union ratio threshold for the first level of matching is 0.5; The cross-union ratio threshold for the second-level matching is 0.3; The cosine similarity threshold for the third-level matching is 0.75.

[0045] Specifically, the first level is designed for active, temporarily lost, stable tracking targets with continuous inter-frame motion, no significant occlusion, and accurate position prediction. It uses a high intersection-union ratio (IUU) threshold of 0.5 as the matching decision threshold. This threshold means that a position match is only considered successful when the overlap area between the head detection box and the Kalman prediction box in the current frame is ≥50%. It can effectively filter out nearby pedestrians, background interference boxes, and slight offset pseudo-matches, avoiding mismatches and ID errors in stable scenarios from the source. It ensures the matching reliability of most normally walking targets and is suitable for unobstructed, uniformly moving pedestrian tracking scenarios. For targets with less than 50% overlap, it is determined that there is a displacement deviation or slight interference and is automatically downgraded to the second level of fault-tolerant matching.

[0046] The second level targets difficult targets such as those that have been lost and restarted, those that have experienced short-term occlusion, or those that have shifted in position. The matching benchmark is the last valid ground truth bounding box of the target. Since the predicted trajectory has become invalid, a lower intersection-union (IU) threshold of 0.3 is used. This threshold means that a successful match is determined when the overlap area between the detected bounding box and the historical valid bounding box is ≥30%. It has strong scene inclusiveness and can adapt to problems such as pose changes after pedestrians reappear after occlusion, small displacements, bounding box offsets, and low bounding box overlap caused by edge out of view. This significantly improves the recall capability of lost targets. At the same time, this threshold is not too lenient and can effectively filter out false matches of distant irrelevant targets, achieving a balance between fault tolerance and accuracy, and identifying abnormally displaced targets that cannot be matched in the first level.

[0047] The third layer addresses densely overlapping and highly proximate targets where location is completely ineffective. It abandons positional constraints and relies on appearance features to achieve the final association, setting a high cosine similarity threshold of 0.75. The cosine similarity value ranges from -1 to 1. The closer the value is to 1, the higher the similarity between the two feature vectors and the stronger the consistency of pedestrian appearance. The threshold of 0.75 is the industry's optimal empirical value, meaning that the current target is only identified as the same target when the appearance feature similarity with historical targets is ≥75%. This threshold strictly constrains the feature matching accuracy, distinguishing different pedestrians with similar clothing and body shape, and avoiding cross-identity mismatches caused by similar appearance in dense crowds. At the same time, it retains a reasonable margin of error, accommodating small fluctuations in features caused by changes in pedestrian lighting, slight adjustments in posture, and minor color differences in the image. This is the core guarantee for accurately distinguishing targets and preventing ID jumps in densely overlapping scenarios. The combination of the three-layer threshold gradient strictly screens regular targets, tolerates abnormal displacements, and accurately identifies overlapping targets, comprehensively adapting to various complex pedestrian tracking conditions.

[0048] The threshold for the number of frames temporarily lost is 3 frames; The maximum number of frames that can survive is 20.

[0049] Specifically, the temporary loss frame threshold N (N=3 in this embodiment) is used to determine whether a target is downgraded from a temporary loss state to a lost state. When a target that was originally active fails to match in the current frame, its consecutive unmatched frame count MISS_COUNT increments by 1. Once MISS_COUNT≥1, the target immediately switches from an active state to a temporary loss state, enabling the fault-tolerant tracking mechanism. The temporary loss state serves as a buffer for short-term disconnection, allowing the target to briefly lose match without being judged as lost in scenarios such as slight occlusion, side occlusion, or instantaneous movement out of the frame edge, thus avoiding direct loss of the target and the creation of a new ID due to slight interference. When a target in the temporary loss state continues to fail to match, and MISS_COUNT accumulates to the temporary loss frame count threshold N=3 frames, the target is downgraded from a temporary loss state to a lost state. This threshold is set based on the extremely short interval between video frames. Three consecutive frames of unmatched failure are sufficient to eliminate accidental errors such as detection jitter, image noise, and instantaneous occlusion, accurately identifying the real continuous disconnection scenario of pedestrians. After a target enters the lost state, the update of the invalid Kalman prediction trajectory is stopped, the last valid position and appearance features of the target are locked, and it is transferred to the second-level low-threshold IoU fault-tolerant matching queue. This reserves a matching channel for subsequent target re-encounter and effectively solves the pain point of traditional algorithms that directly lose targets due to short-term occlusion. The maximum survival frame threshold MAX_AGE (MAX_AGE=20 in this embodiment) is the final cleanup threshold of the target's life cycle. When a target in the lost state has no matching results for a long time and the MISS_COUNT reaches 20 frames, it is determined that the pedestrian has completely left the monitoring screen, has been completely occluded for a long time and has no possibility of re-encounter. A permanent removal operation is performed, and the target data is completely deleted from the historical trajectory library. The time taken for a pedestrian to return to the screen after being occluded by a squatting or standing, dense crowds, or obstacles for a long time is usually within 20 frames. Sufficient caching time can maximize the recall of deeply lost targets and prevent the target ID from changing after re-encountering occlusion. At the same time, the upper limit constraint of 20 frames can promptly clean up zombie targets that have completely left the field, avoid invalid trajectories from continuously occupying computing resources and interfering with the matching of new targets, and realize the dynamic purification of the tracking queue.

[0050] The head detection uses a YOLOv8-det detector; The instance segmentation uses the YOLOv8n-seg segmenter; The feature extraction employs a ResNet-50 network pre-trained on a pedestrian re-identification dataset.

[0051] Specifically, YOLOv8 is a single-stage real-time detection algorithm with advantages such as fast inference speed, lightweight design, and strong generalization ability. It is optimized for video streaming detection scenarios. The pixel ratio of human head targets is small and the features are concentrated, making it less susceptible to interference from changes in pedestrian posture, torso occlusion, and limb crossing. It can stably detect effective targets in complex conditions such as dense crowds, overlapping human bodies, half-body shots, and pedestrians bending or turning sideways. The YOLOv8-det detector, based on an efficient C2f structure and a lightweight backbone network, significantly reduces the inference time per frame while ensuring high detection recall and accuracy, meeting the frame rate requirements of real-time video tracking and reducing invalid detection, missed detection, and false detection problems from the source.

[0052] The YOLOv8n-seg lightweight instance segmenter is used to complete the pixel-level shape segmentation of pedestrians. YOLOv8n-seg is a lightweight segmentation version of the YOLOv8 series. Its nano lightweight structure has a small number of parameters and low computational cost, making it suitable for terminal and real-time video processing scenarios. At the same time, it retains high-precision instance segmentation capabilities, enabling independent pixel segmentation of multiple pedestrians in the image, differentiating it from the global pixel classification of semantic segmentation. Instance segmentation can accurately distinguish between different pedestrians that overlap, are close together, or intersect, outputting a unique binary mask matrix for each independent pedestrian in the image, outlining the complete shape contour of a single person, including the torso and the edges of the limbs. This model is suitable for high-frequency scenarios where pedestrians are densely gathered, walking side by side, and occluding each other from the perspective of surveillance, and can completely solve the defect of traditional detection boxes that cannot distinguish overlapping individuals.

[0053] This study employs a ResNet-50 deep network pre-trained on a Person Re-identification (ReID) dataset as the core for appearance feature extraction. Adapted to cross-frame pedestrian identity association tasks, ResNet-50, with its deep residual structure, possesses powerful high-dimensional feature fitting and detail representation capabilities, effectively capturing fine-grained differential features such as pedestrian clothing texture, color distribution, body contours, and accessory details. Furthermore, the network is pre-trained on a large-scale public person re-identification dataset, rather than a general image classification dataset. This ensures the model weights are highly adapted to pedestrian identity differentiation tasks, exhibiting strong resistance to lighting changes, pose fine-tuning, image scale variations, and slight occlusion interference. The output feature vectors demonstrate high intra-class similarity and high inter-class discriminative power. Combined with the pure human body region image input after masking purification, ResNet-50 can output pedestrian-specific deep appearance feature vectors with fixed dimensions and high robustness, providing reliable feature support for third-level cosine similarity matching. This ensures accurate cross-frame association without ID jumps in dense crowd scenes.

[0054] Example Step 1: Model and detector initialization 1.1 Head Detection Model Loading and Initialization Load the pre-trained head detection model weight file and initialize YOLOv8-det (YOLO series object detector). Configure detector parameters: set the confidence threshold to 0.5, the IoU threshold for non-maximum suppression (NMS) to 0.45, and uniformly scale the input image resolution to 640×640 pixels. The detector output is the bounding box coordinates (x_min, y_min, x_max, y_max) of each head region and the corresponding confidence score.

[0055] 1.2 Video Segmentation Model Loading and Initialization Load the pre-trained video instance segmentation model weight file and initialize YOLOv8n-seg (a YOLO segmenter). Configure the segmenter parameters: set the confidence threshold to 0.5, the IoU threshold for NMS to 0.45, and uniformly scale the input image resolution to 640×640 pixels. The segmenter output is a set of segmentation polygon coordinates {(x1,y1),(x2,y2),...,(x...}} for each pedestrian instance. n ,y n )}, and the corresponding confidence score.

[0056] 1.3 Feature Extraction Model Loading and Initialization Load the pre-trained feature extraction model ResNet-50 and initialize the feature extractor. This model is pre-trained on the person re-identification dataset Market-1501 and outputs a depth appearance feature vector with dimensions of 512 or 256. In this embodiment, 512 dimensions are used, but in actual applications, the dimensions can be adjusted according to computing resources. Configure the feature extractor parameters: crop the input image to 128×384 pixels and normalize it to the [0,1] interval.

[0057] Step 2: Video Frame Preprocessing Read the video stream file or live camera feed to be processed, and uniformly extract frames at a preset frame rate of 20 frames per second. For each extracted frame, perform the following preprocessing operations: Convert the image from the BGR color space to the RGB color space; According to the input requirements of the detector and segmenter, the image is scaled to 640×640 pixels, maintaining the aspect ratio and filling the gray border; Normalize the pixel values ​​to the [0,1] range; Convert to tensor format and add batch dimension.

[0058] Step 3: Multi-task parallel sensing 3.1 Head Detection The preprocessed current frame image is input into the head detector initialized in step 1.1, and forward inference is performed. The detector outputs a set of head detection boxes H={h1,h2,...,h...}n}, where each detection box hᵢ contains the following information: Bounding box coordinates: (x_min_i, y_min_i, x_max_i, y_max_i), in pixels, relative to the original image size; Confidence score: score_i∈[0,1].

[0059] NMS is applied to all detection boxes to remove redundant detection results, and finally the detection boxes with confidence scores higher than the threshold are retained.

[0060] 3.2 Pedestrian Segmentation The preprocessed current frame image is input into the segmenter initialized in step 1.2, and forward inference is performed. The segmenter outputs a set of segmented polygon coordinates for a group of pedestrian instances. Each segmented polygon Given its vertex coordinate sequence {(x1,y1),(x2,y2),...,(x k ,y k The expression )} represents the unit in pixels, relative to the original image size. It also outputs the confidence score for each instance, filtering out results with a confidence score below 0.5.

[0061] Step 4: Feature Extraction and Mapping Construction 4.1 Segmentation Feature Extraction For each segmented polygon obtained in step 3.2 Perform the following operations: (1) Generate a binary mask image based on the vertex coordinates of the polygon. The mask size is the same as the original image. The pixel value inside the polygon is 1, and the pixel value outside is 0. (2) Crop out the smallest bounding rectangle containing the pedestrian instance from the original image. The rectangle's range is determined by the extreme values ​​of the polygon's vertex coordinates. (3) Scale the cropping area to 128×384 pixels and set the pixel values ​​outside the mask area to zero (i.e., only keep the pixels inside the human body outline). (4) Input the processed image into the feature extractor initialized in step 1.3 to obtain the depth appearance feature vector. It has 512 dimensions.

[0062] 4.2 Head-Segmentation Mapping Matching The `intersection()` method from the Boost.Geometry library is used to perform spatial intersection-union (IUU) matching between the head detection bounding box set H from step 3.1 and the segmented polygon set P from step 3.2. The specific steps are as follows: Traverse each head detection box in H : Iterate through each segmented polygon in P : Calculate rectangle and polygons Intersection: Intersections=bg::intersection( , ) Intersections are intersecting polygons. Since the head is rectangular and the body segments are polygons of arbitrary shapes, their intersection may be one or more polygons (for example, when the body surrounds the head, the intersection is the head itself; when the body part covers the head, the intersection is an irregular polygon).

[0063] Accumulate the intersection area: Iterate through each polygon p in the intersections container, calculate its area using bg::area(p), and accumulate it to inter_area. Normally, the intersection has only one polygon, but Boost.Geometry supports returning multiple separate intersection regions (e.g., the body has two separate parts that intersect the head separately), so a loop is used to sum them.

[0064] Calculate the area of ​​the head rectangle: head_area=(x2-x1)*(y2-y1).

[0065] Calculate the partial intersection-union ratio (PIoU): PIoU=inter_area / head_area.

[0066] Select the part with the largest intersection-union ratio To map polygons, establish a mapping table M={(bᵢ,f j )|bᵢ∈B,f j ∈F}, where F is the set of all successfully matched feature vectors. If the maximum PIOU ≥ θ_iou_map (default 0.3), then a mapping is established; otherwise, the head detection box is not associated with segmentation features for the time being, and its SFEAT is left blank.

[0067] For segmented polygons that do not match any head detection bounding boxes, they are considered missed targets and will not be included in the tracking system for the time being; for head detection bounding boxes that do not match any segmented polygons, they are considered newly appearing targets, but their feature vectors are temporarily empty and will be supplemented in subsequent frames.

[0068] Step 5: Initialization and Management of Historical Track Sets 5.1 Definition of Historical Trajectory Set Load the historical tracking target set T={t1,t2,...,t...} that was processed in the previous frame. k}. Each tracked target t contains the following key attributes as shown in Table 1: Table 1 Historical Tracked Target Attribute Table

[0069] 5.2 State Definition and Transition Rules ACTIVE: The latest frame is successfully matched and MISS_COUNT=0. The ACTIVE state indicates that the target is in a normal tracking state after the processing of the current frame, and this state is determined by the matching result of the previous frame.

[0070] INACTIVE (temporarily lost): The number of consecutive unmatched frames MISS_COUNT of the target satisfies 1≤MISS_COUNT<N, where N is the temporary loss threshold, which is set to 3 frames by default.

[0071] LOST (lost): The number of consecutive unmatched frames MISS_COUNT of the target ≥ N; That is, a target enters the temporary loss state after 1 consecutive unmatched frame, and is downgraded to the lost state after 3 consecutive unmatched frames.

[0072] REMOVED: The number of consecutive lost frames MISS_COUNT of the target ≥ MAX_AGE, where MAX_AGE is the maximum survival frame threshold, which is set to 20 frames by default. A target in the REMOVED state will be deleted from the historical trajectory set.

[0073] 5.3 Initialization Logic At the first startup (the first frame), the historical target set T is empty; In the processing of subsequent frames, T is loaded from the processing result of the previous frame.

[0074] Step 6: Motion State Prediction For all targets whose STATUS is ACTIVE or INACTIVE in the historical trajectory set T, the Kalman filter algorithm is used to predict all tracked targets in the target pool, that is, given the posterior state mean of the previous frame and the posterior covariance , the prior estimate of the current frame is predicted.

[0075] 6.1 State Vector Definition The state vector of the Kalman filter is 8-dimensional, denoted as x:

[0076] where( is the center coordinate of the target detection box, is the aspect ratio, is the height of the detection box, is the corresponding velocity component.

[0077] The measurement vector is 4-dimensional, denoted as :

[0078] 6.2 State transition matrix A constant velocity model is adopted, with a time step Δt=1. The state transition matrix F is an 8×8 matrix.

[0079] This matrix linearly combines the position and velocity components to achieve one-step prediction.

[0080] 6.3 Process Noise Covariance Matrix Based on the current height and preset hyperparameters , Construct an 8-dimensional standard deviation vector:

[0081] The process noise covariance matrix is:

[0082] 6.4 Prediction Given the posterior state estimate of the previous time step and posterior covariance Predict the prior state and covariance at the current moment: (1) State mean prediction

[0083] (2) State covariance prediction

[0084] If the trajectory state is not ACTIVE, then the velocity component will be included before prediction. (Right now Forced to zero to prevent velocity divergence of inactive trajectories.

[0085] After prediction is completed, the mean of the prior states of the current frame is obtained. and prior covariance .

[0086] Step 7: Multi-level data association and matching A three-level progressive matching strategy is adopted, and the following matching operations are performed sequentially. During the matching process, matched detection boxes and trajectories will be marked and will not participate in subsequent levels of matching.

[0087] 7.1 First-level matching: Matching active / temporarily lost targets Matching objects: The current frame's set of head detection boxes, B_current; For targets with STATUS in the historical trajectory set T that are ACTIVE and INACTIVE, their Kalman prediction boxes pred_BBOX are used.

[0088] Matching method: (1) Calculate the IoU value between the current frame detection box and the historical target prediction box: IoU(A,B)=Area(A∩B) / Area(A∪B) (2) Construct the IoU cost matrix; (3) Apply the Hungarian algorithm to find the optimal match and obtain the set of candidate matching pairs; (4) For each candidate matching pair, check whether its IoU value is greater than or equal to θ_iou_1 (default 0.5): if it is satisfied, accept the match; if it is not satisfied, reject the match and classify the corresponding detection box and target into the unmatched set respectively. (5) Output the accepted matching pairs as the first-level matching results.

[0089] Successful match handling: Update the target's BBOX to the coordinates of the head detection box matched in the current frame; Update the target's SFEAT to the feature vector in the head-segmentation feature map table corresponding to the current frame; Update the target's STATUS to ACTIVE; Reset the target's MISS_COUNT to 0; Update the target's FRAME_ID to the current frame number; Increment TRACK_LENGTH of the target by 1.

[0090] Handling matching failures: Unmatched historical targets proceed to the next level of matching; Unmatched detection boxes in the current frame proceed to the next level of matching.

[0091] 7.2 Second-level matching: Lost target recovery matching Matching objects: The set of head detection boxes remaining in the current frame after the first level of matching is B_remain_1; In the historical trajectory set T, the target with STATUS set to LOST is...

[0092] Matching method: (1) Use the last successfully matched box of the LOST target as the reference box; (2) Construct the IoU cost matrix; (3) Apply the Hungarian algorithm to find the optimal match and obtain the set of candidate matching pairs; (4) For each candidate matching pair, check whether its IoU value is greater than or equal to θ_iou_2 (default 0.3): if it is satisfied, accept it; otherwise, reject it and add it to the unmatched set. (5) Output the accepted matching pairs as the second-level matching results.

[0093] Successful match handling: Restore the target's STATUS from LOST to ACTIVE; Update the target's BBOX, SFEAT, and FRAME_ID; Reset MISS_COUNT to 0; Update TRACK_LENGTH.

[0094] Handling matching failures: Unmatched LOST targets proceed to the next level of matching; Unmatched detection boxes in the current frame proceed to the next level of matching.

[0095] Active and temporarily lost targets that remain unmatched after the first-level matching are not included in the second-level matching but are directly retained in the third-level candidate historical target set T_remain. Lost targets that remain unmatched after the second-level matching are also included in T_remain. Therefore, the third-level candidate historical target set T_remain consists of the following two parts: (1) Unmatched active and temporarily lost targets at the first level; (2) Lost targets that are not matched at the second level.

[0096] 7.3 Third-level matching: Appearance feature re-identification matching Matching objects: The set of head detection boxes in the current frame that have not yet matched after the first two layers of matching; The set of historical trajectories T_remain that were not matched after the first two layers of matching (containing targets in ACTIVE, INACTIVE, and LOST states); As described in step 7.2, T_remain consists of active and temporarily lost targets that are not matched in the first level and lost targets that are not matched in the second level.

[0097] Matching method: (1) For each detection box in B_remain_2, obtain its corresponding segmentation feature vector through the mapping table established in step 4.2 (if it does not exist in the mapping table, skip the detection box). (2) For each historical target in T_remain, retrieve its stored SFEAT; For historical targets in T_remain, if their SFEAT is empty (i.e., they have never successfully established a head-segmentation mapping), the target will be removed from the third-level matching and will not participate in the cosine similarity calculation; these targets will directly enter the state update process in step 9, waiting for subsequent frames to supplement features or eventually be removed. (3) Calculate the cosine similarity between the detection box features and the historical target features:

[0098] in and These are two feature vectors, each with a dimension of 512.

[0099] (4) Construct a similarity cost matrix; (5) Apply the Hungarian algorithm to find the optimal match and obtain the set of candidate matching pairs; (6) For each candidate matching pair, check whether the cosine similarity is greater than or equal to θ_feat (default 0.75): if it is satisfied, accept it; otherwise, reject it and add it to the unmatched set. (7) Output the accepted matching pairs as the third-level matching results.

[0100] Successful match handling: Update the target's STATUS to ACTIVE; Update the target's BBOX, SFEAT, and FRAME_ID; Reset MISS_COUNT to 0; Update TRACK_LENGTH.

[0101] Handling matching failures: Unmatched historical targets proceed to step 8 for status update; Unmatched detection boxes in the current frame proceed to step 9 for new target creation.

[0102] Step 8. For already matched targets, update the predicted location using a Kalman filter. Given the measurement value of the current frame and prior estimation and Calculate the posterior estimate.

[0103] 8.1 Measurement Model and Projection The Kalman filter measurement model maps an 8-dimensional state space to a 4-dimensional measurement space. The measurement matrix H is a 4×8 matrix.

[0104] The matrix extracts the first four components of the state (detection box position and size) as predicted measurements.

[0105] First, the predicted state is projected onto the measurement space, that is, given the mean of the prior state. and prior covariance Calculate the predicted measurement mean y and measurement covariance S:

[0106]

[0107] Where R is the measurement noise covariance matrix, derived from the current state. and hyperparameters Adaptive construction:

[0108] 8.2 Kalman Gain Calculation The Kalman gain K is an 8×4 matrix used to balance the confidence levels of prediction and measurement. First, calculate the intermediate matrix:

[0109] Solving using Cholesky decomposition Then calculate the gain:

[0110] 8.3 Post-hoc state update Calculate the measurement residuals:

[0111] in, This represents the actual measured values ​​(detection box coordinates and dimensions) for the current frame.

[0112] Update state mean:

[0113] Update state covariance:

[0114] 8.4 Update the predicted target position based on the state mean From the posterior state mean The first four bits are decoded to obtain the bounding box parameters in the image coordinate system. Let:

[0115] in: The coordinates of the detection box center are, is the aspect ratio, and h is the height.

[0116] The coordinates of the top-left corner (x, y), the width w, and the height h of the rectangle are:

[0117]

[0118]

[0119]

[0120] Step 9: Status Update and Lifecycle Management Handling unmatched historical targets For historical trajectory targets that are still not matched after three levels of matching, perform a status update; The state transition logic is specifically as follows: For historical targets that remain unmatched after multiple levels of matching, increment the consecutive unmatched count MISS_COUNT by 1. Let N be the temporary loss threshold and MAX_AGE be the maximum number of live frames threshold, then: If MISS_COUNT < N (and MISS_COUNT ≥ 1), the target remains in a lost state (if it is already lost) or changes from active to lost (if it was previously active). If N ≤ MISS_COUNT < MAX_AGE, the target changes from a temporarily lost state to a lost state. If MISS_COUNT ≥ MAX_AGE, the target is changed to a removal state and deleted from the historical trajectory set.

[0121] It should be noted that the above logic applies to historical targets that were not matched in the current frame. If a target is successfully matched in the current frame, its MISS_COUNT is reset to 0, and its status is restored to active.

[0122] Preferably, N is 3 frames and MAX_AGE is 20 frames.

[0123] Step 10: Creating a New Goal For head detection boxes in the current frame that still do not match after three levels of matching, they are considered newly appearing targets, and the following creation operations are performed: (1) Allocate a new globally unique TRACK_ID, using an auto-incrementing integer, with an initial value of 0; (2) Set BBOX to the coordinates of the head detection box in the current frame; (3) Obtain the corresponding SFEAT through the mapping table in step 4.2: If a corresponding segmentation feature vector exists in the mapping table, then that feature vector is assigned to the new target; If there is no corresponding segmentation feature vector in the mapping table (i.e., the head detection box fails to match any segmentation mask), the SFEAT of the target is marked as empty, and it will participate in position-based matching (first level and second level) in subsequent frames first. The feature vector will be supplemented after the head-segmentation mapping is successfully established in subsequent frames. If the mapping cannot be established in several consecutive frames, the target will be tracked only by position information. (4) Set STATUS to ACTIVE; (5) Set FRAME_ID to the current frame number; (6) Set MISS_COUNT to 0; (7) Set START_FRAME to the current frame number; (8) Set TRACK_LENGTH to 1; (9) Add the new target to the historical trajectory set T.

[0124] Step 11: Update the historical trajectory set Update all successfully matched targets (with updated attributes) from step 7, targets with updated status from step 9, and newly created targets from step 10 to the historical trajectory set T. Remove targets with a REMOVED status to release the memory resources they occupy.

[0125] Step 12: Output Results 12.1 Tracking Result Output Iterate through the updated historical trajectory set T, selecting all tracked targets with a STATUS of ACTIVE, and use this as the final tracking result for the current frame. Output information includes: TRACK_ID: Tracking target number; BBOX: Coordinates of the head detection bounding box in the current frame; SFEAT: Human body segmentation feature vector of the current frame; FRAME_ID: Current frame number; TRACK_LENGTH: The total number of frames in which the target has been successfully tracked.

[0126] 12.2 Output Format Supports multiple output formats: Video file: The tracking results are directly overlaid on the original video and then output. JSON / XML files: Output the tracking results of each frame in a structured data format for easier subsequent analysis; Real-time display: The tracking results are displayed on the screen in real time.

[0127] Step 13: Iterative Processing Increment the frame number of the current frame by 1, jump to step 2, read the next frame image, and repeat steps 2 to 12 until the video stream ends.

[0128] The parameter configurations are summarized in Table 2: Table 2 Summary of Parameter Configurations

[0129] See Figure 3 A pedestrian multi-target tracking system based on head detection and body segmentation includes: The perception module is used to perform head detection and pedestrian instance segmentation on video frames to obtain head detection boxes and pedestrian segmentation masks; The feature module is used to extract depth appearance features from the pedestrian segmentation mask and establish a mapping relationship between the human head and the segmentation features; The tracking module is used to maintain historical trajectory sets, perform Kalman filter prediction and updates, perform multi-level data association matching based on location and features, and manage the tracking status and lifecycle of the target.

[0130] Specifically, the perception module performs dual-layer intelligent perception on each frame of the input video image, completing head target detection and pedestrian instance segmentation respectively, and outputting two types of basic data: head detection bounding boxes and pedestrian segmentation masks. Traditional pedestrian tracking relies solely on full-body bounding boxes for localization, which has significant drawbacks: in crowded situations, when pedestrians overlap, or when limbs are occluded, full-body bounding boxes are prone to overlapping and sticking together, leading to blurred target localization and failure to distinguish between different targets. On the other hand, simple body segmentation lacks lightweight keypoint localization, is computationally redundant, and is easily affected by limb posture. This module adopts a complementary perception strategy of head detection and pedestrian instance segmentation, balancing localization accuracy and scene adaptability. In the head detection stage, the target detection algorithm focuses on the head region, which is least likely to be occluded and has the highest distinguishability, for detection. The head has a small volume, stable features, and low overlap probability, i.e. Even in scenes with multiple people brushing shoulders, limbs obscuring the view, or half-body shots, the system can still accurately output the bounding rectangle detection box for each pedestrian's head, recording the pixel coordinates, size, and confidence information of the head. This enables rapid localization and preliminary differentiation of individual pedestrians in complex scenes, reducing the problem of confusion among dense targets at its source. In the pedestrian instance segmentation stage, instance-level semantic segmentation is performed on all pedestrians in the image. Instance segmentation can distinguish each individual pedestrian, removing background interference such as ground, buildings, vehicles, and greenery. Finally, a binary segmentation mask corresponding to each pedestrian is output. This mask can outline the complete shape of a single pedestrian, including pixel-level boundaries of the torso and limbs, and remove pedestrian pixels in overlapping areas. This solves the problem that the detection box cannot distinguish overlapping pedestrians, providing a clean target area for extracting specific pedestrian features.

[0131] The feature module includes a feature extraction unit and a mapping matching unit. The feature extraction unit extracts a depth appearance feature vector from the pedestrian segmentation mask. For each pedestrian instance detected in each frame, the feature extraction unit first crops the corresponding pedestrian region from the original image according to the segmentation mask, sets the pixels outside the mask to zero, and retains only the effective pixels within the human body contour. Then, the purified pedestrian image is input into a pre-trained deep convolutional neural network and encoded into a fixed-dimensional depth appearance feature vector through forward inference. This feature vector represents fine-grained appearance information such as the color, texture, and body contour of the pedestrian's clothing. The mapping matching unit spatially matches the head detection box with the pedestrian segmentation mask and establishes a mapping relationship. For the set of head detection boxes and pedestrian segmentation masks in the current frame, the mapping and matching unit calculates the partial intersection-union ratio (PIoU) of each head detection box and each segmentation mask, which is the proportion of the intersection area of ​​the two to the area of ​​the head detection box. The segmentation mask with the largest PIO is selected as the candidate matching object. When the largest PIO reaches the preset mapping threshold, the two are determined to be a valid match. After a successful match, the head detection box, the corresponding segmentation mask, and the depth appearance feature vector output by the feature extraction unit are bound together and stored in the head-segmentation feature mapping table for subsequent level matching.

[0132] The tracking module assigns a unique ID to each detected pedestrian target in the image and establishes a historical trajectory set; it uses a Kalman filter unit to predict and correct motion state, a multi-level matching unit to complete inter-frame association based on position and appearance features, and a state management unit to manage the entire lifecycle of the target from addition, tracking, occlusion and lingering to departure and disappearance.

[0133] The Kalman filter unit is responsible for predicting the motion state of pedestrian targets and updating the posterior time.

[0134] The system manages the entire lifecycle of all pedestrian targets, including new targets, continuous tracking, occlusion-induced lingering, and departure / death. For new pedestrian targets detected in the current frame without matching historical trajectories, a new ID is automatically assigned, trajectory information is initialized, and the target is added to the tracking list. For targets that are successfully matched between frames, the trajectory and status are continuously updated to maintain continuous tracking. For targets that are temporarily lost due to occlusion or exiting the frame, the trajectory is not immediately deleted; instead, a lingering timer is activated to retain their historical features and trajectory, allowing for rapid matching and resumption of tracking when the target reappears. For targets that have been absent for an extended period and have not been detected within a timeout period, they are automatically classified as absent, their tracking ID is deregistered, system resources are released, invalid data accumulation is avoided, and the long-term stable operation of the system is ensured.

[0135] The tracking module includes: Kalman filter unit, used for target motion state prediction and posterior state update; A multi-level matching unit is used to sequentially perform first-level matching based on intersection-union ratio (CIRR), second-level matching based on CIRR, and third-level matching based on cosine similarity. The status management unit is used to update the target tracking status and manage the lifecycle based on the matching results.

[0136] Specifically, the Kalman filter unit in the tracking module is responsible for temporal modeling of pedestrian target motion information in video frames, completing prior motion state prediction and posterior state correction and update. In the motion state prediction stage, based on the temporal motion data of the pedestrian's historical trajectory, a motion model adapted to the nonlinear micro-motion characteristics of the pedestrian is constructed. Using the head coordinates, movement speed, and movement direction of the target in past frames, the theoretical optimal position and movement rate of the target in the current frame are calculated as prior state information. For dynamic changes such as pedestrian walking speed changes, small turning backs, and posture shifts, the filter unit can predict the target's movement trend based on historical motion patterns and lock the area where the target is likely to appear in the current frame in advance. Even if the pedestrian is momentarily missed due to occlusion or image blur, the trajectory continuity can be maintained through the prediction results to avoid direct interruption of the target trajectory. In the posterior state update stage, the actual head detection position output by the current frame perception module is compared and verified with the prior position predicted by the filter. Combining the observation noise and motion noise weights, the prediction deviation is dynamically corrected and the optimal motion state parameters of the target are iteratively updated.

[0137] The multi-level matching unit adopts a three-level progressive matching mechanism of two-level intersection-union ratio (IOU) coarse and fine screening and cosine similarity feature fine correction. It verifies from shallow to deep, from position to feature, taking into account both tracking real-time performance and matching accuracy, and is suitable for complex scenarios with dense crowds, overlapping targets, and varied postures.

[0138] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for multi-target pedestrian tracking based on head detection and body segmentation, characterized in that, Includes the following steps: S1. Using a pre-trained head detection model and instance segmentation model, perform head detection and pedestrian instance segmentation on the current frame of the video respectively to obtain a set of head detection boxes and a set of pedestrian segmentation maps; S2. Based on the pedestrian segmentation maps, a pre-trained feature extraction model is used to extract depth appearance feature vectors from the original images. Each head detection box is spatially matched with the corresponding pedestrian segmentation map and depth appearance feature vector to establish a head-segmentation feature mapping table. S3. Obtain the currently existing historical tracking targets, perform Kalman filter motion prediction based on the historical tracking targets, and sequentially perform the first-level matching based on location, the second-level matching based on location, and the third-level matching based on appearance features in the head-segmentation feature mapping table. Update the Kalman filter posterior state using the detection box of the current frame according to the matching results of each level. S4. Based on the matching results of the multi-level data association, update the continuous unmatched count and tracking status of each historical tracking target, create new targets for the unmatched current frame detection box, and remove overdue unmatched targets. S5. Output the tracking result of the current frame, jump to S1 to process the next frame, until the video stream ends.

2. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 1, characterized in that, In step S2, the depth appearance feature vector is extracted from the original image, including: A segmentation mask is generated using the pedestrian segmentation map. Based on the segmentation mask, the pedestrian region is cropped from the original image to obtain pixels within the human body contour. The pixels within the human body contour are then input into the feature extraction model to obtain a depth appearance feature vector. The spatial matching includes: The ratio of the intersection area of ​​the head detection box and each pedestrian segmentation map to the area of ​​the head detection box is calculated as the partial intersection-union ratio (MIR). The pedestrian segmentation map with the largest MIR is selected as the matching object. When the MIR reaches a preset mapping threshold, a mapping relationship is established and the corresponding feature vector is associated.

3. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 2, characterized in that, Calculate the intersection area between the head detection bounding box and each pedestrian segmentation map, including: The intersection set between the head detection box and the polygons of the pedestrian segmentation map is calculated using a geometric operation library. The intersection set is a set of one or more intersecting polygons. The sum of the areas of each intersecting polygon in the intersection set is taken as the intersection area. If the maximum intersection-union ratio does not reach the preset mapping threshold, the head detection box will not be associated with segmentation features for the time being, and its corresponding feature vector will be set to empty, waiting for features to be added in subsequent frames.

4. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 3, characterized in that, The geometry operation library uses the Boost.Geometry library. The intersection() function of the Boost.Geometry library is used to calculate the intersection of the head detection box and the polygons of the pedestrian segmentation map, and the area() function of the Boost.Geometry library is used to calculate the area of ​​each polygon in the intersection. For pedestrian segmentation maps that do not match any head detection bounding boxes, they are considered missed targets and are not included in the tracking system for the time being; for head detection bounding boxes that do not match any pedestrian segmentation map, they are considered newly appearing targets, and their tracking feature vectors are temporarily empty, waiting to be supplemented in subsequent frames.

5. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 1, characterized in that, In S3, the first hierarchical matching is as follows: taking the historical target in the active state and the historical target in the temporarily lost state as the first matching objects, the predicted box obtained by Kalman filtering of each of the first matching objects is matched with the head detection boxes of the current frame by intersection-union comparison. The second-level matching is as follows: taking the historical target with the state of being lost as the second matching object, taking the head detection box that was last successfully matched in each of the second matching objects as the reference box, and performing cross-union matching with each of the remaining head detection boxes after the first-level matching, and the second-level cross-union threshold is less than the first-level cross-union threshold; The third-level matching is as follows: the remaining head detection boxes after the first two levels of matching are matched with the remaining historical targets after the first two levels of matching, and cosine similarity matching is performed based on the appearance feature vectors in the head-segmentation feature mapping table.

6. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 1, characterized in that, In step S4, the tracking state includes active, temporarily lost, lost, and removed, and each state transitions based on the number of consecutive unmatched frames of the historical target: When the number of consecutive unmatched frames is 0, the target is in the active state; When the number of consecutive unmatched frames is greater than or equal to 1 and less than 3, the target is in the temporary loss state; When the number of consecutive unmatched frames is greater than or equal to 3 and less than 20, the target is in the lost state; When the number of consecutive unmatched frames is greater than or equal to 20, the target is in the removal state and is deleted from the historical tracked target set.

7. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 1, characterized in that, In S3, for targets in an inactive state, the altitude change rate in their velocity component is set to zero before Kalman filter prediction.

8. The pedestrian multi-target tracking method based on head detection and body segmentation according to claim 1, characterized in that, The head detection uses a YOLOv8-det detector; The instance segmentation uses the YOLOv8n-seg segmenter; The feature extraction employs a ResNet-50 network pre-trained on a pedestrian re-identification dataset.

9. A pedestrian multi-target tracking system based on head detection and body segmentation, characterized in that, include: The perception module is used to perform head detection and pedestrian instance segmentation on video frames to obtain head detection boxes and pedestrian segmentation masks; The feature module is used to extract depth appearance features from the pedestrian segmentation mask and establish a mapping relationship between the human head and the segmentation features; The tracking module is used to maintain historical trajectory sets, perform Kalman filter prediction and updates, perform multi-level data association matching based on location and features, and manage the tracking status and lifecycle of the target.

10. The pedestrian multi-target tracking system based on head detection and body segmentation according to claim 9, characterized in that, The tracking module includes: Kalman filter unit, used for target motion state prediction and posterior state update; A multi-level matching unit is used to sequentially perform first-level matching based on intersection-union ratio (IU), second-level matching based on IU, and third-level matching based on cosine similarity. The status management unit is used to update the target tracking status and manage the lifecycle based on the matching results.

Images