A multi-target tracking and monitoring method and system with high trajectory consistency

Abstract

A multi-target tracking and monitoring method and system with high trajectory consistency, belonging to the field of computer vision, solves the problems of identity switching and trajectory fragmentation caused by long-term target occlusion and dense interaction in multi-target tracking, especially in UAV monitoring scenarios. The method includes the following steps: 1. Inputting a video sequence, performing target detection and feature acquisition for each frame; 2. Performing preliminary association through a basic tracker to obtain an initial trajectory; 3. Detecting the disappearance and appearance of abnormal trajectories using a position awareness submodule; 4. Storing abnormal trajectory features using a long-term memory submodule; 5. Performing secondary association using a multi-step cross-frame matching submodule; 6. Completing trajectory updates. This invention is applicable to UAV monitoring scenarios such as high-altitude inspection, low-altitude operations, and outdoor scene monitoring.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to multi-target tracking technology, and particularly to a high trajectory consistency multi-target tracking method based on lightweight appearance feature association matching. Background Technology

[0002] Multi-target tracking is a core task in computer vision, involving the continuous identification and maintenance of the unique identity of multiple dynamic targets in a video sequence. This technology has wide applications in autonomous driving, intelligent surveillance, and sports analytics, with a particularly urgent need in drone surveillance. Drones, with their advantages of maneuverability, wide coverage, and lack of ground-based limitations, are widely used in outdoor safety monitoring, high-altitude inspections, emergency rescue, and park management. Accurate multi-target tracking is the core support for drones to achieve intelligent monitoring and efficiently complete tasks.

[0003] Existing multi-object tracking methods are mainly divided into the Joint Detection and Embedding (JDE) paradigm and the Detection-Based Tracking (TBD) paradigm. The TBD paradigm, due to its guaranteed detection accuracy, has become the mainstream method. It mainly consists of two core components: a Kalman filter-based motion predictor and a deep appearance feature extractor. This field subsequently diverged into two development paths. One direction pursues algorithm simplification, represented by ByteTrack, which, leveraging the YOLOX real-time detector, achieves state-of-the-art performance through refined motion modeling, demonstrating that well-designed heuristics can outperform complex architectures in many scenarios. In contrast, methods such as BotSORT and StrongSORT, employing enhanced motion models and appearance feature optimization, bring gains in complex scenes, but at the cost of higher computational overhead.

[0004] It is particularly important to note that drone surveillance scenarios present more complex tracking challenges compared to traditional fixed-view surveillance: the drone's perspective changes dynamically during flight, the shooting distance fluctuates significantly, and the surveillance scenarios are mostly open outdoor areas prone to problems such as dense target interactions, prolonged occlusion (e.g., crowds, vehicle intersections, tree / building obstructions), and variable lighting conditions (e.g., backlighting, shadows, day / night cycles). These issues further exacerbate the tracking difficulty. Recent innovations have addressed specific shortcomings. The Transformer architecture has recently reshaped the field of detection-based tracking, providing a new approach to feature representation. Although TransTrack uses DETR's attention mechanism for tracking tasks, comprehensive evaluations show that these methods still suffer from trajectory fragmentation issues in complex occlusion scenarios during drone surveillance, especially when dealing with prolonged occlusion from the drone's perspective, similar-looking targets (e.g., batches of similar vehicles, uniformly dressed personnel), and abnormal movements (e.g., sudden changes in target direction, rapid movement). When a target is obscured for a long time or disappears abnormally due to drone shaking or excessive distance causing detector failure, its motion estimation becomes unreliable. Its appearance features may also degrade due to changes in lighting, viewpoint shift, or partial visibility, causing the tracker to be unable to re-identify the target, resulting in identity switching (IDSW) and trajectory fragmentation (Frag), which seriously affects the accuracy and practicality of drone monitoring. Summary of the Invention

[0005] This invention provides a multi-target tracking and monitoring method and system with high trajectory consistency. Its purpose is to solve the problems of identity switching and trajectory fragmentation caused by long-term target occlusion and dense interaction in multi-target tracking, especially in UAV monitoring scenarios, improve trajectory consistency and re-identification accuracy, and adapt to the application needs of UAV dynamic perspective and complex outdoor environment.

[0006] Firstly, the present invention aims to provide a multi-target tracking and monitoring method with high trajectory consistency, comprising the following steps: S1. Use the detection-tracking paradigm to detect targets in each frame of the video sequence to obtain a set of detection boxes. Then, associate the set of detection boxes with the existing trajectory set to obtain preliminary tracking results. S2. Determine whether the trajectory has not matched any detection box in K consecutive frames. If so, trigger the position-aware trajectory verification module and determine whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory. S3. For trajectories that are determined to have disappeared abnormally, extract multi-frame features from their historical frames, store the multi-frame features in a long-term memory bank, and use a self-attention mechanism to fuse global appearance features and local body part features. S4. When a new trajectory is detected, the validity of the new trajectory is first verified by the location-aware trajectory verification module. If it is determined to be an invalid new trajectory, the multi-step cross-frame matching module is triggered. The invalid new trajectory is matched with the abnormally disappeared trajectory in the long-term memory through the multi-step cross-frame matching module. The double threshold matching strategy is used to realize the re-identification of the abnormally disappeared trajectory. S5. Update the status of all trajectories according to the matching results of the multi-step cross-frame matching module, update the successfully matched abnormal disappearance trajectories to the active state, keep the unmatched abnormal disappearance trajectories in the pending recovery state, and delete the relevant records of normally terminated trajectories.

[0007] Furthermore, a preferred embodiment is provided: S1 includes: An initial trajectory sequence is obtained, and a target detector is used to detect video frames to obtain detection boxes. Motion prediction and appearance feature extraction operations are performed on the detection boxes in sequence. Then, the detection boxes with extracted features are associated with the previous target boxes to obtain preliminary tracking results. The target detector is any one of the YOLO, Fast-RCNN, or DETR algorithm models; the motion prediction is implemented using the Kalman filter state estimation method, and the appearance feature extraction is implemented using the lightweight network model OsNet; the association between the detection box and the preceding target box is determined by weighted combination of the Mahalanobis distance of the motion branch and the cosine distance of the appearance branch. When the weighted combination result is lower than a preset threshold, the target is determined to be successfully matched with the trajectory and the trajectory box is updated.

[0008] Furthermore, a preferred solution is provided: In step S2, the location-aware trajectory verification module determines whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory, specifically including: When the trajectory is continuous When no detection box is matched in the frame, the positional relationship between the center point coordinates of the last detection box that appears on the trajectory and the image boundary is calculated. If the preset boundary discrimination condition is met, the trajectory is determined to be a normal boundary termination trajectory. If the boundary discrimination condition is not met, then the trajectory is... Frame timing consistency verification, if the trajectory is continuous If no detection box is matched in the frame, it is determined to be an abnormally disappearing trajectory; if the trajectory is continuous The continuous detection of targets in the frame indicates that the trajectory has been abnormally newly added. in, and The value of is 12.

[0009] Furthermore, a preferred embodiment is provided: S3 includes: For the abnormal disappearance trajectory marked as to be recovered, multi-frame features from its historical frames are extracted. The global appearance features of the detection box are extracted through a pre-trained CNN network, and the local features of the target are extracted through a human pose estimation network. The global appearance features and local features are fused using a self-attention mechanism to obtain fused features; The fused features are associated with the trajectory's ID, occurrence frame number, and spatial location information and stored in a long-term memory.

[0010] Furthermore, a preferred embodiment is provided: S3 further includes a memory management strategy for the long-term memory bank, the memory management strategy specifically including: Effective feature memory strategy: Extract and store the fused features for each trajectory according to the rule of interval T frames; Periodic cleaning strategy: Perform cleaning operations on the long-term memory at preset cycles, deleting trajectory features that have not matched beyond the time threshold and outdated features with feature similarity below the threshold; if the number of entries retained in the memory exceeds the maximum capacity, delete the oldest entry.

[0011] Furthermore, a preferred solution is provided: In step S4, the re-identification of the abnormal disappearance trajectory using a dual threshold matching strategy specifically includes: The invalid new trajectory is input into the multi-step cross-frame matching module. The feature distance between the features of the new trajectory and the features of all trajectory features in the long-term memory is calculated, and the Top-K candidate features with the smallest feature distance are selected. Calculate the matching frequency and average feature distance of the trajectory ID to which each candidate feature belongs; A dual-threshold matching strategy is used to determine the matching result. When both the frequency matching condition and the feature distance matching condition are met, the new trajectory is determined to be a successful match with the corresponding anomaly disappearance trajectory. If the valid frame of the new trajectory matches multiple stored trajectories, the trajectory with the smallest average feature distance is selected as the matching result. After a successful match, the ID of the abnormal disappearance trajectory is assigned to the new trajectory, and the relevant entry for the abnormal disappearance trajectory is deleted from the long-term memory.

[0012] Furthermore, a preferred solution is provided: In step S5, updating the state of all trajectories based on the matching results of the multi-step cross-frame matching module specifically includes: For a new trajectory that is successfully matched, update its status to active, assign a corresponding abnormal disappearance trajectory ID, and update its appearance features and spatial location information. For abnormal disappearance trajectories in the pending recovery state, if the current frame still does not match, the pending recovery state is maintained, and the last missing frame number is updated. For normally terminated trajectories, their records are deleted from the existing trajectory set. If the feature entries of the trajectory exist in the long-term memory, they are deleted simultaneously. The system counts the number of active trajectories, trajectories awaiting recovery, and newly added trajectories in the current frame for system status monitoring and performance evaluation.

[0013] Secondly, the purpose of this invention is to propose a multi-target tracking and monitoring system with high trajectory consistency, said system being implemented based on a multi-target tracking and monitoring method with high trajectory consistency as described in any one or more of the above-mentioned schemes, said system comprising: The detection and association module is used to detect targets in each frame of the video sequence using the detection-tracking paradigm, obtain a set of detection boxes, and associate the set of detection boxes with the existing trajectory set to obtain preliminary tracking results. The trajectory verification module is used to determine whether the trajectory has not matched any detection box in K consecutive frames. If so, the position-aware trajectory verification module is triggered to determine whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory. Feature fusion and storage module: For trajectories that are determined to have disappeared abnormally, extract multi-frame features from their historical frames, store the multi-frame features in a long-term memory bank, and use a self-attention mechanism to fuse global appearance features and local body part features. Cross-frame matching module: When a new trajectory is detected, the validity of the new trajectory is first verified by the position-aware trajectory verification module. If the new trajectory is determined to be invalid, the multi-step cross-frame matching module is triggered. The multi-step cross-frame matching module matches the invalid new trajectory with the abnormally disappeared trajectory in the long-term memory. A dual threshold matching strategy is used to realize the re-identification of the abnormally disappeared trajectory. The trajectory status update module is used to update the status of all trajectories based on the matching results of the multi-step cross-frame matching module. It updates the successfully matched abnormally disappeared trajectories to the active state, keeps the unmatched abnormally disappeared trajectories in the pending recovery state, and deletes the relevant records of normally terminated trajectories.

[0014] Thirdly, the present invention aims to provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and when the processor runs the computer program stored in the memory, the processor executes a high trajectory consistency multi-target tracking and monitoring method according to any one or more of the above-described schemes.

[0015] Fourthly, the present invention aims to provide a computer-readable storage medium for storing a computer program that executes a high trajectory consistency multi-target tracking and monitoring method described in any one or more of the above-described schemes.

[0016] Compared with the prior art, the advantages of the present invention are: This invention proposes a multi-target tracking and monitoring method with high trajectory consistency. First, it introduces a novel position-aware trajectory verification mechanism that integrates boundary-based single-frame discrimination with K-frame temporal consistency verification, effectively combining spatial constraints and temporal continuity. This mechanism exhibits stronger robustness in distinguishing between normal trajectory termination and abnormal trajectory disappearance, thereby significantly reducing misjudgments when the target reappears.

[0017] Secondly, this invention addresses the lack of cross-frame memory in existing tracking methods when dealing with long-term occlusion by proposing a long-term memory storage module. The core innovation of this module lies in dynamically storing multi-frame features of abnormally lost trajectories and maintaining memory efficiency through an automatic pruning mechanism. Simultaneously, it employs a self-attention fusion strategy of global and local features to improve the discriminative power of features under occlusion and pose changes.

[0018] Furthermore, this invention addresses the matching ambiguity problem in the cross-frame re-identification process by proposing a multi-step cross-frame matching module. It employs a dual threshold strategy of frequency matching and spatial distance to reduce the false matching rate and improve the accuracy of re-identification.

[0019] Finally, this invention is plug-and-play, and can be seamlessly integrated into any existing TBD tracker without modifying the core algorithm of the baseline tracker, making it easy to deploy and promote its application.

[0020] This invention significantly improves trajectory consistency in multi-target tracking systems under dense scenes and long-term occlusion, reduces the number of identity switching events, and enhances identity-related metrics such as IDF1, IDR, and IDP. On the Emporium dense scene dataset, this invention improved the IDF1 metric by 10.29% and reduced the number of identity switching events by 17.2%. On the MOT17 standard dataset, the HOTA metric was improved by an average of 1.3-1.7%.

[0021] The significance of this invention lies in its significant improvement of the trajectory consistency and robustness of multi-target tracking systems in complex scenarios, enabling more accurate maintenance of target identity information in densely populated areas such as shopping mall entrances and subway stations, and providing more reliable data support for downstream applications such as passenger flow analysis, behavior recognition, and anomaly detection.

[0022] This invention is applicable to fields such as drone monitoring, video surveillance, autonomous driving, and intelligent transportation, and is especially suitable for drone monitoring scenarios such as high-altitude drone inspection, low-altitude operation, and outdoor scene monitoring. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the process of obtaining the set of detection boxes and the initial trajectory set using the detection-tracking (TBD) paradigm; Figure 2 The present invention proposes an overall architecture diagram of the PMMM module, which illustrates the collaborative workflow of the three sub-modules: position awareness, long-term memory storage, and multi-step cross-frame matching. Figure 3 This is a flowchart illustrating the workflow of the location-aware trajectory verification module, demonstrating boundary-based single-frame discrimination and... Detailed steps for frame timing consistency verification; Figure 4 The flowchart of the multi-step cross-frame matching module shows the specific implementation of candidate retrieval and dual threshold matching strategy; Figure 5 A graph showing the influence of position module parameters; Figure 6 The graph shows the impact of multi-step cross-frame matching parameters. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0028] Implementation Method 1 This implementation proposes a high trajectory consistency multi-target tracking method based on location awareness and multi-step memory matching, including the following steps: S1. The detection-tracking (TBD) paradigm is used to detect targets in each frame of the video sequence, resulting in a set of detection boxes. Combine the set of detection boxes with the existing set of trajectories. Data correlation was performed to obtain preliminary tracking results; S2, Determine the trajectory Is it in a continuous If no detection box is matched in the frame, the position-aware trajectory verification module is triggered. The position-aware trajectory verification module determines whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory. S3. For trajectories that are determined to have disappeared abnormally, extract multi-frame features from their historical frames, store the multi-frame features in a long-term memory bank, and use a self-attention mechanism to fuse global appearance features and local body part features. S4. When a new trajectory is detected First, the validity of the new trajectory is verified by the location-aware trajectory verification module. If it is determined to be an invalid new trajectory, the multi-step cross-frame matching module is triggered. The invalid new trajectory is matched with the abnormal disappearance trajectory in the long-term memory through the multi-step cross-frame matching module. The double threshold matching strategy is used to realize the re-identification of the abnormal disappearance trajectory. S5. Update the status of all trajectories according to the matching results of the multi-step cross-frame matching module, update the successfully matched abnormal disappearance trajectories to the active state, keep the unmatched abnormal disappearance trajectories in the pending recovery state, and delete the relevant records of normally terminated trajectories.

[0029] Implementation Method 2 This embodiment is a further illustrative example of S1 in the high trajectory consistency multi-target tracking method based on position awareness and multi-step memory matching described in Embodiment 1.

[0030] S1 specifically includes the following steps: An initial trajectory sequence is obtained, and a target detector is used to detect video frames to obtain detection boxes. Motion prediction and appearance feature extraction operations are performed on the detection boxes in sequence. Then, the detection boxes with extracted features are associated with the previous target boxes to obtain preliminary tracking results. The target detector can employ algorithms such as YOLO, Fast-RCNN, or DETR.

[0031] The motion prediction is achieved using the Kalman filter state estimation method, expressed as: , (1) , (2) in, To detect the target in The x-coordinate of the center position of the rectangle at time [time]. To detect the target in The ordinate of the center position of the rectangle at time [time]. This indicates the width of the rectangle. This indicates the height of the rectangle. and It is the first derivative at the center position, representing the velocity at the center position and reflecting the target's positional dynamics. and This represents the scale variation of the target. The target detection algorithm outputs a target observation vector (the center position, width, and height of the target rectangle). The correlation between the trajectory and the target box is obtained by comparing the state estimation vector with that of the Kalman filter.

[0032] The appearance feature extraction is implemented using the lightweight network model OsNet: , (3) in, Indicates the first The first goal in Extract the appearance feature vector of the target bounding box in the frame. Indicates the first The trajectory in the first Frame appearance feature update results It is a dynamic weighting parameter that updates the appearance features of the trajectory when the target matches the trajectory of the preceding frame.

[0033] The association between the detection box and the preceding target box is determined by weighted combination of the Mahalanobis distance of the motion branch and the cosine distance of the appearance branch. When the weighted combination result is lower than a preset threshold, the target is determined to be successfully matched with the trajectory, and the trajectory box is updated. Specifically: The trajectory is associated with the target bounding box. The motion branch uses the Mahalanobis distance formula to measure the correlation between the trajectory and the target, through: , (4) in, Indicates the first Location information of each pedestrian detection box. Indicates the first Predicted location information for a tracked target. This represents the covariance between the detection position and the average tracking position calculated by the Kalman filter. The Mahalanobis distance transforms the uncertainty of the state measurement by calculating the standard deviation between the detection position and the average tracking position. If the matched Mahalanobis distance is less than a preset threshold, the motion state is considered successfully associated. The appearance branch uses cosine distance to calculate the similarity between two feature vectors, through: , (5) in, Distance for appearance features and They represent the first The appearance feature vector of the trajectory and the first The appearance feature vector of each target.

[0034] Finally, the association is determined by a weighted combination of motion prediction and appearance features to obtain the initial trajectory ID. , (6) in Indicates the distance of the IOU of motion. It is a weighted distance for matching. If the distance is lower than the threshold, the target and the trajectory are successfully matched, and the trajectory bounding box is updated accordingly.

[0035] Implementation Method 3 This embodiment is a further illustrative example of S2 in the high trajectory consistency multi-target tracking and monitoring method described in Embodiment 1.

[0036] In S2, the flowchart of the location verification module is as follows: Figure 3 As shown, the location-aware trajectory verification module determines whether the trajectory terminates normally or disappears abnormally. Specifically, this includes: When the trajectory In continuous When no bounding box is associated with a frame, the position-aware trajectory verification module is triggered. First, the trajectory is calculated. The coordinates of the center point of the final detection frame If the positional relationship with the image boundary meets the preset boundary discrimination conditions, then the trajectory is determined to be a normal boundary termination trajectory. The boundary discrimination conditions are: , (7) in, This indicates the x-coordinate of the center of the detection box when the trajectory last appears. This represents the center coordinate of the detection box when the trajectory last appears. The width of the image. The height of the image. Boundary discrimination threshold. and Used to determine whether a target is close to the image boundary, where, Threshold for determining the horizontal boundary This is the threshold for determining the vertical boundary. When a target approaches the boundary, it indicates that it is leaving the frame. In this case, even if the trajectory is interrupted, it is judged as a normal trajectory termination, rather than an abnormal disappearance.

[0037] If the boundary discrimination condition is not met, then the trajectory is... Frame timing consistency verification, if the trajectory is continuous If no detection box is matched in the frame, it is determined to be an abnormally disappearing trajectory; if the trajectory is continuous The continuous detection of targets in the frame indicates an abnormally newly added trajectory. Among them, for Frame temporality detection vanishing trajectory threshold, for A new trajectory threshold has been added for frame temporal detection.

[0038] Preferred, , = =12.

[0039] Implementation Method 4 This embodiment is a further illustrative example of S3 in the high trajectory consistency multi-target tracking method based on position awareness and multi-step memory matching described in Embodiment 1.

[0040] S3 includes: For the abnormal disappearance trajectory marked as to be recovered The system extracts multi-frame features from its historical frames and stores them in long-term memory. First, it extracts the global appearance features of the detection boxes using a pre-trained CNN network (such as ResNet). The target detection box is input into a pre-trained ResNet50 network, and the output of its penultimate layer is extracted, resulting in a 512-dimensional object. Simultaneously, key points of the target are extracted using a human pose estimation network (such as HRNet), thereby extracting local features, including head features. Trunk features etc., which is represented as: , (8) in, For local appearance features, Indicates the location The weight of the position, Indicates the location The original feature response value at that location.

[0041] Then, a self-attention mechanism is used to fuse the global appearance features and local features to obtain the fused features, represented as: (9) Finally, the fusion features The trajectory's ID, occurrence frame number, and spatial location information are associated and stored in long-term memory. .

[0042] In this embodiment, the global feature is 0.4, the head feature is 0.2, the torso feature is 0.2, and the other features are 0.2.

[0043] Implementation Method 5 This embodiment is a further illustrative example of S3 in the high trajectory consistency multi-target tracking and monitoring method described in Embodiment 1.

[0044] S3 in this embodiment also includes a memory management strategy for the long-term memory bank, the memory management strategy specifically including: Effective feature memory strategy: For each trajectory, at intervals The frame rules extract and fuse features and store them, controlling system storage and computational complexity while ensuring feature validity; Periodic cleanup strategy: Perform cleanup operations on the long-term memory at preset intervals, deleting items that exceed the time threshold. Unmatched trajectory features, feature similarity below the threshold The entry is considered obsolete; if the number of entries retained in the memory exceeds the maximum capacity, the oldest entry is deleted.

[0045] The memory management strategy ensures system memory efficiency and feature validity by periodically cleaning the memory database. Cleanup parameters can be set as follows: Frame (approximately 12 seconds) Memory cleanup is performed every 100 frames. First, the last frame number of each memory entry is checked. If the current frame number minus the last memory frame number is greater than... If the entry is not found to be similar to the newly appearing feature, then delete the entry; secondly, calculate the average similarity between the entry's features and the newly appearing features. If the similarity is lower than the average similarity between the newly appearing features and the newly appearing features, then delete the entry. If an item is deemed outdated, it is marked as obsolete and deleted during the next cleanup. Finally, the number of retained entries is counted, and if it exceeds the maximum capacity N_max, the oldest entry is deleted. Experiments show that, on average, about 30% of obsolete features are deleted per cleanup cycle, while 70% of valid features are retained. This ensures the ability to re-associate across frames while avoiding waste of computational resources.

[0046] Implementation Method Six This embodiment is a further illustrative example of S4 in the high trajectory consistency multi-target tracking and monitoring method described in Embodiment 1.

[0047] In S4, when a new trajectory is detected At that time, the multi-step cross-frame matching module is triggered to attempt to match the abnormal trajectory in the memory bank; the flowchart is shown below. Figure 4 As shown. The specific process is as follows: When a new trajectory is detected If the location-aware module verifies that the trajectory is an abnormally added trajectory, then the multi-step cross-frame matching module is triggered to attempt to match it with the abnormally disappeared trajectory in the memory. First, calculate the new trajectory. Features With all memories in the memory bank The feature distance is used to select the top-K candidate features with the lowest feature distance. Then, for each candidate feature... Calculate the frequency and average feature distance of a given ID: , (10) in, For frequency matching value, That is, the number of candidate features. That is, the candidate features all belong to the trajectory The number of elements is calculated to obtain the trajectory. With new trajectory The matching frequency, For trajectory With new trajectory The average feature distance of the match.

[0048] Finally, a dual-threshold matching strategy is adopted. A match is considered successful if both of the following conditions are met: Condition 1: Frequency matching. The stored trajectory is most recently found alongside the new trajectory. In each valid frame (all time-series frames that appear, at intervals of 10 ... (The frame is taken as a valid frame) The matching frequency is greater than a threshold. In this embodiment, the preferred value of the matching frequency threshold is: .

[0049] Condition 2: Feature distance matching. Average feature distance. Less than the matching feature distance threshold In this embodiment, the preferred value is... (After normalization). When the new trajectory When the stored trajectories matched by each valid frame are different, the trajectory with the smallest average feature distance is selected.

[0050] For successfully matched candidate trajectories Assign its trajectory ID to the new trajectory The entry is then deleted from the memory bank. A match is considered successful only when both frequency matching and feature distance matching are satisfied; this dual-threshold strategy significantly reduces the false match rate. This strategy also employs effective feature memory techniques and newly emerging trajectories. Using a long-term matching strategy with a few valid frames can significantly enhance the effectiveness of the matching.

[0051] Implementation Method Seven This embodiment is a further illustrative example of S5 in the high trajectory consistency multi-target tracking and monitoring method described in Embodiment 1.

[0052] In step S5, updating the state of all trajectories based on the matching results of the multi-step cross-frame matching module specifically includes: For a new trajectory that is successfully matched, update its status to active, assign a corresponding abnormal disappearance trajectory ID, and update its appearance features and spatial location information. For abnormal disappearance trajectories in the pending recovery state, if the current frame still does not match, the pending recovery state is maintained, and the last missing frame number is updated. For normally terminated trajectories, their records are deleted from the existing trajectory set. If the feature entries of the trajectory exist in the long-term memory, they are deleted simultaneously. The system counts the number of active trajectories, trajectories awaiting recovery, and newly added trajectories in the current frame for system status monitoring and performance evaluation.

[0053] Specifically, trajectory state updates are implemented through the following strategy: First, for new trajectories that have successfully matched... First, update its status from "new" to "active", assign the original trajectory ID to it, and update its appearance features and spatial location information; second, for trajectories to be recovered, if they are still not matched in the current frame, maintain their "to be recovered" status and update their last missing frame number; third, for trajectories determined to have terminated normally, retrieve them from the trajectory set. Delete the record and remove its feature entry from the memory (if it exists); finally, count the number of active trajectories, the number of trajectories to be recovered, and the number of newly added trajectories in the current frame for system status monitoring and performance evaluation.

[0054] Implementation Method Seven Application in scene monitoring and inspection of shopping mall outdoor plazas This embodiment is a verification description of a multi-target tracking and monitoring method with high trajectory consistency, as described in any one of embodiments one to six.

[0055] This embodiment uses a shopping mall outdoor plaza as an example to explain in detail the implementation process of the present invention.

[0056] BoT-SORT was used as the baseline tracker, and YOLOv10x was used as the detector with a detection threshold of 0.6. The video frame rate was 25fps, and the resolution was 1920×1080. Position-aware parameters were set as follows: = =0.06, = =12, , .

[0057] In the video, a child target (trajectory ID: 16) appears in the center of the frame at t=56, but is obscured by a crowd at t=70, and is not detected for 12 consecutive frames. At this point, the position awareness verification module is triggered. It checks that the last appearance position of trajectory ID: 16 is (960, 540), which is in the center of the frame and does not meet the boundary judgment condition; the number of consecutive missing frames is counted as 12, reaching the threshold. Threshold; if the result is abnormal disappearance, mark trajectory ID: 16 as to be recovered, extract its multiple effective fusion features and store them in the memory bank.

[0058] At frame t=95, the child target reappears in the frame and is detected as a new trajectory (ID: 30). At this time, the multi-step cross-frame matching module is triggered, the memory is searched, and 10 candidate features are matched, of which 8 features are the features of trajectory ID: 16; the feature similarity is calculated to be 0.65 < 0.7; the matching frequency is calculated to be 0.8 ≥ 0.8; the double threshold check is passed, the match is successful, the ID of trajectory ID: 30 is updated to 16, and the entry of trajectory ID: 16 in the memory is deleted.

[0059] Experimental results show that in the shopping mall entrance scenario, the baseline method generates 8 identity switching events, while this method generates only 1, and the IDF1 index increases from 68.5% to 76.8%.

[0060] Implementation Method 8: Performance Comparison on Different Baseline Trackers This implementation demonstrates the performance improvement of the PMMM module on different baseline trackers. The dataset used is the MOT17 validation set and the Emporium test set. The baseline trackers include BoT-SORT, StrongSORT, BoostTrack, and ByteTrack. The evaluation metrics include HOTA, MOTA, IDF1, IDSW, IDR, and IDP. The evaluation results are shown in Tables 1 and 2.

[0061] Table 1 Implementation results on the MOT17 dataset

[0062] Table 2 Implementation results on the Emporium dataset

[0063] Experimental analysis shows that the performance improvement of the method described in this invention is more significant in Emporium-intensive scenarios, particularly in terms of IDF1 metric and the number of identity switching attempts. This demonstrates the effectiveness of this invention in handling long-term occlusion and dense interactions.

[0064] Implementation Method Nine: Comparison of PMMM Module with Existing Method DAAW (Dynamic Appearance and Adaptive Weights) This embodiment compares the performance of the present invention with that of the existing method DAAW. OCSort is used as the baseline, and the comparison methods include DeepOCSort (integrating DA+AW modules) and the present invention (integrating PMMM modules). Experimental results are shown in Table 3.

[0065] Table 3 Comparison of PMMM Module and DeepOCSort Enhancement Module

[0066] Comparative analysis shows that DeepOCSort exhibits a decline in HOTA and IDP metrics, while the method described in this invention shows significant improvements in all ID-related metrics. The IDF1 improvement of the method described in this invention (+4.383%) is far greater than that of DeepOCSort (+0.817%), and the identity switching reduction (-4) is superior to that of DeepOCSort (+3). This demonstrates that the long-term memory mechanism and location-aware strategy of this method are more effective than the DA+AW module of DeepOCSort.

[0067] Implementation Method 10: Parameter Sensitivity Analysis This implementation analyzes the impact of key parameters on performance. These parameters include the boundary discrimination threshold (…). = ∈[0.02,0.10]), temporal verification threshold ( = ∈[8, 16]), matching threshold ( ∈[0.4,0.8], (∈[0.3,0.7]). The implementation results are as follows: Figure 5 , Figure 6 As shown.

[0068] Experimental results on the boundary discrimination threshold show that when = The performance is optimal when the threshold is 0.06. If the threshold is too small, normal boundary deviations will be misjudged as abnormal disappearances; if the threshold is too large, genuine abnormal disappearances will be missed.

[0069] Experimental results for the timing verification threshold show that when = Performance is optimal when the threshold is 12. A threshold that is too small may lead to false positives that temporarily miss anomalies, while a threshold that is too large may cause the true anomalies to be missed.

[0070] Implementation Method Eleven: Computational Efficiency Analysis This embodiment analyzes the computational overhead of the method described in this invention. The experimental environment consisted of an Intel i7-10700K CPU, an NVIDIA RTX 3080 GPU, and software including Python 3.8, PyTorch 1.10, and CUDA 11.3.

[0071] Experimental results show that the baseline method BoT-SORT has an FPS of 18.5, memory usage of 1250MB, and inference time of 54.05 milliseconds / frame. After integrating the PMMM module, the FPS decreased to 6.2, memory usage increased to 1520MB, and inference time decreased to 161.29 milliseconds / frame. The computational overhead is: FPS decreased by 12.3, memory usage increased by 270MB, and inference time increased by 107.24 milliseconds.

[0072] Analysis shows that the FPS reduction of this invention by approximately 2-3 times is mainly due to the increased computational load from long-term memory retrieval and matching. Memory usage increases by approximately 22% to store features in the memory bank. Although computational overhead has increased, 6.2 FPS still meets real-time requirements in practical applications (such as shopping mall monitoring) (25fps video uses frame skipping). Computational overhead can be reduced by optimizing the memory bank retrieval algorithm (e.g., using an approximate nearest neighbor search ANN).

[0073] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method of multi-target tracking and surveillance with high trajectory consistency, characterized in that, Includes the following steps: S1. Use the detection-tracking paradigm to detect targets in each frame of the video sequence to obtain a set of detection boxes. Then, associate the set of detection boxes with the existing trajectory set to obtain preliminary tracking results. S2. Determine whether the trajectory has not matched any detection box in K consecutive frames. If so, trigger the position-aware trajectory verification module and determine whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory. S3. For trajectories that are determined to have disappeared abnormally, extract multi-frame features from their historical frames, store the multi-frame features in a long-term memory bank, and use a self-attention mechanism to fuse global appearance features and local body part features. S4. When a new trajectory is detected, the validity of the new trajectory is first verified by the location-aware trajectory verification module. If it is determined to be an invalid new trajectory, the multi-step cross-frame matching module is triggered. The invalid new trajectory is matched with the abnormally disappeared trajectory in the long-term memory through the multi-step cross-frame matching module. The double threshold matching strategy is used to realize the re-identification of the abnormally disappeared trajectory. S5. Update the status of all trajectories according to the matching results of the multi-step cross-frame matching module, update the successfully matched abnormal disappearance trajectories to the active state, keep the unmatched abnormal disappearance trajectories in the pending recovery state, and delete the relevant records of normally terminated trajectories.

2. The multi-target tracking and surveillance method of claim 1, wherein, S1 includes: An initial trajectory sequence is obtained, and a target detector is used to detect video frames to obtain detection boxes. Motion prediction and appearance feature extraction operations are performed on the detection boxes in sequence. Then, the detection boxes with extracted features are associated with the previous target boxes to obtain preliminary tracking results. The target detector is any one of the YOLO, Fast-RCNN, or DETR algorithm models; the motion prediction is implemented using the Kalman filter state estimation method, and the appearance feature extraction is implemented using the lightweight network model OsNet; the association between the detection box and the preceding target box is determined by weighted combination of the Mahalanobis distance of the motion branch and the cosine distance of the appearance branch. When the weighted combination result is lower than a preset threshold, the target is determined to be successfully matched with the trajectory and the trajectory box is updated.

3. The multi-target tracking and monitoring method with high trajectory consistency according to claim 1, characterized in that, In step S2, the location-aware trajectory verification module determines whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory, specifically including: When the trajectory is continuous When no detection box is matched in the frame, the positional relationship between the center point coordinates of the last detection box that appears on the trajectory and the image boundary is calculated. If the preset boundary discrimination condition is met, the trajectory is determined to be a normal boundary termination trajectory. If the boundary discrimination condition is not met, then the trajectory is... Frame timing consistency verification, if the trajectory is continuous If no detection box is matched in the frame, it is determined to be an abnormally disappearing trajectory; if the trajectory is continuous The continuous detection of targets in the frame indicates that the trajectory has been abnormally newly added. in, and The value of is 12.

4. The multi-target tracking and monitoring method with high trajectory consistency according to claim 1, characterized in that, S3 includes: For the abnormal disappearance trajectory marked as to be recovered, multi-frame features from its historical frames are extracted. The global appearance features of the detection box are extracted through a pre-trained CNN network, and the local features of the target are extracted through a human pose estimation network. The global appearance features and local features are fused using a self-attention mechanism to obtain fused features; The fused features are associated with the trajectory's ID, occurrence frame number, and spatial location information and stored in a long-term memory.

5. The multi-target tracking and monitoring method with high trajectory consistency according to claim 1, characterized in that, The S3 further includes a memory management strategy for the long-term memory bank, the memory management strategy specifically including: Effective feature memory strategy: Extract and store the fused features for each trajectory according to the rule of interval T frames; Periodic cleaning strategy: Perform cleaning operations on the long-term memory at preset cycles, deleting trajectory features that have not matched beyond the time threshold and outdated features with feature similarity below the threshold; if the number of entries retained in the memory exceeds the maximum capacity, delete the oldest entry.

6. The multi-target tracking and monitoring method with high trajectory consistency according to claim 1, characterized in that, In step S4, the re-identification of the abnormal disappearance trajectory using a dual threshold matching strategy specifically includes: The invalid new trajectory is input into the multi-step cross-frame matching module. The feature distance between the features of the new trajectory and the features of all trajectory features in the long-term memory is calculated, and the Top-K candidate features with the smallest feature distance are selected. Calculate the matching frequency and average feature distance of the trajectory ID to which each candidate feature belongs; A dual-threshold matching strategy is used to determine the matching result. When both the frequency matching condition and the feature distance matching condition are met, the new trajectory is determined to be a successful match with the corresponding anomaly disappearance trajectory. If the valid frame of the new trajectory matches multiple stored trajectories, the trajectory with the smallest average feature distance is selected as the matching result. After a successful match, the ID of the abnormal disappearance trajectory is assigned to the new trajectory, and the relevant entry for the abnormal disappearance trajectory is deleted from the long-term memory.

7. The multi-target tracking and monitoring method with high trajectory consistency according to claim 1, characterized in that, In step S5, updating the state of all trajectories based on the matching results of the multi-step cross-frame matching module specifically includes: For a new trajectory that is successfully matched, update its status to active, assign a corresponding abnormal disappearance trajectory ID, and update its appearance features and spatial location information. For abnormal disappearance trajectories in the pending recovery state, if the current frame still does not match, the pending recovery state is maintained, and the last missing frame number is updated. For normally terminated trajectories, their records are deleted from the existing trajectory set. If the feature entries of the trajectory exist in the long-term memory, they are deleted simultaneously. The system counts the number of active trajectories, trajectories awaiting recovery, and newly added trajectories in the current frame for system status monitoring and performance evaluation.

8. A multi-target tracking and monitoring system with high trajectory consistency, characterized in that, The system is implemented based on a high trajectory consistency multi-target tracking and monitoring method as described in any one of claims 1-7, and the system includes: The detection and association module is used to detect targets in each frame of the video sequence using the detection-tracking paradigm, obtain a set of detection boxes, and associate the set of detection boxes with the existing trajectory set to obtain preliminary tracking results. The trajectory verification module is used to determine whether the trajectory has not matched any detection box in K consecutive frames. If so, the position-aware trajectory verification module is triggered to determine whether the trajectory is a normally terminated trajectory or an abnormally disappeared trajectory. Feature fusion and storage module: For trajectories that are determined to have disappeared abnormally, extract multi-frame features from their historical frames, store the multi-frame features in a long-term memory bank, and use a self-attention mechanism to fuse global appearance features and local body part features. Cross-frame matching module: When a new trajectory is detected, the validity of the new trajectory is first verified by the position-aware trajectory verification module. If the new trajectory is determined to be invalid, the multi-step cross-frame matching module is triggered. The multi-step cross-frame matching module matches the invalid new trajectory with the abnormally disappeared trajectory in the long-term memory. A dual threshold matching strategy is used to realize the re-identification of the abnormally disappeared trajectory. The trajectory status update module is used to update the status of all trajectories based on the matching results of the multi-step cross-frame matching module. It updates the successfully matched abnormally disappeared trajectories to the active state, keeps the unmatched abnormally disappeared trajectories in the pending recovery state, and deletes the relevant records of normally terminated trajectories.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes a multi-target tracking and monitoring method with high trajectory consistency according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that executes a multi-target tracking and monitoring method with high trajectory consistency according to any one of claims 1-7.

Images