A dense multi-target tracking method, system, device, and storage medium
By combining macroscopic motion direction consistency and microscopic pixel motion consistency, the problems of trajectory breakage and identity loss in scenarios with dense occlusion and overlapping targets are solved, enabling real-time multi-target tracking on edge computing devices.
Patent Information
- Application Number
- CN202611158613.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multi-target tracking methods suffer from issues such as trajectory breakage, identity loss, and ambiguous associations in scenarios involving dense crowds, occlusion, and overlapping targets. In particular, they are limited by the computing power of edge computing devices, making it difficult to meet the requirements for real-time deployment.
By combining the macroscopic motion direction consistency and microscopic pixel motion consistency of the target, a direction-aware cross-union cost matrix and a second-stage association mechanism are constructed through Kalman filtering and local pixel motion observation methods to improve trajectory continuity and identity preservation capabilities.
Without relying on deep appearance re-identification networks, it improves the association accuracy and trajectory continuity in scenarios with dense occlusion and overlapping targets, meeting the real-time deployment requirements of edge computing devices.
Smart Images

Figure CN123118690A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a dense multi-target tracking method, system, device, and storage medium. Background Technology
[0002] Multi-object tracking is a crucial task in computer vision, aiming to continuously locate multiple targets of interest in a video sequence while maintaining the consistency of their identification. Existing multi-object tracking methods typically employ a "detection-tracking" paradigm, where an object detector first acquires the target bounding boxes in each frame, and then a data association algorithm connects the detection results from different frames to form the target trajectory.
[0003] However, existing multi-target tracking methods still have certain limitations in complex scenarios such as dense crowds, severe occlusion, and frequent target overlap.
[0004] On the one hand, some high-precision multi-target tracking methods rely on re-identification networks to extract target appearance features to alleviate the identity switching problem caused by occlusion or target overlap. However, such methods usually require the introduction of a deep appearance feature extraction network in addition to the target detection model, leading to increased system computational overhead, memory usage, and inference latency. When there are a large number of targets in the video frame at the same time, the appearance feature extraction process will further reduce the overall processing frame rate, making it difficult to meet the real-time deployment requirements of computing-constrained devices such as drones, vehicle computing platforms, and edge security cameras.
[0005] On the other hand, existing lightweight multi-target tracking methods typically rely on Kalman filtering to predict target motion states. When the target moves at approximately a constant velocity and the observation is stable, Kalman filtering can provide good short-term prediction results; however, when the target is occluded, undergoes nonlinear motion, makes sharp turns, or the video acquisition equipment experiences jitter, the prediction results of the linear motion model are prone to position drift. If the predicted bounding box deviates significantly from the target's true position, it becomes difficult to match the trajectory with the correct detection result in the subsequent data association stage, leading to trajectory breakage or identity loss.
[0006] Furthermore, to improve inference speed, some tracking methods rely primarily on the Intersection over Union (IoU) ratio between detected bounding boxes for association matching. While these methods are computationally simple and efficient, in scenarios with densely packed targets, the detected bounding boxes of different targets may highly overlap on the 2D image plane, resulting in multiple candidate detection results having similar IoU values with the same trajectory. In this case, simply relying on spatial overlap makes it difficult to distinguish the true direction of target motion, easily leading to association ambiguity, which in turn causes identity switching and trajectory confusion, affecting the continuity and accuracy of multi-target tracking results.
[0007] Therefore, a more reliable solution is needed that can fully utilize the spatiotemporal motion consistency information of targets without relying on a deep appearance re-identification network in the multi-target tracking process, improve the trajectory continuity and identity preservation capabilities in dense occlusion and target interleaving scenarios, and meet the real-time deployment requirements of edge computing devices. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a dense multi-target tracking method, system, device and storage medium that can combine the consistency of the macroscopic motion direction of the target with the consistency of the microscopic pixel motion. Without relying on the depth appearance re-identification network, it can improve the correlation accuracy and trajectory continuity in dense occlusion and target interleaving scenarios, thereby improving the accuracy and reliability of multi-target tracking.
[0009] To achieve the above objectives, the present invention employs the following technical solution:
[0010] On one hand, the present invention provides a dense multi-target tracking method, the method comprising:
[0011] The acquired current frame image is input into the trained multi-object detection model to generate a set of detection bounding boxes corresponding to the current frame image and its corresponding confidence score;
[0012] Based on a preset confidence threshold, the set of detection bounding boxes is divided into a set of high-confidence detection boxes and a set of low-confidence detection boxes.
[0013] The predicted bounding box of each target trajectory in the current frame is determined by the mean of the predicted state of each target trajectory in the current frame, which is obtained by prior prediction of each target trajectory maintained in the previous frame using Kalman filtering.
[0014] Based on each predicted bounding box and the set of high-confidence detection boxes, a direction-aware intersection-union cost matrix is constructed, and a first-stage association is performed on each target trajectory and the set of high-confidence detection boxes to obtain the first-stage association result; the direction-aware intersection-union cost matrix integrates spatial overlap with the consistency of historical motion direction;
[0015] For the set of unmatched tracked trajectories in the first stage of the association results, the overall translation amount between the previous frame and the current frame is estimated based on the local pixel motion observation method.
[0016] The mean of the corresponding predicted state is corrected and updated using the overall translation of the set of unmatched tracked trajectories in the first stage, and the bounding boxes of each unmatched correction prediction are determined.
[0017] Based on the unmatched correction prediction bounding boxes and the low-confidence detection box set, a second-stage association cost matrix is constructed, and the first-stage unmatched tracked trajectory set and the low-confidence detection box set are associated in the second stage to obtain the second-stage association result.
[0018] Update the trajectory status of each target based on the first-stage association results and the second-stage association results, and output the multi-target tracking results.
[0019] In some possible implementations, the target trajectory maintained in the previous frame includes the tracked target trajectory; the construction of the direction-aware intersection-union cost matrix based on each predicted bounding box and the set of high-confidence detection boxes includes:
[0020] Extract the sequence of detection center points that are successfully matched in k consecutive frames of images for any tracked target trajectory; the k consecutive frames are k consecutive frames forward from the current frame.
[0021] By determining the mean displacement difference of the detection center point sequence, the historical motion velocity vector of any tracked target trajectory is obtained;
[0022] Determine the instantaneous displacement vector of any high-confidence detection box relative to the most recent frame matching position of any tracked target trajectory, wherein the most recent frame matching position is the detection bounding box that was most recently successfully matched before the current frame;
[0023] Determine the directional similarity between the historical motion velocity vector and the instantaneous displacement vector, and truncate the directional similarity to the [0,1] interval;
[0024] If a preset trigger threshold is met, the spatial intersection-union ratio between the predicted bounding box corresponding to any tracked target trajectory and any high-confidence detection box is corrected according to the directional similarity to obtain the directional-aware intersection-union ratio.
[0025] Based on the direction-aware cross-union ratio between any tracked target trajectory and any high-confidence detection box, a direction-aware cross-union cost matrix is constructed.
[0026] In some possible implementations, the determination of the directional similarity between the historical motion velocity vector and the instantaneous displacement vector, and the directional similarity being truncated to the [0,1] interval, is shown in the following formula:
[0027] ,and ;
[0028] in, Indicates directional similarity. Represents the historical velocity vector. Represents the instantaneous displacement vector. Used to avoid a denominator of zero;
[0029] When a preset trigger threshold is met, the spatial intersection-union ratio between any tracked target trajectory and any high-confidence detection box is adjusted based on the directional similarity to obtain the directional-aware intersection-union ratio, and the directional-aware intersection-union ratio is restricted to the [0,1] interval, as shown in the following formula:
[0030] ,and ;
[0031] in, Indicates the intersection-union ratio of direction perception. This indicates motion direction perception based on historical velocity direction. This represents the spatial intersection-union ratio (IoU) between the predicted bounding boxes of the tracked target trajectory and the high-confidence detection boxes. This represents the directional consistency weight.
[0032] In some possible implementations, estimating the overall translation amount between the previous frame and the current frame for the set of unmatched tracked trajectories in the first stage association result based on a local pixel motion observation method includes:
[0033] Corner features are extracted from the local region obtained by cropping the bounding box of the previous frame that does not match the tracked trajectory in each first stage;
[0034] The Lucas-Kanade local optical flow method with an image pyramid structure is used to track the corner features of each unmatched tracked trajectory in the first stage between the previous frame and the current frame, and the predicted center displacement is used as the initial flow guess of the Lucas-Kanade local optical flow method; the predicted center displacement is the displacement between the center point of the predicted bounding box of each unmatched tracked trajectory in the first stage and the center point of the detected bounding box in the previous frame or the most recently observed bounding box.
[0035] Based on the initial flow guess, the set of pixel-level displacement vectors of all successfully tracked corner point features that did not match the tracked trajectory in each first stage is obtained by optical flow tracing;
[0036] Anomaly filtering is performed on the pixel-level displacement vector set according to a preset displacement threshold to obtain the effective corner point displacement set of each first-stage unmatched tracked trajectory and the effective corner point features corresponding to each effective corner point displacement vector.
[0037] When the number of effective corner features is greater than or equal to a preset effective point number threshold, the median of the effective corner displacement set of each unmatched tracked trajectory in the first stage is determined as the overall translation amount of each unmatched tracked trajectory in the first stage between two adjacent frames.
[0038] In some possible implementations, the step of using the overall translation of the set of unmatched tracked trajectories in the first stage to correct and update the mean of the corresponding predicted state, and determining the bounding boxes of each unmatched correction prediction, includes:
[0039] The mean predicted state of each unmatched tracked trajectory in the first stage is corrected and updated using the overall translation amount of each unmatched tracked trajectory in the first stage, resulting in the mean corrected predicted state of each unmatched trajectory, as shown in the following formula:
[0040]
[0041] in, This represents the mean of the unmatched corrective prediction state. This represents the predicted mean state. This indicates the total translation amount of the unmatched tracked trajectory in the first stage between two adjacent frames;
[0042] The mean value of each unmatched correction prediction state is decoded to obtain the bounding box of each unmatched correction prediction state.
[0043] In some possible implementations, the first-stage association result includes a set of matched trajectories and a set of unmatched and lost trajectories from the first stage; the second-stage association result includes a set of matched trajectories from the second stage and a set of unmatched trajectories from the second stage; the step of updating the status of each target trajectory based on the first-stage association result and the second-stage association result, and outputting multi-target tracking results, includes:
[0044] Each matched trajectory in the first stage is marked as active, and its predicted state mean is updated using its matched high-confidence detection box to obtain the first updated state mean.
[0045] Each matched trajectory in the second stage is marked as an active state, and its predicted state mean is updated using the low-confidence detection boxes it matches, to obtain the second updated state mean.
[0046] Mark each unmatched lost trajectory in the first stage and each unmatched trajectory in the second stage as a lost state, and accumulate the number of consecutive lost frames;
[0047] When the number of consecutive lost frames of a target trajectory in a lost state is greater than or equal to a preset buffer threshold, the target trajectory in a lost state is deleted.
[0048] Based on the newly created detection bounding box, the initial state mean is determined, and a new target trajectory is generated. The newly created detection bounding box is a detection bounding box that has not matched any target trajectory after the first stage association and whose confidence score is greater than or equal to the preset new creation threshold.
[0049] Output the multi-target tracking results based on the matched trajectories that are active in the current frame.
[0050] In some possible implementations, dividing the set of detection bounding boxes into a set of high-confidence detection boxes and a set of low-confidence detection boxes according to a preset confidence threshold includes:
[0051] Obtain the preset confidence threshold and the confidence score corresponding to the detection bounding box set; the preset confidence threshold includes a preset high threshold and a preset low threshold;
[0052] Detection bounding boxes with confidence scores greater than or equal to the preset high threshold are included in the high-confidence detection box set;
[0053] Detection bounding boxes with confidence scores greater than or equal to the preset low threshold and less than the preset high threshold are included in the low confidence detection box set;
[0054] Detection bounding boxes with confidence scores lower than the preset low threshold are filtered out.
[0055] On the other hand, a dense multi-target tracking system is provided, the system comprising:
[0056] The object detection module is used to input the acquired current frame image into the trained multi-object detection model to generate a set of detection bounding boxes corresponding to the current frame image and their corresponding confidence scores;
[0057] The confidence leveling module is used to divide the set of detection bounding boxes into a set of high-confidence detection boxes and a set of low-confidence detection boxes according to a preset confidence threshold.
[0058] The motion state prediction module is used to determine the predicted bounding box of each target trajectory in the current frame by using the average predicted state of each target trajectory in the current frame based on the prior prediction of each target trajectory maintained in the previous frame by Kalman filtering.
[0059] The direction-aware primary association module is used to construct a direction-aware intersection-union cost matrix based on each predicted bounding box and the set of high-confidence detection boxes, and to perform a first-stage association between each target trajectory and the set of high-confidence detection boxes to obtain the first-stage association result; the direction-aware intersection-union cost matrix integrates spatial overlap with the consistency of historical motion direction;
[0060] The micro-pixel motion correction module is used to estimate the overall translation amount between the previous frame and the current frame for the set of unmatched tracked trajectories in the first stage of the association result of the first stage based on the local pixel motion observation method; and to correct and update the mean of the corresponding prediction state by using the overall translation amount of the set of unmatched tracked trajectories in the first stage, and to determine the bounding box of each unmatched correction prediction.
[0061] The second-stage association module is used to construct a second-stage association cost matrix based on the unmatched correction prediction bounding boxes and the low-confidence detection box set, and to perform second-stage association on the first-stage unmatched tracked trajectory set and the low-confidence detection box set to obtain the second-stage association result.
[0062] The trajectory update and tracking result determination module is used to update the trajectory status of each target based on the first stage association result and the second stage association result, and output the multi-target tracking result.
[0063] On the other hand, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the dense multi-target tracking method as described above.
[0064] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction and at least one program are stored therein, the at least one instruction and the at least one program being loaded and executed by a processor to implement the dense multi-target tracking method as described above.
[0065] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0066] In this invention, the current frame image is input into a trained multi-object detection model to generate a set of detection bounding boxes corresponding to the current frame image and their corresponding confidence scores; according to a preset confidence threshold, the set of detection bounding boxes is divided into a set of high-confidence detection boxes and a set of low-confidence detection boxes.
[0067] The predicted bounding box of each target trajectory in the current frame is determined by the mean of the predicted state of each target trajectory in the current frame, which is obtained by prior prediction of each target trajectory maintained in the previous frame using Kalman filtering.
[0068] Next, based on each predicted bounding box and the set of high-confidence detection boxes, a direction-aware cross-union cost matrix is constructed, and the first-stage association is performed on each target trajectory and the set of high-confidence detection boxes to obtain the first-stage association results. By constructing a direction-aware cross-union cost matrix that integrates spatial overlap and historical motion direction consistency, high-confidence detection boxes with consistent motion directions can have higher association priority, and the matching probability of high-confidence detection boxes with spatial overlap but inconsistent motion directions can be reduced. This can alleviate the direction confusion problem caused by relying solely on spatial cross-union when dense targets are moving towards each other, crossing each other, or overlapping for a short time, reduce identity switching and trajectory confusion, thereby reducing association ambiguity in dense and intersecting scenarios and improving identity preservation capabilities.
[0069] Next, for the set of unmatched tracked trajectories in the first stage of the association results, the overall translation amount between the previous frame and the current frame is estimated based on the local pixel motion observation method. The overall translation amount of the set of unmatched tracked trajectories in the first stage is used to correct and update the mean of the corresponding prediction state to determine each unmatched corrected prediction bounding box. Based on each unmatched corrected prediction bounding box and the set of low-confidence detection boxes, the second stage association cost matrix is constructed, and the set of unmatched tracked trajectories in the first stage and the set of low-confidence detection boxes are associated in the second stage to obtain the second stage association results. The overall offset is used to correct and update the prediction bounding boxes, and the corrected prediction bounding boxes are used to perform the second stage remedial association with the low-confidence detection boxes. This can pull the drifted prediction position back to a position that is more consistent with the local real motion of the target, and improve the effectiveness and reliability of the second stage association, thereby improving the trajectory continuation capability under occlusion or short-term loss states.
[0070] Next, the trajectory states of each target are updated based on the association results of the first and second stages, and the multi-target tracking results are output. This combines the macroscopic motion direction consistency constraint of the first stage with the microscopic pixel motion correction mechanism of the second stage. This reduces the probability of erroneous association when targets intersect, and compensates for and restores the predicted state when occlusion occurs or detection confidence decreases. Through this collaborative mechanism, the trajectory continuity, identity stability, and adaptability to complex scenes of dense multi-target tracking results can be improved. Furthermore, by introducing historical motion direction consistency and local pixel motion consistency information during the association process, without needing to configure an additional deep appearance re-identification network outside the target detection model, effective continuation and identity preservation of trajectories in densely occluded and intersecting target scenes can be achieved. This improves the effectiveness, accuracy, and convenience of determining multi-target tracking results, thereby reducing reliance on deep appearance re-identification networks and enhancing trajectory continuity, identity preservation capabilities, and real-time deployment adaptability in densely occluded and intersecting target scenes. Attached Figure Description
[0071] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0072] Figure 1 This is a flowchart illustrating a dense multi-target tracking method provided in an embodiment of the present invention;
[0073] Figure 2 This is a schematic diagram of a dense multi-target tracking process provided in an embodiment of the present invention;
[0074] Figure 3 This is a schematic diagram of the structure of a dense multi-target tracking system provided in an embodiment of the present invention. Detailed Implementation
[0075] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0076] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0077] In this embodiment of the invention, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0078] Various exemplary embodiments, features, and aspects of the present invention will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.
[0079] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0080] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0081] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without certain specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.
[0082] This invention provides a lightweight, dense multi-target tracking method that addresses the challenges of high computational cost and deployment complexity associated with relying on re-identification networks in scenarios with dense occlusion and overlapping targets. It also addresses the issues of trajectory fragmentation caused by the distorted predictions of traditional Kalman filtering under long-term occlusion or nonlinear motion, and identity switching caused by relying solely on spatial overlap when multiple candidate detection boxes have similar IoU values with the same trajectory. This method improves trajectory continuity and identity preservation in dense occlusion and overlapping target scenarios without additional reliance on depth-appearance-enriched identification networks, ensuring real-time inference while meeting the real-time deployment requirements of edge computing devices. It is applicable to computationally limited scenarios such as edge computing, intelligent security, and dense crowd monitoring. Optionally, the dense multi-target tracking system provided by this invention can be deployed on edge computing nodes, servers, or other electronic devices with image processing capabilities.
[0083] Figure 1 This is a flowchart illustrating a dense multi-target tracking method provided by an embodiment of the present invention. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive methods, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server product execution, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 1 As shown, the above method may include:
[0084] S101: Input the acquired current frame image into the trained multi-object detection model to generate the set of detection bounding boxes corresponding to the current frame image and its corresponding confidence score;
[0085] In one specific embodiment, the present invention can be applied to video analysis scenarios involving dense target occlusion and staggered motion, such as dense surveillance, edge security, vehicle-mounted sensing, or drone inspection. The object to be processed can be a continuous video frame sequence. Specifically, during standard dataset verification or engineering deployment, the video frame sequence can contain configuration files for recording information such as image resolution and frame rate.
[0086] In one specific embodiment, a continuous video frame sequence to be processed is obtained, and the current frame image is read in chronological order. Then, the current frame image is input into a pre-trained multi-object detection model, and the set of detection bounding boxes in the current frame image and the confidence score corresponding to each detection bounding box are output.
[0087] In one specific embodiment, the pre-trained multi-object detection model is a deep learning model used to continuously locate multiple targets of interest in a video frame sequence and identify their categories. These targets of interest include, but are not limited to, pedestrians, vehicles, or other objects whose identities need to be maintained continuously in the video frame sequence. Optionally, the pre-trained multi-object detection model is a lightweight model, and it can be a single-stage object detection model. Specifically, this single-stage object detection model can be used to output the set of detected bounding boxes and corresponding confidence scores for the current frame in a single forward inference process. Optionally, this single-stage object detection model can be an object detection model based on the YOLO series network architecture. Specifically, this object detection model based on the YOLO series network architecture can be used to perform bounding box detection and confidence estimation for targets in the current frame in dense target scenes. Optionally, the multi-object detection model can also employ other single-stage object detection models with equivalent detection capabilities.
[0088] In one specific embodiment, to improve the detection stability of the multi-object detection model in scenarios with occlusion, scale variations, and dense object distribution, it can be pre-trained on a dataset containing dense or occluded objects. The model parameters can then be further optimized on a target application scenario dataset through data augmentation and parameter fine-tuning, based on the actual application scenario. During the inference process of the multi-object detection model, if the input image is scaled to a fixed resolution, when the detection model outputs the tracking result, the normalized or scaled coordinates output by the detection model are restored to the true pixel coordinates based on the original image's width and height information. This reduces trajectory coordinate deviation caused by image scaling, thereby ensuring the consistency of coordinates in subsequent data association and trajectory output. Optionally, the tracking result output by the multi-object detection model can include frame number, object identification, top-left corner coordinates of the bounding box, bounding box width and height, and confidence level, etc. The output format can also be adjusted according to actual application requirements.
[0089] S102: Based on a preset confidence threshold, divide the set of detection bounding boxes into a set of high-confidence detection boxes and a set of low-confidence detection boxes;
[0090] In an optional embodiment, dividing the detection bounding box set into a high-confidence detection box set and a low-confidence detection box set according to a preset confidence threshold may include:
[0091] Obtain the confidence scores corresponding to the preset confidence threshold and the set of detected bounding boxes; the preset confidence threshold includes a preset high threshold and a preset low threshold;
[0092] Detection bounding boxes with confidence scores greater than or equal to a preset high threshold are included in the high-confidence detection box set;
[0093] Detection bounding boxes with confidence scores greater than or equal to a preset low threshold and less than a preset high threshold are included in the low-confidence detection box set.
[0094] Filter out detection bounding boxes with confidence scores lower than a preset low threshold.
[0095] In a specific embodiment, to address the significant differences in detection result quality in densely occluded scenes, the aforementioned set of detection bounding boxes is graded by confidence level based on preset high and low thresholds, resulting in a high-confidence detection box set and a low-confidence detection box set. The preset high and low thresholds can be set according to actual application requirements. Optionally, the high-confidence detection box set provides reliable observation results for the first-stage association; specifically, it corresponds to observation boxes with clearer target appearances and more reliable detection results, used for subsequent first-stage association. The low-confidence detection box set is used for second-stage remedial association of occluded or blurred targets; specifically, it corresponds to targets that are partially occluded, blurred, or have weak detection responses, used for subsequent second-stage association to remediate and continue unmatched trajectories from the first stage.
[0096] S103: Based on the Kalman filter, the mean of the predicted state of each target trajectory in the current frame is obtained by predicting the prior of each target trajectory maintained in the previous frame, and the predicted bounding box of each target trajectory in the current frame is determined.
[0097] In one specific embodiment, prior predictions are performed on the target trajectories maintained in the previous frame based on Kalman filtering to obtain the predicted state mean and prediction covariance of each target trajectory in the current frame. Then, the predicted state mean is decoded to obtain the predicted bounding box of each target trajectory in the current frame. Optionally, the maintained target trajectories can refer to target trajectory records that have been established and saved in the trajectory management list before the current frame is processed, and have not yet been deleted; optionally, the target trajectories maintained in the previous frame can include tracked target trajectories and target trajectories in a lost state, wherein the tracked target trajectories can be active trajectories in a tracked state, and the target trajectories in a lost state can be trajectories in a lost state but have not exceeded a preset buffer threshold and can still be retrieved by subsequent detection boxes; that is, the maintained target trajectories can be divided into tracked target trajectories and target trajectories in a lost state. Optionally, the predicted state mean is used to characterize the predicted center position, scale, and motion state of the target trajectory in the current frame; the predicted covariance is used to characterize the uncertainty of the prediction result; the predicted bounding box can serve as a spatial reference for subsequent first-stage association and second-stage correction association. Specifically, the predicted bounding box can serve as a spatial reference for subsequent orientation-aware association and micro-pixel motion correction.
[0098] In a specific embodiment, an eight-dimensional constant velocity model can be used to construct the state vector of the target trajectory, as shown in the following equation:
[0099] ;
[0100] in, and The center coordinates of the detection bounding box representing the target trajectory coordinates and coordinate, The aspect ratio of the detection bounding box representing the target trajectory. The height of the detection bounding box representing the target trajectory. These represent the rate of change of the velocity of the corresponding state components.
[0101] Specifically, the state vector can be recursively derived using the state transition matrix of the Kalman filter to obtain the mean of the prior predicted state for the current frame. and prior prediction covariance Among them, the prior predicted state mean is used to estimate the target's position and scale in the current frame, and the prior predicted covariance is used to characterize the uncertainty of the prediction result.
[0102] S104: Based on each predicted bounding box and the set of high-confidence detection boxes, construct the direction-aware intersection-union cost matrix, and perform the first-stage association between each target trajectory and the set of high-confidence detection boxes to obtain the first-stage association result;
[0103] In a specific embodiment, the direction-aware intersection-union (IUU) cost matrix can be an association cost matrix that fuses spatial overlap and historical motion direction consistency. Specifically, in densely intersecting target scenarios, relying solely on the IUU of predicted bounding boxes and detected bounding boxes for association can easily lead to mismatches of target trajectories to candidate detected bounding boxes that are spatially close but have inconsistent motion directions. Therefore, to restore the macroscopic motion direction consistency in densely intersecting scenarios, a direction-aware IUU cost matrix that fuses spatial overlap and historical motion direction consistency is constructed based on each predicted bounding box and the set of high-confidence detected boxes. The Hungarian algorithm is then used to perform a first-stage association between the maintained target trajectories and the set of high-confidence detected boxes, resulting in a first-stage matched trajectory set and a first-stage unmatched trajectory set. Here, a candidate detected bounding box can refer to any high-confidence detected box in the set of high-confidence detected boxes.
[0104] Optionally, the first-stage association targets the target trajectories maintained in the previous frame and the set of high-confidence detection boxes in the current frame. The target trajectories maintained in the previous frame include tracked target trajectories and target trajectories in a lost state. The first-stage association prioritizes the use of the high-confidence detection box set, which can improve the reliability of the initial association results.
[0105] In an optional embodiment, the target trajectory maintained in the previous frame includes the tracked target trajectory; the construction of the orientation-aware intersection-union cost matrix based on each predicted bounding box and the set of high-confidence detection boxes may include:
[0106] Extract the sequence of detection center points that are successfully matched in k consecutive frames of images for any tracked target trajectory; the k consecutive frames are k consecutive frames forward from the current frame.
[0107] By determining the mean displacement difference of the detection center point sequence, the historical motion velocity vector of any tracked target trajectory can be obtained;
[0108] Determine the instantaneous displacement vector of any high-confidence detection box relative to the most recent frame matching position of any tracked target trajectory. The most recent frame matching position is the detection box that was most recently successfully matched before the current frame.
[0109] Determine the directional similarity between the historical velocity vector and the instantaneous displacement vector, and truncate the directional similarity to the [0,1] interval;
[0110] Under the condition of meeting the preset trigger threshold, the spatial intersection-union ratio between the predicted bounding box corresponding to any tracked target trajectory and any high-confidence detection box is corrected according to the orientation similarity to obtain the orientation-aware intersection-union ratio.
[0111] Based on the direction-aware cross-union ratio between any tracked target trajectory and any high-confidence detection box, a direction-aware cross-union cost matrix is constructed.
[0112] In an optional embodiment, the directional similarity between the historical motion velocity vector and the instantaneous displacement vector is determined, and the directional similarity is truncated to the interval [0,1], as shown in the following formula:
[0113] ,and ;
[0114] in, Indicates directional similarity. Represents the historical velocity vector. Represents the instantaneous displacement vector. Used to avoid a denominator of zero;
[0115] Under the condition of meeting the preset trigger threshold, the spatial intersection-union ratio between any tracked target trajectory and any high-confidence detection box is adjusted according to the directional similarity to obtain the directional-aware intersection-union ratio, and the directional-aware intersection-union ratio is restricted to the interval [0,1], as shown in the following formula:
[0116] ,and ;
[0117] in, Indicates the intersection-union ratio of direction perception. This indicates motion direction perception based on historical velocity direction. This represents the spatial intersection-union ratio (IoU) between the predicted bounding boxes of the tracked target trajectory and the high-confidence detection boxes. This represents the directional consistency weight.
[0118] In one specific embodiment, the tracked target trajectory is a subset of the maintained target trajectories. Specifically, the tracked target trajectory refers to the trajectory whose status is tracked and has a sufficient number of historically successful observation center points. Optionally, the directional consistency reward only applies to trajectories in the tracked state; the directional consistency reward is not enabled for trajectories in the lost state.
[0119] In one specific embodiment, the sequence of detection center points that successfully match any tracked target trajectory in the most recent k frames is extracted. The most recent k frames can be k consecutive frames preceding the current frame. Then, the historical motion velocity vector of any tracked target trajectory is obtained by calculating the mean displacement difference of the detection center sequence. In this case, instead of directly using the instantaneous velocity estimation within the Kalman filter, the impact of noise can be reduced, thereby improving the accuracy of determining the association cost matrix. Optionally, the historical center coordinate sequence of any tracked target trajectory within a preset sliding window time before the current frame can be obtained, and the historical motion velocity vector corresponding to any tracked target trajectory can be smoothly calculated based on the historical center coordinate sequence.
[0120] Simultaneously, determine the instantaneous displacement vector of the center point of any high-confidence detection box relative to the most recently observed center point of any of the tracked target trajectories. The most recent observation center point of any tracked target trajectory can be the center point of the most recent matching position of any tracked target trajectory in the most recent frame. Specifically, the most recent observation center point of any tracked target trajectory can be the center point of the detection box that was most recently successfully matched with the target trajectory in the previous frame.
[0121] Next, by determining the cosine similarity between the historical velocity vector and the instantaneous displacement vector, the directional similarity between the two vectors is obtained, and this similarity is truncated to the [0,1] interval to avoid penalizing reverse motion, as shown in the above formula. ,and ,in, To prevent extremely small positive numbers with a denominator of zero, the directional similarity is non-negative. Optionally, truncating the directional similarity to the [0,1] interval ensures that the method only rewards motion in the same direction, avoiding unreasonable association penalties when the target makes a reasonable turnback or short-term reverse motion.
[0122] The preset trigger threshold can be set according to actual application requirements. Optionally, the preset trigger threshold can be that the spatial intersection-union ratio (CIU) between the preset bounding box and the high-confidence detection box is greater than or equal to a preset CIU threshold, and the magnitude of the historical motion velocity vector is greater than or equal to a preset velocity threshold. The preset CIU threshold and preset velocity threshold can be set according to actual application requirements. Before fusing directional similarity, invalid noise is filtered out using the preset trigger threshold. Optionally, if the preset trigger threshold is met, directional consistency reward is enabled. The CIU between the predicted bounding box corresponding to any of the tracked target trajectories and any of the high-confidence detection boxes is corrected based on directional similarity to obtain the directional-aware CIU, which is then restricted to the [0,1] interval, as shown in the above formula. ,and ,in It can take real numbers from 0 to 1, preferably... It can be 0.1~0.5, and in this embodiment it is taken as 0.3. Optionally, The directional consistency weight can be used as a directional alignment adjustment coefficient. Optionally, the directional-aware cross-union ratio (CUI) can be obtained by fusing the spatial CUI and directional similarity, and the influence of motion direction consistency on the association result can be controlled by the directional alignment adjustment coefficient. Optionally, if any of the preset trigger threshold conditions are not met, the original spatial CUI between any tracked target trajectory and any high-confidence detection box is used, i.e. In the above embodiments, the spatial intersection-union ratio is corrected by reward weighting based on the consistency between the historical motion direction and the instantaneous displacement direction, and only the same-direction motion is rewarded while the reverse direction penalty is truncated. This can reduce the association priority of candidate detection bounding boxes that overlap in space but have inconsistent motion directions.
[0123] Next, based on the direction-aware intersection-union ratio (CUIR) between any of the tracked target trajectories and any of the high-confidence detection boxes, a direction-aware CUIR cost matrix is constructed. Specifically, based on the direction-aware CUIR between any of the tracked target trajectories and any of the high-confidence detection boxes, the cost based on the direction-aware CUIR is determined. Based on the cost of direction-aware intersection-union ratio A direction-aware intersection-union ratio (IUU) cost matrix is constructed, which serves as the first-stage association cost matrix. In the above embodiment, constructing the first-stage association cost matrix based on the direction-aware IUU can give higher association priority to candidate detection bounding boxes with consistent motion directions and reduce the matching probability of candidate detection bounding boxes that overlap spatially but have inconsistent motion directions.
[0124] S105: For the set of unmatched tracked trajectories in the first stage of the association results, estimate the overall translation amount between the previous frame and the current frame based on the local pixel motion observation method.
[0125] In a specific embodiment, the first-stage association result may include a first-stage matched trajectory set and a first-stage unmatched trajectory set; the first-stage matched trajectory set may be a set of target trajectories successfully matched after the first-stage association, and the first-stage unmatched trajectory set may be a set of target trajectories that were not successfully matched after the first-stage association; wherein, the first-stage unmatched trajectory set may include a first-stage unmatched tracked trajectory set and a first-stage unmatched lost trajectory set, the first-stage unmatched tracked trajectory set may be a set of target trajectories that were not successfully matched in the first stage but are in a tracked state, and the first-stage unmatched lost trajectory set may be a set of target trajectories that were not successfully matched in the first stage but are in a lost state.
[0126] In one specific embodiment, after the first-stage association, some target trajectories may fail to match the high-confidence detection box set due to local occlusion, decreased detection confidence, or abrupt changes in motion state. For target trajectories that fail to match after the first-stage association, a micro-pixel motion observation correction process is initiated to compensate the Kalman filter prediction results based on local pixel motion information between adjacent frames, thereby restoring the consistency of micro-physical motion in occlusion or short-term loss states.
[0127] In an optional embodiment, estimating the overall translation amount between the set of unmatched tracked trajectories in the first stage of the first stage association results based on the local pixel motion observation method may include:
[0128] Corner features are extracted from the local region obtained by cropping the bounding box of the previous frame that does not match the tracked trajectory in each first stage;
[0129] The Lucas-Kanade local optical flow method with an image pyramid structure is adopted to track the corner features of each unmatched tracked trajectory in the first stage between the previous frame and the current frame, and the predicted center displacement is used as the initial flow guess of the Lucas-Kanade local optical flow method. The predicted center displacement is the displacement between the center point of the predicted bounding box of each unmatched tracked trajectory in the first stage and the center point of the detected bounding box in the previous frame or the most recently observed bounding box.
[0130] Based on the initial flow guess, the set of pixel-level displacement vectors of all successfully tracked corner points that did not match the tracked trajectory in the first stage is obtained by optical flow tracing;
[0131] Anomaly filtering is performed on the pixel-level displacement vector set based on a preset displacement threshold to obtain the effective corner point displacement set of each unmatched tracked trajectory in the first stage and the effective corner point features corresponding to each effective corner point displacement vector.
[0132] When the number of effective corner features is greater than or equal to the preset effective point number threshold, the median of the effective corner displacement set of each unmatched tracked trajectory in the first stage is determined as the overall translation amount of the unmatched tracked trajectory in the first stage between two adjacent frames.
[0133] In a specific embodiment, a local pixel motion observation method can be used to estimate the pixel-level displacement vector of each corner feature between two adjacent frames. Specifically, corner features are extracted in the local region obtained by cropping the bounding box of the previous frame in each first stage where the tracked trajectory is not matched. Optionally, the corner features of each corner feature that is not matched by the tracked trajectory in each first stage can be local feature points extracted in the local region obtained by cropping the bounding box of the previous frame in each first stage where the tracked trajectory is not matched. Optionally, selecting the center region of the bounding box as the feature point extraction region can help reduce the interference of background, adjacent targets or occlusions in the boundary region on motion estimation.
[0134] Next, the Lucas-Kanade local optical flow method with an image pyramid structure is used to track the corner features between the previous frame and the current frame. The predicted center displacement is used as the initial flow guess for the Lucas-Kanade local optical flow method to improve the tracking stability of corner features under large inter-frame displacement.
[0135] Furthermore, based on the initial flow guess, the set of pixel-level displacement vectors of all successfully tracked corner features that did not match the tracked trajectory in the first stage is obtained through optical flow tracking; that is, after Lucas-Kanade optical flow tracking with initial flow guess, the set of pixel-level displacement vectors of all successfully tracked corner features that did not match the tracked trajectory in the first stage between the previous frame and the current frame can be obtained.
[0136] Next, anomaly filtering is performed on the pixel-level displacement vector set according to a preset displacement threshold to obtain the effective corner point displacement set for each unmatched tracked trajectory in the first stage, as well as the effective corner point features corresponding to each effective corner point displacement vector; optionally, the effective corner point displacement set obtained by optical flow tracking can be as shown in the following formula:
[0137] ;
[0138] in, Indicates the number of valid corner features. This represents the effective corner displacement vector corresponding to the i-th effective corner feature, i.e., the pixel-level displacement vector of the i-th effective corner feature between two adjacent frames. Optionally, the preset displacement threshold can be set according to actual application requirements. Optionally, the preset displacement threshold is a preset maximum displacement threshold. Specifically, based on the preset maximum displacement threshold, the set of pixel-level displacement vectors of all successfully tracked corner features that do not match the tracked trajectory in each first stage is filtered for anomalies to obtain the set of effective corner displacements that do not match the tracked trajectory in each first stage.
[0139] Next, to reduce the impact of abnormal displacement caused by local occlusion, background noise, or interference from adjacent targets, when the number of effective corner point features after filtering is greater than or equal to a preset effective point number threshold, the median of the effective corner point displacement set of each unmatched tracked trajectory in the first stage can be used as a robust estimate to obtain the overall translation amount of each unmatched tracked trajectory in the first stage between two adjacent frames. Specifically, the median of the x-direction displacement vector and the median of the y-direction displacement vector in the effective corner point displacement set of each unmatched tracked trajectory in the first stage are determined as the overall translation amount of each unmatched tracked trajectory in the first stage, as shown in the following formula:
[0140] ;
[0141] in, This represents the overall translation amount of the unmatched tracked trajectory in the first stage between two adjacent frames, i.e., the overall translation amount of the target corresponding to the unmatched tracked trajectory in the first stage between two adjacent frames, which can characterize the local true motion trend of the target. Optionally, the preset effective point threshold can be set according to actual application requirements. Specifically, when the number of effective corner point features is not less than the preset effective point threshold, the median of the effective corner point displacement set of the unmatched tracked trajectory in the first stage is used as a robust estimate to obtain the overall translation amount representing the overall motion trend of the target corresponding to the unmatched tracked trajectory in the first stage. Optionally, the overall translation amount can represent the overall motion trend of the target corresponding to the unmatched tracked trajectory in the first stage.
[0142] S106: Use the overall translation of the set of unmatched tracked trajectories in the first stage to correct and update the mean of the corresponding predicted state, and determine the bounding box of each unmatched correction prediction.
[0143] In an optional embodiment, the above-mentioned use of the overall translation of the first-stage unmatched tracked trajectory set to correct and update the corresponding predicted state mean, and determining each unmatched correction prediction bounding box may include:
[0144] The mean predicted state of each unmatched tracked trajectory in the first stage is corrected and updated using the overall translation amount of each unmatched tracked trajectory in the first stage, resulting in the mean corrected predicted state of each unmatched trajectory, as shown in the following formula:
[0145]
[0146] in, This represents the mean of the unmatched corrective prediction state. This represents the predicted mean state. This indicates the total translation amount of the unmatched tracked trajectory in the first stage between two adjacent frames;
[0147] Decode the mean value of each unmatched correction prediction state to obtain the bounding box of each unmatched correction prediction.
[0148] In a specific embodiment, the mean of the unmatched correction prediction state can be the mean of the prediction state obtained by Kalman filtering after correction for the unmatched tracked trajectory in the first stage; the unmatched correction prediction bounding box can be the prediction bounding box after correction for the unmatched tracked trajectory in the first stage.
[0149] In a specific embodiment, the aforementioned overall translation amount can be used to correct and update the mean of the prior predicted states output by the Kalman filter. Taking the aforementioned eight-dimensional state vector as an example, the mean of the prior predicted states output by the Kalman filter is defined as... Its state components can include the target center coordinates, scale parameters, and velocity components. The mean value of the predicted state after correction is shown in the following formula.
[0150]
[0151] The aforementioned correction method only performs translational correction on the target center coordinate components, without directly altering the target scale and velocity components. This allows the predicted center position to be pulled back to a location more consistent with local pixel motion observations, while preserving the Kalman filter motion state estimation results. Optionally, the overall translation amount can be used to pull the drifted predicted center position back to a location more consistent with local pixel motion observations. This predicted center position can be the detection box center coordinates corresponding to the mean of the prior predicted state output by the Kalman filter.
[0152] In one specific embodiment, the prediction covariance can be adaptively dilated based on the number of consecutive unmatched frames of the unmatched tracked trajectory in the first stage, as shown in the following equation:
[0153] ,in, This represents the inflated predicted covariance. This represents the adaptive inflation coefficient, which is optional. The inflated predicted covariance can be used for the second-stage association.
[0154] S107: Based on the set of unmatched correction prediction bounding boxes and low-confidence detection boxes, construct the second-stage association cost matrix, and perform the second-stage association on the set of unmatched tracked trajectories and the set of low-confidence detection boxes in the first stage to obtain the second-stage association result;
[0155] In one specific embodiment, the second-stage association can be used to remediate and continue target trajectories that failed to match successfully in the first stage due to occlusion, blurring, or weakened detection response. Optionally, the second-stage association objects are the set of unmatched tracked trajectories in the first stage and the set of low-confidence detection boxes in the current frame. Specifically, the set of target trajectories that failed to match successfully in the first stage but are in a tracked state are associated with the set of low-confidence detection boxes in the second stage, while the set of target trajectories that failed to match successfully in the first stage but are in a lost state are not included in the second-stage association. Specifically, a second-stage association cost matrix can be constructed based on the spatial intersection-union ratio between each unmatched predicted bounding box and each low-confidence detection box, and the Hungarian algorithm or a greedy matching strategy can be used to associate the set of unmatched tracked trajectories that failed to match successfully in the first stage with the set of low-confidence detection boxes in the second stage to obtain the second-stage association result. This allows for the remedial continuation of target trajectories that failed to match successfully in the first stage but are in a tracked state due to occlusion, blurring, or short-term loss.
[0156] S108: Update the trajectory status of each target based on the first-stage association results and the second-stage association results, and output the multi-target tracking results.
[0157] In an optional embodiment, the first-stage association result includes a set of matched trajectories and a set of unmatched and lost trajectories in the first stage, and the second-stage association result includes a set of matched trajectories in the second stage and a set of unmatched trajectories in the second stage; updating the status of each target trajectory based on the first-stage association result and the second-stage association result, and outputting multi-target tracking results, may include:
[0158] Each matched trajectory in the first stage is marked as active, and its predicted state mean is updated using its matched high-confidence detection box to obtain the first updated state mean.
[0159] Each matched trajectory in the second stage is marked as an active state, and its predicted state mean is updated using the low-confidence detection boxes it matches, to obtain the second updated state mean.
[0160] Mark each unmatched lost trajectory in the first stage and each unmatched trajectory in the second stage as a lost state, and accumulate the number of consecutive lost frames;
[0161] When the number of consecutive lost frames of a target trajectory in a lost state is greater than or equal to a preset buffer threshold, the target trajectory in a lost state will be deleted.
[0162] Based on the newly created detection bounding box, determine the initial state mean and generate a new target trajectory. The newly created detection bounding box is the detection bounding box that has not matched any target trajectory after the first stage association and has a confidence score greater than or equal to the preset new threshold.
[0163] Output the multi-target tracking results based on the matched trajectories that are active in the current frame.
[0164] In one specific embodiment, the set of matched trajectories in the second stage may include target trajectories that were successfully matched after being associated in the second stage from the set of unmatched tracked trajectories in the first stage; the set of unmatched trajectories in the second stage may include target trajectories that were not successfully matched after being associated in the second stage from the set of unmatched tracked trajectories in the first stage. Specifically, the unmatched trajectories in the second stage may be target trajectories that were not successfully matched in both the first stage association and the second stage association.
[0165] In a specific embodiment, for target trajectories successfully matched in the first-stage association, their corresponding high-confidence detection boxes are directly used as observations to update their predicted state values, and their trajectory states are marked as active. For target trajectories successfully matched in the second-stage association, their corresponding low-confidence detection boxes are used as observations to update their predicted state mean, and their trajectory states are marked as active. Optionally, trajectory attributes are updated simultaneously. Optionally, the first and second updated state mean values obtained from updating the predicted state mean values can be used for Kalman filtering prior prediction in the next frame. Specifically, the first updated state mean value can be determined in the following way: within the high-confidence detection bounding box represented as... , among which, among which The coordinates of the bounding box center are The width of the bounding box. The bounding box height is converted into a Kalman filter observation vector. ,in The aspect ratio is represented, and then the first updated state mean is constructed based on the Kalman filter observation vectors, where the position and scale components are determined by detecting bounding boxes with high confidence. It is determined that the velocity component can be indirectly corrected by posterior correction after Kalman filtering based on the deviation between the predicted bounding box and the high-confidence detection box.
[0166] For target trajectories that fail to match after the first and second stages of association (i.e., the first-stage unmatched lost trajectory and the second-stage unmatched trajectory), their trajectory status is marked as lost, and the number of consecutive lost frames is accumulated. When the number of consecutive lost frames of a target trajectory in the lost state exceeds a preset buffer threshold, the target trajectory is considered to have left the field of view or cannot be reliably recovered, and it is deleted. The preset buffer threshold can be set according to actual application requirements. Optionally, if the number of consecutive lost frames of a target trajectory in the lost state does not exceed the preset buffer threshold, it can still be found by subsequent detection boxes, and detection box matching can continue in subsequent frames.
[0167] For detection bounding boxes that do not match any target trajectory after the first-stage association and whose confidence score is not lower than the preset new detection threshold (i.e., the new detection bounding boxes), a new target identity can be assigned to them. The initial state mean and initial covariance are determined based on the initial state mean of the new detection bounding box, and a new target trajectory is generated. The preset new detection threshold can be set according to actual application requirements. Optionally, the trajectory lifecycle parameters corresponding to the new detection bounding box can also be determined. Specifically, the initial state mean can be determined in the following way: (The text then repeats the definition of the new detection bounding box.) ,in The coordinates of the bounding box center are The width of the bounding box. The bounding box height is converted into a Kalman filter observation vector. ,in The aspect ratio is represented, and then an eight-dimensional initial state mean is constructed based on the Kalman filter observation vector: In this process, the position and scale components are determined by the corresponding bounding boxes to be newly created for detection, while the velocity component is initialized to 0. Specifically, the covariance is initialized based on the scale of the bounding boxes to be newly created for detection. Furthermore, a new trajectory ID can be assigned to the trajectory corresponding to the bounding box to be newly created for detection, and trajectory attributes such as trajectory state, starting frame number, current frame number, confidence level, category, and historical observation center point can be initialized to generate a new target trajectory. Optionally, low-confidence detection boxes are preferentially used for the second-stage remedial association and are generally not directly used to initialize new trajectories to reduce the probability of false trajectory generation due to false detections.
[0168] Next, based on the active target trajectories in the current frame, multi-target tracking results can be output. These results may include frame number, target identifier, bounding box position, bounding box size, and trajectory status. Further outputs of trajectory history, velocity estimation, or alarm event information can be provided based on application requirements.
[0169] Figure 2 This is a schematic diagram of a dense multi-target tracking process provided by an embodiment of the present invention; as shown below. Figure 2As shown, the system acquires a continuous sequence of video frames to be processed and reads the current frame image in chronological order. Then, the current frame image is input into a pre-trained single-stage multi-object detection model, which outputs a set of detection bounding boxes in the current frame image and the confidence score corresponding to each bounding box. Based on a preset confidence threshold, the set of detection bounding boxes is divided into a high-confidence detection box set and a low-confidence detection box set. The system also acquires the target trajectories maintained in the previous frame, performs prior predictions on these trajectories based on Kalman filtering, obtains the mean predicted state of each target trajectory in the current frame, and decodes the mean predicted state to obtain the predicted bounding boxes of each target trajectory in the current frame. Next, based on each predicted bounding box and the set of high-confidence detection boxes, a direction-aware intersection-union cost matrix is constructed. Based on the direction-aware intersection-union cost matrix, the Hungarian algorithm is used to perform the first-stage association between each target trajectory and the set of high-confidence detection boxes, obtaining the first-stage association result. The first-stage association result includes the first-stage matched trajectory set and the first-stage unmatched trajectory set. The first-stage unmatched trajectory set includes the first-stage unmatched tracked trajectory set and the first-stage unmatched lost trajectory set. Then, each first-stage matched trajectory can be marked as active, and its predicted state mean is updated using its matched high-confidence detection boxes to obtain the first updated state mean. At this time, the first-stage matched trajectory can be the target trajectory in the tracked state. Next, corner features are extracted from the local region obtained by cropping the bounding box of the previous frame for each unmatched tracked trajectory in the first-stage unmatched trajectory set. The Lucas-Kanade local optical flow method with an image pyramid structure is used to track the corner features of each unmatched tracked trajectory between the previous frame and the current frame, and to determine the set of pixel-level displacement vectors of all successfully tracked corner features for each unmatched tracked trajectory. Anomalies are filtered from the set of pixel-level displacement vectors according to a preset displacement threshold to obtain the set of effective corner displacements for each unmatched tracked trajectory and the effective corner features corresponding to each effective corner displacement vector. When the number of effective corner features is greater than or equal to the preset effective number threshold, the median of the set of effective corner displacements for each unmatched tracked trajectory is determined to be the overall translation amount of each unmatched tracked trajectory between two adjacent frames.Next, the overall translation of the first-stage unmatched tracked trajectory set is used to correct and update the corresponding predicted state mean, determining the bounding boxes for each unmatched tracked trajectory. Based on the unmatched tracked trajectory predicted bounding boxes and the low-confidence detection box set, a second-stage association cost matrix is constructed. Based on the second-stage association cost matrix, the Hungarian algorithm is used to perform second-stage association between the first-stage unmatched tracked trajectory set and the low-confidence detection box set, obtaining the second-stage association result, which includes the second-stage matched trajectory set and the second-stage unmatched trajectory set. Then, each second-stage matched trajectory is marked as active, and its predicted state mean is updated using its matched low-confidence detection boxes, obtaining the second updated state mean. Each first-stage unmatched lost trajectory and each second-stage unmatched trajectory are marked as lost, and their consecutive lost frame count is accumulated. When the consecutive lost frame count of a target trajectory in a lost state is greater than or equal to a preset buffer threshold, the target trajectory in a lost state is deleted. Finally, based on the active matched trajectories in the current frame, the multi-target tracking result is output.
[0170] In the above embodiments, a complete multi-target tracking process is formed through the collaborative processing of target detection, confidence grading, Kalman prior prediction, direction-aware primary association, micro-pixel motion correction, and second-stage remedial association. In the first stage, historical motion direction consistency constraints are used to reduce association ambiguity in densely intersecting scenes. In the second stage, local pixel motion observations are used to predict and correct the state of unmatched tracked trajectories. Therefore, without relying on a deep appearance re-identification network, the trajectory continuity, identity preservation capability, and real-time deployment adaptability in densely occluded and target-intersecting scenes can be improved. Specifically, this embodiment is applicable to application scenarios with high requirements for real-time performance and computational cost, such as edge security, drone inspection, vehicle-mounted sensing, and dense crowd monitoring.
[0171] As can be seen from the technical solutions provided in the embodiments of this specification above, this specification generates a set of detection bounding boxes and their corresponding confidence scores by inputting the acquired current frame image into a trained multi-object detection model; according to a preset confidence threshold, the set of detection bounding boxes is divided into a set of high-confidence detection boxes and a set of low-confidence detection boxes; based on the Kalman filter, the mean of the predicted states of each target trajectory in the current frame, obtained from the prior predictions of each target trajectory maintained in the previous frame, is used to determine the predicted bounding boxes of each target trajectory in the current frame; then, based on each predicted bounding box and the set of high-confidence detection boxes, a direction-aware intersection-union cost matrix is constructed, and the target trajectory and the high-confidence... The detection box set undergoes a first-stage association process to obtain the first-stage association result. By constructing a direction-aware intersection-union cost matrix that integrates spatial overlap and historical motion direction consistency, high-confidence detection boxes with consistent motion directions can be given higher association priority, while reducing the matching probability of high-confidence detection boxes that overlap spatially but have inconsistent motion directions. This can alleviate the direction confusion problem caused by relying solely on spatial intersection-union when dense targets are moving towards each other, crossing each other, or overlapping for short periods, reducing identity switching and trajectory confusion, thereby reducing association ambiguity in dense and intersecting scenarios and improving identity preservation capabilities. Next, for the set of tracked trajectories that were not matched in the first-stage association result, based on local... The pixel motion observation method estimates the overall translation between the previous and current frames. The mean of the corresponding predicted state is then corrected and updated using the overall translation of the first-stage unmatched tracked trajectory set to determine each unmatched corrected predicted bounding box. Based on each unmatched corrected predicted bounding box and the low-confidence detection box set, a second-stage association cost matrix is constructed. The first-stage unmatched tracked trajectory set and the low-confidence detection box set are then associated in the second stage to obtain the second-stage association result. The overall offset is used to correct and update the predicted bounding boxes, and the corrected predicted bounding boxes are then associated with the low-confidence detection boxes in the second stage to identify the drifted predicted positions. The system pulls the target back to a position that better reflects its actual local motion, improving the effectiveness and reliability of the second-stage association and thus enhancing trajectory continuity under occlusion or short-term loss conditions. Then, it updates the trajectory state of each target based on the first-stage and second-stage association results, outputting multi-target tracking results. This combines the macroscopic motion direction consistency constraint of the first stage with the microscopic pixel motion correction mechanism of the second stage, reducing the probability of erroneous association when targets intersect and compensating for and restoring the predicted state when occlusion or decreased detection confidence occurs. Through this collaborative mechanism, the trajectory continuity, identity stability, and adaptability to complex scenes of dense multi-target tracking results can be improved.Furthermore, by introducing the consistency of historical motion direction and local pixel motion information of the target during the association process, without the need to configure an additional deep appearance re-identification network outside the target detection model, the trajectory in densely occluded and target-interleaved scenes can be effectively continued and the identity preserved. This improves the effectiveness, accuracy and convenience of determining multi-target tracking results. As a result, the dependence on the deep appearance re-identification network can be reduced, and the trajectory continuity, identity preservation capability and real-time deployment adaptability in densely occluded and target-interleaved scenes can be improved. It also reduces the number of model parameters, memory usage and inference latency caused by appearance feature extraction, which is conducive to improving the overall processing frame rate of the system and improving its deployment adaptability on computing-constrained devices such as drones, vehicle computing platforms, and edge security cameras.
[0172] This invention also provides a dense multi-target tracking system, accordingly, Figure 3 This is a schematic diagram of the structure of a dense multi-target tracking system provided in an embodiment of the present invention; as shown below. Figure 3 As shown, the above system includes:
[0173] The object detection module 310 is used to input the acquired current frame image into the trained multi-object detection model to generate a set of detection bounding boxes corresponding to the current frame image and their corresponding confidence scores.
[0174] The confidence leveling module 320 is used to divide the set of detection bounding boxes into a set of high-confidence detection boxes and a set of low-confidence detection boxes according to a preset confidence threshold.
[0175] The motion state prediction module 330 is used to determine the predicted bounding box of each target trajectory in the current frame based on the average predicted state of each target trajectory obtained by prior prediction of each target trajectory maintained in the previous frame using Kalman filtering.
[0176] The direction-aware primary association module 340 is used to construct a direction-aware intersection-union cost matrix based on each predicted bounding box and the set of high-confidence detection boxes, and to perform a first-stage association between each target trajectory and the set of high-confidence detection boxes to obtain a first-stage association result; the direction-aware intersection-union cost matrix integrates spatial overlap with the consistency of historical motion direction;
[0177] The micro-pixel motion correction module 350 is used to estimate the overall translation amount between the previous frame and the current frame for the set of unmatched tracked trajectories in the first stage association result based on the local pixel motion observation method; and to correct and update the mean of the corresponding prediction state by using the overall translation amount of the set of unmatched tracked trajectories in the first stage to determine the bounding box of each unmatched correction prediction.
[0178] The second-stage association module 360 is used to construct a second-stage association cost matrix based on the unmatched correction prediction bounding boxes and the low-confidence detection box set, and to perform second-stage association on the first-stage unmatched tracked trajectory set and the low-confidence detection box set to obtain the second-stage association result.
[0179] The trajectory update and tracking result determination module 370 is used to update the trajectory status of each target based on the first stage association result and the second stage association result, and output the multi-target tracking result.
[0180] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the dense multi-target tracking method as described in any of the method embodiments.
[0181] Embodiments of the present invention also provide a computer storage medium, which may be disposed in a server to store at least one instruction, at least one program, code set, or instruction set for implementing the method embodiments, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the dense multi-target tracking method as described in any of the method embodiments.
[0182] Optionally, in embodiments of the present invention, the storage medium may be located at at least one of a plurality of network servers in a computer network. Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0183] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0184] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more flowcharts and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more flowcharts and / or boxes Figure 1 The function specified in one or more boxes.
[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more flowcharts and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0188] Finally, it should be noted that the embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.
Claims
1. A dense multi-target tracking method, characterized in that, The method includes: The acquired current frame image is input into the trained multi-object detection model to generate a set of detection bounding boxes corresponding to the current frame image and its corresponding confidence score; Based on a preset confidence threshold, the set of detection bounding boxes is divided into a set of high-confidence detection boxes and a set of low-confidence detection boxes. The predicted bounding box of each target trajectory in the current frame is determined by the mean of the predicted state of each target trajectory in the current frame, which is obtained by prior prediction of each target trajectory maintained in the previous frame using Kalman filtering. Based on each predicted bounding box and the set of high-confidence detection boxes, a direction-aware intersection-union cost matrix is constructed, and a first-stage association is performed on each target trajectory and the set of high-confidence detection boxes to obtain the first-stage association result; the direction-aware intersection-union cost matrix integrates spatial overlap with the consistency of historical motion direction; For the set of unmatched tracked trajectories in the first stage of the association results, the overall translation amount between the previous frame and the current frame is estimated based on the local pixel motion observation method. The mean of the corresponding predicted state is corrected and updated using the overall translation of the set of unmatched tracked trajectories in the first stage, and the bounding boxes of each unmatched correction prediction are determined. Based on the unmatched correction prediction bounding boxes and the low-confidence detection box set, a second-stage association cost matrix is constructed, and the first-stage unmatched tracked trajectory set and the low-confidence detection box set are associated in the second stage to obtain the second-stage association result. Update the trajectory status of each target based on the first-stage association results and the second-stage association results, and output the multi-target tracking results.
2. The dense multi-target tracking method according to claim 1, characterized in that, The target trajectory maintained in the previous frame includes the target trajectory that has been tracked; The construction of the orientation-aware intersection-union cost matrix based on each predicted bounding box and the set of high-confidence detection boxes includes: Extract the sequence of detection center points that are successfully matched in k consecutive frames of images for any tracked target trajectory; the k consecutive frames are k consecutive frames forward from the current frame. By determining the mean displacement difference of the detection center point sequence, the historical motion velocity vector of any tracked target trajectory is obtained; Determine the instantaneous displacement vector of any high-confidence detection box relative to the most recent frame matching position of any tracked target trajectory, wherein the most recent frame matching position is the detection bounding box that was most recently successfully matched before the current frame; Determine the directional similarity between the historical motion velocity vector and the instantaneous displacement vector, and truncate the directional similarity to the [0,1] interval; If a preset trigger threshold is met, the spatial intersection-union ratio between the predicted bounding box corresponding to any tracked target trajectory and any high-confidence detection box is corrected according to the directional similarity to obtain the directional-aware intersection-union ratio. Based on the direction-aware cross-union ratio between any tracked target trajectory and any high-confidence detection box, a direction-aware cross-union cost matrix is constructed.
3. The dense multi-target tracking method according to claim 2, characterized in that, The directional similarity between the historical motion velocity vector and the instantaneous displacement vector is determined, and the directional similarity is truncated to the interval [0,1], as shown in the following formula: ,and ; in, Indicates directional similarity. Represents the historical velocity vector. Represents the instantaneous displacement vector. Used to avoid a denominator of zero; When a preset trigger threshold is met, the spatial intersection-union ratio between any tracked target trajectory and any high-confidence detection box is adjusted based on the directional similarity to obtain the directional-aware intersection-union ratio, and the directional-aware intersection-union ratio is restricted to the [0,1] interval, as shown in the following formula: ,and ; in, Indicates the intersection-union ratio of direction perception. This indicates motion direction perception based on historical velocity direction. This represents the spatial intersection-union ratio (IoU) between the predicted bounding boxes of the tracked target trajectory and the high-confidence detection boxes. This represents the directional consistency weight.
4. The dense multi-target tracking method according to claim 1, characterized in that, The step of estimating the overall translation amount between the set of unmatched tracked trajectories in the first stage of the association results based on the local pixel motion observation method includes: Corner features are extracted from the local region obtained by cropping the bounding box of the previous frame that does not match the tracked trajectory in each first stage; The Lucas-Kanade local optical flow method with an image pyramid structure is used to track the corner features of each unmatched tracked trajectory in the first stage between the previous frame and the current frame, and the predicted center displacement is used as the initial flow guess of the Lucas-Kanade local optical flow method; the predicted center displacement is the displacement between the center point of the predicted bounding box of each unmatched tracked trajectory in the first stage and the center point of the detected bounding box in the previous frame or the most recently observed bounding box. Based on the initial flow guess, the set of pixel-level displacement vectors of all successfully tracked corner point features that did not match the tracked trajectory in each first stage is obtained by optical flow tracing; Anomaly filtering is performed on the pixel-level displacement vector set according to a preset displacement threshold to obtain the effective corner point displacement set of each first-stage unmatched tracked trajectory and the effective corner point features corresponding to each effective corner point displacement vector. When the number of effective corner features is greater than or equal to a preset effective point number threshold, the median of the effective corner displacement set of each unmatched tracked trajectory in the first stage is determined as the overall translation amount of each unmatched tracked trajectory in the first stage between two adjacent frames.
5. The dense multi-target tracking method according to claim 1, characterized in that, The step of using the overall translation of the unmatched tracked trajectory set in the first stage to correct and update the mean of its corresponding predicted state, and determining the bounding boxes of each unmatched correction prediction, includes: The mean predicted state of each unmatched tracked trajectory in the first stage is corrected and updated using the overall translation amount of each unmatched tracked trajectory in the first stage, resulting in the mean corrected predicted state of each unmatched trajectory, as shown in the following formula: in, This represents the mean of the unmatched corrective prediction state. This represents the predicted mean state. This indicates the total translation amount of the unmatched tracked trajectory in the first stage between two adjacent frames; The mean value of each unmatched correction prediction state is decoded to obtain the bounding box of each unmatched correction prediction state.
6. The dense multi-target tracking method according to claim 1, characterized in that, The first-stage association result includes the first-stage matched trajectory set and the first-stage unmatched lost trajectory set; the second-stage association result includes the second-stage matched trajectory set and the second-stage unmatched trajectory set. The step of updating the trajectory status of each target based on the first-stage association result and the second-stage association result, and outputting multi-target tracking results, includes: Each matched trajectory in the first stage is marked as active, and its predicted state mean is updated using the high-confidence detection box that it matches, to obtain the first updated state mean. Each matched trajectory in the second stage is marked as an active state, and its predicted state mean is updated using the low-confidence detection boxes it matches, to obtain the second updated state mean. Mark each unmatched lost trajectory in the first stage and each unmatched trajectory in the second stage as a lost state, and accumulate the number of consecutive lost frames; When the number of consecutive lost frames of a target trajectory in a lost state is greater than or equal to a preset buffer threshold, the target trajectory in a lost state is deleted. Based on the newly created detection bounding box, the initial state mean is determined, and a new target trajectory is generated. The newly created detection bounding box is a detection bounding box that has not matched any target trajectory after the first stage association and whose confidence score is greater than or equal to the preset new creation threshold. Output the multi-target tracking results based on the matched trajectories that are active in the current frame.
7. The dense multi-target tracking method according to claim 1, characterized in that, The step of dividing the detection bounding box set into a high-confidence detection box set and a low-confidence detection box set according to a preset confidence threshold includes: Obtain the preset confidence threshold and the confidence score corresponding to the detection bounding box set; the preset confidence threshold includes a preset high threshold and a preset low threshold; Detection bounding boxes with confidence scores greater than or equal to the preset high threshold are included in the high-confidence detection box set; Detection bounding boxes with confidence scores greater than or equal to the preset low threshold and less than the preset high threshold are included in the low confidence detection box set; Detection bounding boxes with confidence scores lower than the preset low threshold are filtered out.
8. A dense multi-target tracking system, characterized in that, The system includes: The object detection module is used to input the acquired current frame image into the trained multi-object detection model to generate a set of detection bounding boxes corresponding to the current frame image and their corresponding confidence scores; The confidence leveling module is used to divide the set of detection bounding boxes into a set of high-confidence detection boxes and a set of low-confidence detection boxes according to a preset confidence threshold. The motion state prediction module is used to determine the predicted bounding box of each target trajectory in the current frame by using the average predicted state of each target trajectory in the current frame based on the prior prediction of each target trajectory maintained in the previous frame by Kalman filtering. The direction-aware primary association module is used to construct a direction-aware intersection-union cost matrix based on each predicted bounding box and the set of high-confidence detection boxes, and to perform a first-stage association between each target trajectory and the set of high-confidence detection boxes to obtain the first-stage association result; the direction-aware intersection-union cost matrix integrates spatial overlap with the consistency of historical motion direction; The micro-pixel motion correction module is used to estimate the overall translation amount between the previous frame and the current frame for the set of unmatched tracked trajectories in the first stage of the association result of the first stage based on the local pixel motion observation method; and to correct and update the mean of the corresponding prediction state by using the overall translation amount of the set of unmatched tracked trajectories in the first stage, and to determine the bounding box of each unmatched correction prediction. The second-stage association module is used to construct a second-stage association cost matrix based on the unmatched correction prediction bounding boxes and the low-confidence detection box set, and to perform second-stage association on the first-stage unmatched tracked trajectory set and the low-confidence detection box set to obtain the second-stage association result. The trajectory update and tracking result determination module is used to update the trajectory status of each target based on the first stage association result and the second stage association result, and output the multi-target tracking result.
9. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction and at least one program, the at least one instruction and the at least one program being loaded and executed by the processor to implement the dense multi-target tracking method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction and at least one program, which are loaded and executed by a processor to implement the dense multi-target tracking method as described in any one of claims 1 to 7.
