Multi-target tracking method based on trajectory pipeline regression prediction and association

Abstract

The invention records a multi-target tracking method based on trajectory pipeline regression prediction and association, and the method comprises the following steps: obtaining and recognizing scene targets, including an existing target and a new target; matching and splicing the existing target and the track pipeline corresponding to the identity label of the existing target, and outputting the updated track pipeline of the existing target; constructing a track pipeline corresponding to the new target identity label, and outputting the track pipeline of the new target; performing position correction on the output track pipeline, and outputting a predicted position of the scene target in the current frame; two track pipelines are randomly selected, the space-time correlation degree, the position error and the shape error between the two track pipelines are calculated, quality scoring is carried out on the two track pipelines, and suppression operation is carried out on the track pipelines which have conflicts and are low in quality score; and executing the operation on all the track pipelines of the scene target, and outputting the identities of all the track pipelines which are not inhibited and the predicted positions of the current frames of the track pipelines to finish tracking of the scene target.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a multi-target tracking method based on trajectory pipeline regression prediction and association. Background Technology

[0002] Multi-object tracking is based on object detection. A separate detection module acquires the positions of candidate objects in each frame of the video. Currently, the following scheme is commonly used to continuously track these candidate objects:

[0003] The independent detection and data association (TBD) scheme decouples the tracking task into two independent steps: detection and association. First, a high-performance target detector generates target bounding boxes in each frame. Then, a data association module links the bounding boxes in different frames to form a trajectory.

[0004] The target regression-based scheme (TBR) uses a regressor to predict the target's new position in the current frame based on the target's position and features in historical frames. For example, the target detector's own regression head can be used as input to directly regress the new position in the current frame, and the trajectory can be implicitly maintained through the continuity of the position.

[0005] An augmentation scheme based on recurrent neural networks (RNNs) encodes the short-term motion and appearance changes of the target through its internal memory units, and uses the last hidden state as input to a fully connected layer to predict the target's next location or association.

[0006] The above schemes are based on the detection results of a single frame, the geometric relationship between targets, or short-term temporal features. Their information sources are limited to isolated time points or too short temporal segments. Once the tracking environment becomes complex, such as long-term target occlusion, rapid movement, or frequent intersection of multiple target trajectories, the tracking effect will significantly decrease due to the inability to effectively handle information interruption and identity ambiguity. It is difficult to effectively track targets in a long-term, stable, and consistent manner in complex dynamic environments. Summary of the Invention

[0007] To address some of the problems existing in the prior art, this invention describes a multi-target tracking method based on trajectory pipeline regression prediction and correlation, the specific technical solution of which is as follows:

[0008] A multi-target tracking method based on trajectory pipeline regression prediction and correlation includes the following steps:

[0009] S1, acquire and identify scene targets, including existing targets and new targets, and output the trajectory pipeline of scene targets, including: matching and splicing the trajectory pipelines corresponding to the existing targets and their identity identifiers, and outputting the updated trajectory pipeline of the existing targets; constructing the trajectory pipeline corresponding to the identity identifier of the new targets, and outputting the trajectory pipeline of the new targets;

[0010] S2, perform position correction on the trajectory pipeline output in step S1, and output the predicted position of the scene target in the current frame;

[0011] S3, arbitrarily select two trajectory pipelines and calculate the spatiotemporal correlation, position error and shape error between them. Then, score the quality of the two trajectory pipelines respectively, and perform suppression operation on the trajectory pipelines that conflict and have low quality scores.

[0012] S4. Perform step S3 on all trajectory pipelines of the scene target, output the identity of all unsuppressed trajectory pipelines and their current frame prediction position, and complete the tracking of the scene target.

[0013] Furthermore,

[0014] The Faster R-CNN network model outputs scene target detection information, including detection boxes and raw classification confidence scores. The detection box information includes bounding box coordinates and appearance feature vectors.

[0015] Calculate the Mahalanobis distance between the detection information of each existing and new target and the predicted position of the trajectory pipeline, and use this Mahalanobis distance as the motion cost;

[0016] Define a set of detection information and trajectory pipelines that need to be matched as a detection-trajectory pair, set an association threshold, and proceed to the next layer of matching for detection-trajectory pairs whose motion cost is lower than the association threshold; otherwise, terminate the matching and output the cost matrix.

[0017] Input the cost matrix into the Hungarian algorithm and output the matching results of detection information and trajectory pipelines, including successfully matched detection-trajectory pairs and unsuccessfully matched detection information. The unsuccessfully matched detection information includes unmatched detection information that failed to match any trajectory pipeline and / or unmatched trajectory pipelines that failed to match any detection information.

[0018] For a successfully matched detection-trajectory pair, the bounding box coordinates of the matching detection information are used as new observations and input into the Kalman filter for motion state correction and update. Trajectory pipelines that do not match are not updated.

[0019] Furthermore,

[0020] The following steps will output the predicted position of the scene target in the current frame:

[0021] Extract the first line from the trajectory pipeline Frame and the The image features of the frame are used to predict the preliminary regression results of the scene target at the current position from time t and time t-1, respectively;

[0022] Based on the position loss between the initial regression bounding box and the historical bounding box of the scene target, a position loss matrix of the scene target is constructed;

[0023] For scene targets with the same identity identifier, the intersection-union ratio between the two sets of regression positions after correction is used as the confidence level of the scene target's regression prediction, and the confidence level of the scene target's regression prediction is compared with a preset threshold:

[0024] If the confidence level is greater than or equal to the preset threshold, one of them is randomly selected as the final regression bounding box for the predicted location of the target in the scene;

[0025] If the confidence level is less than the preset threshold, the regression position with higher confidence level among the two sets of regression positions of the target in the scene is used as the final regression bounding box, and the regression position with lower confidence level is defined as the potential target, thus constructing a set of potential targets.

[0026] Furthermore,

[0027] The following expression is used to predict the preliminary regression result of the scene target at the current position;

[0028]

[0029]

[0030] In the formula, and These are preliminary regression results; and For the collection of historical bounding boxes; and For the corresponding image features; This is a regression prediction network.

[0031] Furthermore,

[0032] Construct the potential target set using the following expression:

[0033]

[0034] In the formula, Let be the set of potential targets in frame t+1; This represents the k-th low-confidence prediction location (k=1,2,…); and For two sets of regression positions (i, j are the target indices); This is a preset threshold.

[0035] Furthermore,

[0036] Constructing the location loss matrix of the scene target includes the following steps:

[0037] I, output offset through Faster R-CNN network model Construct the initial regression bounding box, as shown in the following expression:

[0038]

[0039] In the formula, and The center coordinates of the target suggestion box; and Provide the width and height of the target box; and This represents the offset of the center point predicted by the regression network. and This is the scaling factor for the size prediction of the regression network; and To predict the center coordinates of the bounding box; and To predict the width and height of the bounding box;

[0040] The location loss matrix of the scene target is constructed using the following expression:

[0041]

[0042] The formula for calculating the positional loss between the initial regression bounding box and the historical bounding box of the scene target is:

[0043]

[0044] In the formula, For the elements of the position loss matrix; For historical target indexing, ; For indexing regression results, ; For historical context; This is the initial bounding box for regression. For intersection, union, and comparison.

[0045] Furthermore,

[0046] Solve for the optimal match of this loss matrix, and output two sets of regression positions of the scene target after identity information correction, including:

[0047] The first set of regression positions is expressed as follows:

[0048]

[0049]

[0050] In the formula, This is the set of positions predicted for frame t+1 based on frame t data; The position of target k in frame t+1 is predicted based on frame t data; and The coordinates of the predicted center point of target k based on frame t data; and The predicted bounding box width and height of target k are given based on the data of frame t.

[0051] The second set of regression positions is expressed as follows:

[0052]

[0053]

[0054] In the formula, This is the set of positions predicted for frame t+1 based on frame t-1 data; The position of target k in frame t+1 is predicted based on frame t-1 data; and The coordinates of the predicted center point of target k based on frame t-1 data; and The width and height of the predicted bounding box for target k based on frame t-1 data.

[0055] Furthermore,

[0056] The spatiotemporal correlation between the two trajectory pipelines is calculated using the following formula:

[0057]

[0058] In the formula, The correlation between trajectory pipe i and trajectory pipe j in the current frame; Let i be the predicted position of trajectory pipe i in the current frame; Let j be the predicted position of the trajectory pipe in the current frame; The historical correlation between trajectory pipe i and trajectory pipe j in the previous frame; The correlation threshold is 0.5 to 0.7, with 0.6 being preferred here; t is the previous frame, and t+1 is the current frame.

[0059] Furthermore,

[0060] The positional error between the two trajectory pipes is calculated using the following formula:

[0061]

[0062] In the formula, This represents the absolute displacement error of the center point of the candidate trajectory pipe bounding box in the x-direction between two frames. This represents the absolute displacement error of the center point of the candidate trajectory pipe bounding box in the y direction between two frames. The height of the bounding box of the previous frame (frame t); The width of the bounding box of the previous frame (frame t);

[0063] The shape error of the two trajectory pipelines is calculated using the following formula:

[0064]

[0065] In the formula, The shape error of the candidate trajectory pipeline formed by the two bounding boxes in frame t and frame t+1; This represents the absolute error in the height of the candidate trajectory pipeline between two frames; This represents the absolute error in the width of the candidate trajectory pipeline between two frames. The height of the bounding box of the previous frame (frame t); The width of the bounding box of the previous frame (frame t).

[0066] Furthermore,

[0067] The quality score of the trajectory pipeline is calculated using the following formula:

[0068]

[0069] In the formula, The quality score for the trajectory pipeline; The raw classification confidence score output by the target detector; and The standard deviation of the Gaussian function; This refers to the positional error of the trajectory pipeline; This refers to the shape error of the trajectory pipeline.

[0070] Furthermore,

[0071] The tracking method further includes the following steps:

[0072] Filter out a set of detection information and trajectory pipelines that are spatially adjacent and need to be matched as candidate targets;

[0073] The algorithm calculates the size loss, appearance loss, and orientation loss of candidate targets, and then sums the appearance loss and orientation loss by weight. It constructs an association loss function and finds the minimum value of the association loss function. Finally, it outputs the optimal identity matching relationship and completes the identity reassignment of candidate targets.

[0074] Furthermore,

[0075] A set of spatially adjacent detection information and trajectory pipelines is defined as a detection-trajectory pair. The spatial proximity of the trajectory-detection pair is evaluated by the following expression:

[0076]

[0077] In the formula: The historical bounding box of the i-th trajectory in frame t; The bounding box of the j-th detection in frame t+1; This represents the area of ​​the intersection of the two bounding boxes. The area of ​​the union of the two bounding boxes;

[0078] Set a filtering threshold, and define spatial proximity when IoU > the filtering threshold.

[0079] Furthermore,

[0080] The following method is used to filter spatially adjacent trajectory-detection pairs:

[0081] Input trajectory set and detection set The IoU matrix is ​​calculated using the following formula:

[0082] ;

[0083] Through formula Filter candidate trajectory-detection pairs and output a set of candidate targets;

[0084] In the formula, For the set of candidate targets; For the first A single-track pipeline; For the first One detection box; For trajectory pipeline With detection box The intersection-union ratio (IoU) value; This is the filtering threshold.

[0085] Furthermore,

[0086] The appearance loss of the candidate target is calculated using the following formula:

[0087]

[0088] In the formula, Appearance loss for candidate targets; The size loss of the candidate target; The position loss of the candidate target;

[0089] The formula for calculating the size loss of the candidate target is as follows:

[0090]

[0091] In the formula, The height of the historical bounding box of the candidate target; The height of the current bounding box of the candidate target; The width of the historical bounding box of the candidate target; The width of the current bounding box of the candidate target;

[0092] The formula for calculating the position loss of the candidate target is as follows:

[0093]

[0094] In the formula, This represents the absolute value of the displacement in the x-direction between the center point of the historical bounding box and the current bounding box of the candidate target. This represents the absolute value of the displacement in the y-direction between the center point of the historical bounding box and the current bounding box of the candidate target.

[0095] The directional loss of the candidate target is calculated using the following formula:

[0096]

[0097] In the formula, Directional loss for candidate targets; and These are the motion direction vectors of the candidate target in two consecutive time steps.

[0098] Furthermore,

[0099] The appearance loss and orientation loss of the candidate targets are weighted and summed to construct the final association loss function, which is expressed as follows: In the formula, The weights are used to calculate the minimum value of the association loss function, output the optimal identity matching relationship, complete the identity reassignment of candidate targets, and initialize the unmatched detection boxes as new trajectory pipelines and assign them a new and unique identity identifier.

[0100] Based on the above technical solution, the present invention has the following beneficial effects:

[0101] 1. The method described in this invention, by creating a trajectory pipeline data structure, can structurally store and manage the long-term historical information of the target, including bounding box sequences, appearance feature sequences, timestamps, etc., compared with the single-frame or short-time sequence methods in the prior art, thus providing rich spatiotemporal context for robust tracking in complex scenarios.

[0102] 2. The method described in this invention is based on a trajectory pipeline and uses two-frame independent regression at different historical moments to predict the current position of the target; it corrects the integration result and corrects the identity by matching the position loss; and it actively recovers the occluded / interfered potential target based on the difference in regression confidence, which significantly improves the long-term occlusion recovery capability.

[0103] 3. The method described in this invention addresses the conflict caused by trajectory overlap of the tracked target by using trajectory pipeline nonmaximum suppression, moving the problem from the "detection box level" to the "trajectory pipeline level". By calculating the spatiotemporal correlation between trajectories (i.e., fusing the current predicted position IoU with historical correlation) and combining position / shape errors to re-score trajectory quality (i.e., Gaussian penalty), low-quality conflicting trajectories are suppressed at the trajectory level, effectively resolving the identity switching problem caused by spatial overlap.

[0104] 4. The method described in this invention, in identity association, not only calculates appearance loss but also introduces orientation loss, i.e., cosine similarity based on continuous motion vectors. By using a weighted joint loss function, it weakens the dependence on volatile appearances, strengthens the smoothness and consistency of motion patterns, and improves tracking stability in scenarios with changes in lighting and blur.

[0105] 5. The method described in this invention, using the object detector Faster R-CNN and the appearance feature extractor ResNet-50, can perform structured pruning, removing redundant or low-contribution convolutional channels and network layers. For example, channel pruning can be performed on some convolutional layers of the feature extractor, with a pruning rate of approximately 40%, focusing on retaining activation channels that contribute highly to identity recognition. This significantly reduces the computational load and parameter count of the model with only a slight impact on accuracy. Computational resources are dynamically allocated based on the current state of each trajectory pipeline to achieve more efficient system operation, and different processing priorities can be set.

[0106] 6. The following experimental data further illustrates the beneficial effects of the method described in this invention: The following is an explanation or description of the algorithms appearing in the experimental data: (1) TMOH (Tracking by Motion-Oriented Hypotheses): This is a tracking method based on motion-oriented hypotheses. It is an online multi-target tracking algorithm that emphasizes the continuity and consistency of the target's motion trajectory. It generates candidate trajectories and performs optimized matching based on the motion consistency hypothesis.

[0107] (2) ExDark (Exclusively Dark Dataset): ExDark is a dataset specifically designed for the study of target detection and recognition in low-light (dark) images. It contains 12 classes and more than 7,000 images and is suitable for research on nighttime target detection and visual enhancement.

[0108] (3) TracktorV2 (Tracking without Bells and Whistles (V2), or TracktorVersion 2 for short), which is a regression tracker without additional components (second version). TracktorV2 is an improved version of Tracktor. It improves tracking performance by reusing the regression branch of the target detector for position prediction and combining appearance model with camera motion compensation strategy.

[0109] (4) TADAM (Task Dependent Adaptive Metric): Originally a metric learning framework for few-shot learning, TADAM has been used in some multi-object tracking studies for distance metrics for target feature matching and re-identification.

[0110] Compared to the commonly used algorithms mentioned above, the method described in this invention has the following advantages: (1) Significantly enhances tracking robustness: By constructing a “trajectory pipeline” to perform spatiotemporal continuous modeling of the target, a long-term occlusion recovery rate of approximately 83.6% is achieved in the MOT20 high occlusion scenario, which is about 12.1% higher than that of TMOH.

[0111] (2) In low-light scenarios, such as the ExDark dataset, the false negative rate is reduced to about 23.1%, which is about 12.8% lower than TMOH, effectively overcoming the interference of sudden changes in light.

[0112] (3) Technical attribution: trajectory relocation cross-frame regression of potential targets, and dynamic adjustment of orientation loss weights to weaken appearance dependence.

[0113] (4) Significantly reduced identity switching error rate: In the MOT17 test, the number of ID switching times was reduced to 402, which is about 53.5% lower than TracktorV2 and about 42.2% lower than TMOH.

[0114] (5) The number of vehicle cross-scenario ID switching times is as low as about 0.3 times / vehicle, which is about 62.5% lower than TMOH.

[0115] (6) Technical attribution: T-NMS module trajectory-level conflict suppression, T-RA module directional loss constraint motion consistency.

[0116] (7) Achieving a breakthrough in lightweight model performance: Under a strictly consistent 6.2M parameter quantity, the inference efficiency is approximately 22ms / frame, which is about 50% faster than TMOH, meeting the real-time requirements of in-vehicle systems. Its performance comprehensively surpasses mainstream solutions, as shown in Tables 1 and 2 below: Table 1 Performance Comparison of Multi-Target Tracking Algorithms

[0117]

[0118] Table 2 Performance Verification in Extreme Scenarios

[0119] Attached Figure Description

[0120] Figure 1 This is a schematic diagram of the system structure of the multi-target tracking method described in this invention;

[0121] Figure 2 This is a schematic diagram of the trajectory pipeline data structure;

[0122] Figure 3 This is a schematic diagram of the T-NMS module's workflow. Detailed Implementation

[0123] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0124] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains.

[0125] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0126] In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary and not as a limitation; therefore, other examples of exemplary embodiments may have different values.

[0127] The trajectory pipeline described in this specification and claims is a time-series data structure, which includes at least: a globally unique target identifier (ID), a list of bounding box coordinates arranged in chronological order, a list of appearance feature vectors corresponding to each bounding box, and a list of timestamps.

[0128] Faster R-CNN Algorithm Flow: ① Convolutional layers (conv layers): This is the feature extraction network used to extract features. A set of conv+relu+pooling layers extracts feature maps from the image, which are then used in subsequent RPN layers and to generate proposals. ② Region Proposal Network (RPN): This generates candidate boxes. It includes two tasks: first, determining whether all preset anchors are positive or negative; second, refining the anchors to obtain more accurate proposals. Therefore, the RPN is used to determine the presence of targets and to refine the anchors. ③ RoI Pooling: This collects the proposals generated by the RPN and extracts them from the feature maps in the conv layers, generating proposal feature maps which are then fed into subsequent fully connected layers for classification and regression. ④ Classification and Regression: This calculates the specific category using the proposal feature maps and performs bounding box regression to obtain the final precise location of the detection box.

[0129] like Figures 1 to 3 As shown in the figure, this embodiment describes a multi-target tracking method based on trajectory pipeline regression prediction and correlation, which specifically includes the following steps.

[0130] S1: Acquire and identify existing and new targets in the application scenario, match and splice existing targets with their corresponding trajectory pipelines, and output the updated trajectory pipelines of existing targets; construct trajectory pipelines corresponding to identity identifiers for new targets, and output the trajectory pipelines of new targets.

[0131] Specifically, the following steps are included:

[0132] S101 uses video recording devices such as monocular cameras and depth cameras to capture application scenes in real time / continuously, and uses video decoders (such as FFmpeg) to split the real-time video of the application scene into continuous image frames.

[0133] Each frame of image is resized to a fixed size (e.g., 640×480) using bilinear interpolation and pixel values ​​are normalized ([0,255]→[0,1]). This normalized pixel value is then input into a trained Faster R-CNN object detector network model for forward propagation, identifying all scene targets in the application scenario, including both existing and new targets. All scene targets are uniquely identified using a sequential serial number, e.g., a......i......aa......aaa....... New targets are assigned serial numbers following the order of existing targets.

[0134] S102 outputs scene target detection information through the Faster R-CNN network model, including detection boxes and raw classification confidence scores. The information of the detection boxes includes bounding box coordinates and appearance feature vectors.

[0135] S103: Match the detection information of scene targets with the existing trajectory pipeline. For existing targets that are successfully matched, append the bounding box coordinates and appearance feature vector of the existing target to the end of its historical trajectory pipeline data structure and output the updated trajectory pipeline of the existing target. For new targets that are not successfully matched, initialize a new trajectory pipeline and output the trajectory pipeline of the new target.

[0136] In this step, a successful match means that the joint matching cost C between the detection box and the trajectory pipeline is less than θ (θ=1.5), where: C= α·Mahanobis distance + (1-α)·appearance cosine distance (α=0.6), Mahanobis distance threshold: 9.0 (95% confidence of chi-square distribution), appearance similarity threshold: 0.7 (cosine distance of ResNet-50 features).

[0137] Through the above steps, a trajectory pipeline with a unique identity can be constructed for each scene target in the application scenario. With the real-time dynamic update of the captured video, a structured trajectory pipeline with long-term historical information of each scene target can be constructed.

[0138] As a preferred option:

[0139] The detection information of scene targets is matched with the existing trajectory pipeline using the following method:

[0140] S103-1 uses a Kalman filter to predict the prior position of the scene target in the current frame based on the posterior state of the scene target in the previous moment, and outputs the predicted position of the scene target's trajectory pipeline.

[0141] S103-2 uses an LSTM network to obtain the expected current appearance of scene targets based on the historical feature sequence of scene targets, and outputs the detection information of scene targets.

[0142] S103-3 employs a hierarchical matching strategy for the initial matching of detection information and trajectory pipelines. For ease of description, a set of detection information and trajectory pipelines to be matched is defined as a "detection-trajectory pair." Specifically:

[0143] Calculate the Mahalanobis distance between the detection information of each scene target and the predicted position of the trajectory pipeline, and use this Mahalanobis distance as the motion cost;

[0144] For detection-trajectory pairs with motion costs below the correlation threshold (D_m = 9.0), proceed to the next matching layer. For the appearance matching layer: calculate the cosine similarity of the appearance features of the detection box and the trajectory; if it is greater than 0.7, output a matching pair; otherwise, terminate the matching and output the cost matrix.

[0145] S103-4, input the cost matrix into the Hungarian algorithm, and output the matching results of the scene target detection information and trajectory pipeline, including: successfully matched detection-trajectory pairs and unsuccessfully matched detection information. Among them, the unsuccessfully matched detection information includes "unmatched detection information" that failed to match any trajectory pipeline and / or "unmatched trajectory pipeline" that failed to match any detection information.

[0146] S103-5, Update the state of the trajectory pipeline of the scene target based on the matching results, including the internal state and lifecycle of the trajectory pipeline. The specific steps are as follows:

[0147] For a successfully matched detection-trajectory pair, the bounding box of the matched detection information is used as a new observation and input into the Kalman filter to correct and update its motion state; the appearance features of the detection information are input into the feature queue of the existing trajectory pipeline to update the appearance model of the existing trajectory pipeline, and the "miss counter" of the existing trajectory pipeline is reset to 0.

[0148] For trajectory pipelines that fail to match, they are set to be temporarily occluded, and their motion state is inferred only by the prediction step of the Kalman filter without being updated. The miss counter (c_miss) is incremented: c_miss = c_miss + 1. If the counter (c_miss) > 30 (corresponding to a 1-second duration threshold), the scene target that failed to match is defined as permanently leaving the application scene, the trajectory pipeline is marked as "terminated", and will be cleaned up later.

[0149] S2, based on the historical information contained in the trajectory pipeline output in step S1, including but not limited to historical bounding box sequences, appearance feature vectors, timestamps and spatiotemporal motion features, the position is corrected by regression prediction of trajectory pipeline relocalization (T-RL) and Hungarian matching algorithm, so as to accurately obtain the predicted position of the scene target in the current frame in the application scenario.

[0150] Specifically, the following steps are included:

[0151] S201, using the ResNet-50 backbone network to extract the first... Frame and the The image features of the frame are used to predict the preliminary regression results of the scene target at the current position from time t and time t-1, respectively, as shown in the following formula:

[0152]

[0153]

[0154] In the formula, and These are preliminary regression results; and For the collection of historical bounding boxes; and For the corresponding image features; This is a regression prediction network.

[0155] S202, Based on the positional loss between the initial regression bounding box and the historical bounding box of the scene target, construct the target's positional loss matrix. The specific steps are as follows:

[0156] S202-1, outputting offsets through the Faster R-CNN network model. Construct the initial regression bounding box, as shown in the following expression:

[0157]

[0158] In the formula, and The center coordinates of the target suggestion box; and Provide the width and height of the target box; and This represents the offset of the center point predicted by the regression network. and This is the scaling factor for the size prediction of the regression network; and To predict the center coordinates of the bounding box; and To predict the width and height of the bounding box.

[0159] S202-2, the target's position loss matrix is ​​constructed using the following expression:

[0160]

[0161] The formula for calculating the positional loss between the initial regression bounding box and the historical bounding box of the target is:

[0162]

[0163] In the formula, For the elements of the position loss matrix; For historical target indexing, ; For indexing regression results, ; For historical context; This is the initial bounding box for regression. For intersection, union, and comparison.

[0164] S202-3 uses the Hungarian algorithm to find the optimal match of the loss matrix, obtaining two sets of regression positions for the scene target after identity information correction, including:

[0165] The first set of regression positions (based on frame t) is expressed as follows:

[0166]

[0167]

[0168] In the formula, This is the set of positions predicted for frame t+1 based on frame t data; The position of target k in frame t+1 is predicted based on frame t data; and The coordinates of the predicted center point of target k based on frame t data; and The predicted bounding box width and height of target k are given based on the data of frame t.

[0169] The second set of regression positions (based on frame t-1) is expressed as follows:

[0170]

[0171]

[0172] In the formula, This is the set of positions predicted for frame t+1 based on frame t-1 data; The position of target k in frame t+1 is predicted based on frame t-1 data; and The coordinates of the predicted center point of target k based on frame t-1 data; and The width and height of the predicted bounding box for target k based on frame t-1 data.

[0173] S203, for scene targets with the same identity identifier, the intersection-union ratio between the two sets of regression positions after correction is used as the confidence level of the scene target's regression prediction in this instance, and the confidence level of the scene target's regression prediction is compared with a preset threshold η (wherein, the preset threshold η is determined through verification using historical tracking data, and the preferred range is 0.8~0.95, and in this embodiment, η=0.9 is used):

[0174] If the confidence level is greater than or equal to the preset threshold, it means that the two sets of regression location prediction results of the target in the scene have high similarity and high credibility. One of them is randomly selected as the final regression bounding box of the target's predicted location in the scene.

[0175] If the confidence level is less than a preset threshold, the regression position with higher confidence among the two sets of regression positions of the target in the scene is taken as the final regression bounding box, and the regression position with lower confidence is defined as a potential target recovered due to occlusion or interference, thus constructing a set of potential targets. The potential target set is then incorporated into the original target detection set of the current frame to construct an enhanced detection set.

[0176] Construct the potential target set using the following expression:

[0177]

[0178] In the formula, Let be the set of potential targets in frame t+1; This represents the k-th low-confidence prediction location (k=1,2,…); and There are two sets of regression positions (i,j are the target indices); η is the preset threshold.

[0179] S3. Since the motion trajectories of scene targets are usually overlapping, existing technologies resolve conflicts caused by spatial overlap between motion trajectories at the bounding box level. However, this can lead to the erroneous deletion of targets when they occlude each other, resulting in target loss and trajectory breakage. To effectively solve this technical problem, this step re-scores the trajectory quality of the target by combining the correlation calculation method between the current frame and historical frame information. This resolves the conflicts between trajectory pipelines at the trajectory pipeline level. Compared with existing technologies, this has the beneficial effects of improved anti-occlusion (missed detection rate ↓18.3%), enhanced trajectory continuity (long trajectory proportion ↑5.5%), and optimized real-time performance (processing speed ↑71%).

[0180] Specifically, the following steps are included:

[0181] S301, randomly select any two trajectory pipelines to form a trajectory pipeline pair, such as the trajectory pipeline with identity identifier i and the trajectory pipeline with identity identifier j. Calculate the spatiotemporal correlation between the two by fusing their predicted position intersection-union ratio in the current frame with their historical correlation in the previous frame. The calculation formula is as follows:

[0182]

[0183] In the formula, The correlation between trajectory pipe i and trajectory pipe j in the current frame; Let i be the predicted position of trajectory pipe i in the current frame; Let j be the predicted position of the trajectory pipe in the current frame; The historical correlation between trajectory pipe i and trajectory pipe j in the previous frame; The correlation threshold is 0.5 to 0.7, with 0.6 being preferred here; t is the previous frame, and t+1 is the current frame.

[0184] S302, for the trajectory pipe pair selected in step S301 (i.e., pipes i and j, which can also be described as candidate trajectory pipes), calculate their position error and shape error respectively:

[0185] The position error of the candidate trajectory pipeline formed by the two bounding boxes in frame t and frame t+1 is calculated using the following formula:

[0186]

[0187]

[0188]

[0189] In the formula, This represents the absolute displacement error of the center point of the candidate trajectory pipe bounding box in the x-direction between two frames. This represents the absolute displacement error of the center point of the candidate trajectory pipe bounding box in the y direction between two frames. The height of the bounding box of the previous frame (frame t); The width of the bounding box of the previous frame (frame t);

[0190] The shape error of the candidate trajectory pipeline formed by the two bounding boxes in frame t and frame t+1 is calculated using the following method:

[0191]

[0192]

[0193]

[0194] In the formula, The shape error of the candidate trajectory pipeline formed by the two bounding boxes in frame t and frame t+1; This represents the absolute error in the height of the candidate trajectory pipeline between two frames; This represents the absolute error in the width of the candidate trajectory pipeline between two frames. The height of the bounding box of the previous frame (frame t); The width of the bounding box of the previous frame (frame t).

[0195] S303, based on the position error and shape error of the trajectory pipeline obtained in step S302, a Gaussian penalty function is introduced to adjust the original classification confidence of the target detector Faster R-CNN network model output in step S1, and the trajectory pipeline is re-scored to obtain the final trajectory quality score of the trajectory pipeline.

[0196] The final trajectory quality score of the trajectory pipeline is calculated using the following formula:

[0197]

[0198] In the formula, This is the final trajectory quality score; The raw classification confidence score output by the target detector; and The standard deviation of the Gaussian function; This refers to the positional error of the trajectory pipeline; This refers to the shape error of the trajectory pipeline.

[0199] S304, based on the spatiotemporal correlation calculated in S301 and the final trajectory quality score calculated in S303 Suppression operations are performed on trajectory pipelines that have conflicts (i.e., spatiotemporal correlation is greater than the conflict threshold) and have low quality scores to preserve the optimal trajectory and eliminate redundancy. This outputs the identity of all active trajectory pipelines that have not been suppressed and their current frame prediction position, thus completing the tracking of all scene targets in the application scenario.

[0200] in,

[0201] Conflict detection: Spatiotemporal correlation Conflict threshold hour( (verified using the MOT17 dataset), determining trajectory pipeline conflict;

[0202] Suppression Decision: For each conflicting pipeline pair (e.g., pipelines i and j), the quality score ratio is calculated using the following formula:

[0203]

[0204] If ρ < 0.7 (this threshold was determined by grid search optimization on the MOT17 dataset, achieving the highest long trajectory retention rate on the validation set), then a suppression operation is performed, as follows:

[0205] ①Assign all conflicting pipeline pairs according to their spatiotemporal correlation. Sort from highest to lowest;

[0206] ② Iterate starting from the pipeline pair with the highest correlation: If the current pipeline pair satisfies ρ < 0.7, then permanently delete it. Lower pipe; otherwise, retain two pipes.

[0207] ③ Update the relationships of the remaining pipelines and remove the conflict pairs that have been resolved;

[0208] ④ Repeat the above process until all collision pairs are resolved, output the identity (ID) of all unsuppressed active trajectory pipes and their predicted positions in the current frame (frame t+1), and complete the tracking task for this frame.

[0209] S4. In order to effectively solve the problem of scene target trajectory loss during tracking, this embodiment performs trajectory pipeline reassignment by minimizing a joint loss function that combines appearance and motion direction, thereby realizing the reassignment of lost scene target trajectories and the identity of newly appearing targets.

[0210] Specifically, the following steps are included:

[0211] S401. For the detection-trajectory pairs that were successfully matched in step S1, due to the objective reason that the spatially spaced detection-trajectory pairs have motion continuity constraints (according to physical laws, the maximum displacement of the target between adjacent frames is limited by the frame rate) in the application scenario, in order to reduce unnecessary computational overhead, spatially adjacent trajectory-detection pairs are selected as candidate targets, and the candidate targets are redistributed for computation.

[0212] In this step,

[0213] ① Objectively measure spatial proximity using the intersection-over-union ratio (IoU):

[0214]

[0215] In the formula: The historical bounding box of the i-th trajectory in frame t; The bounding box of the j-th detection in frame t+1; This represents the area of ​​the intersection of the two bounding boxes. The area of ​​the union of the two bounding boxes.

[0216] When IoU > Filtering Threshold Time is considered spatially proximate. The value ranges from 0.3 to 0.5, with 0.4 being the preferred value (this value was verified and determined using the MOT17 dataset, achieving the best balance between computational efficiency and tracking accuracy).

[0217] ② Filter spatially adjacent trajectory-detection pairs using the following method:

[0218] Input trajectory set: (including history bounding box) and detection set: ;

[0219] The IoU matrix is ​​calculated using the following formula:

[0220] ;

[0221] Candidate pairs are selected using the following formula: In the formula, For the first A single-track pipeline; For the first One detection box; For trajectory pipeline With detection box The intersection-union ratio (IoU) value; This is the filtering threshold.

[0222] Output candidate set (Candidate target set).

[0223] S402, calculate the appearance loss of the candidate target using the following formula:

[0224]

[0225] In the formula, Appearance loss for candidate targets; The size loss of the candidate target; The position loss for the candidate target.

[0226] The formula for calculating the size loss of the candidate target is as follows:

[0227]

[0228] In the formula, The height of the historical bounding box of the candidate target; The height of the current bounding box of the candidate target; The width of the historical bounding box of the candidate target; The width of the current bounding box of the candidate target.

[0229] The formula for calculating the position loss of the candidate target is as follows:

[0230]

[0231] In the formula,

[0232] This represents the absolute value of the displacement in the x-direction between the center point of the historical bounding box and the current bounding box of the candidate target.

[0233] This represents the absolute displacement of the center point of the candidate target's historical bounding box and the current bounding box in the y-direction.

[0234] S403, the orientation loss of the candidate target is calculated using the following formula, that is, the smoothness of the orientation change is measured by the cosine similarity between motion vectors:

[0235]

[0236] In the formula, Directional loss for candidate targets; and These are the motion direction vectors of the candidate target in two consecutive time steps.

[0237] S404, the appearance loss and orientation loss of the candidate targets are weighted and summed to construct the final association loss function, expressed as follows: ,in The value is a weight, ranging from 0.3 to 1.2, with 0.9 being the preferred value. In actual operation, it can be adjusted according to the actual scenario.

[0238] The Hungarian algorithm is used to find the minimum value of the association loss function, output the optimal identity matching relationship, and complete the identity reassignment of candidate targets.

[0239] S405. For detection boxes that still fail to match after the above steps, initialize them as new trajectory pipelines and assign them a new, unique identifier.

[0240] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A multi-target tracking method based on trajectory pipeline regression prediction and correlation, characterized in that, Includes the following steps: S1, acquire and identify scene targets, including existing targets and new targets, and output the trajectory pipeline of scene targets, including: matching and splicing the trajectory pipelines corresponding to the existing targets and their identity identifiers, and outputting the updated trajectory pipeline of the existing targets; constructing the trajectory pipeline corresponding to the identity identifier of the new targets, and outputting the trajectory pipeline of the new targets; S2, corrects the position of the scene target's trajectory pipeline and outputs the predicted position of the scene target in the current frame; S3, select any two trajectory pipelines of scene targets and calculate the spatiotemporal correlation, position error and shape error between them, score the quality of the two trajectory pipelines respectively, and perform suppression operation on the trajectory pipelines that conflict and have low quality scores. S4. Perform step S3 on the trajectory pipelines of all scene targets, output the identity of all unsuppressed trajectory pipelines and their current frame prediction position, and complete the tracking of scene targets.

2. The multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 1, characterized in that, The Faster R-CNN network model outputs scene target detection information, including detection boxes and raw classification confidence scores. The detection box information includes bounding box coordinates and appearance feature vectors. Calculate the Mahalanobis distance between the detection information of each existing and new target and the predicted position of the trajectory pipeline, and use this Mahalanobis distance as the motion cost; Define a set of detection information and trajectory pipelines that need to be matched as a detection-trajectory pair, set an association threshold, and proceed to the next layer of matching for detection-trajectory pairs whose motion cost is lower than the association threshold; otherwise, terminate the matching and output the cost matrix. Input the cost matrix into the Hungarian algorithm and output the matching results of detection information and trajectory pipelines, including successfully matched detection-trajectory pairs and unsuccessfully matched detection information. The unsuccessfully matched detection information includes unmatched detection information that failed to match any trajectory pipeline and / or unmatched trajectory pipelines that failed to match any detection information. For a successfully matched detection-trajectory pair, the bounding box coordinates of the matching detection information are used as new observations and input into the Kalman filter for motion state correction and update. Trajectory pipelines that do not match are not updated.

3. The multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 1, characterized in that, The following steps will output the predicted position of the scene target in the current frame: Extract the first line from the trajectory pipeline Frame and the The image features of the frame are used to predict the preliminary regression results of the scene target at the current position from time t and time t-1, respectively; Based on the position loss between the initial regression bounding box and the historical bounding box of the scene target, a position loss matrix of the scene target is constructed; For scene targets with the same identity identifier, the intersection-union ratio between the two sets of regression positions after correction is used as the confidence level of the scene target's regression prediction, and the confidence level of the scene target's regression prediction is compared with a preset threshold: If the confidence level is greater than or equal to the preset threshold, one of them is randomly selected as the final regression bounding box for the predicted location of the target in the scene; If the confidence level is less than the preset threshold, the regression position with higher confidence level among the two sets of regression positions of the target in the scene is used as the final regression bounding box, and the regression position with lower confidence level is defined as the potential target, thus constructing a set of potential targets.

4. The multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 3, characterized in that, The following expression is used to predict the preliminary regression result of the scene target at the current position; ， ， In the formula, and These are preliminary regression results; and For the collection of historical bounding boxes; and For the corresponding image features; For regression prediction networks; Construct the potential target set using the following expression: ， In the formula, Let be the set of potential targets in frame t+1; This is the k-th low-confidence predicted location; and There are two sets of regression positions, where i and j are the target indices; The preset threshold; Construct the location loss matrix of the scene target using the following steps: The offset is output through the Faster R-CNN network model. Construct the initial regression bounding box, as shown in the following expression: ， In the formula, and The center coordinates of the target suggestion box; and Provide the width and height of the target box; and This represents the offset of the center point predicted by the regression network. and This is the scaling factor for the size prediction of the regression network; and To predict the center coordinates of the bounding box; and To predict the width and height of the bounding box; The location loss matrix of the scene target is constructed using the following expression: ， The formula for calculating the positional loss between the initial regression bounding box and the historical bounding box of the scene target is: ， In the formula, For the elements of the position loss matrix; For historical target indexing, ; For indexing regression results, ; For historical context; This is the bounding box for the initial regression; For intersection, union, and comparison; Solve for the optimal matching of the loss matrix, and output two sets of regression positions of the scene target after identity information correction, including: The first set of regression positions is expressed as follows: ， ， In the formula, The set of positions predicted for frame t+1 based on frame t data; The position of target k in frame t+1 is predicted based on frame t data; and The coordinates of the predicted center point of target k based on frame t data; and The predicted bounding box width and height of target k are given based on the data of frame t. The second set of regression positions is expressed as follows: ， ， In the formula, This is the set of positions predicted for frame t+1 based on frame t-1 data; The position of target k in frame t+1 is predicted based on frame t-1 data; and The coordinates of the predicted center point of target k based on frame t-1 data; and The width and height of the predicted bounding box for target k based on frame t-1 data.

5. A multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 1, characterized in that, The spatiotemporal correlation between the two trajectory pipelines is calculated using the following formula: ， In the formula, The correlation between trajectory pipe i and trajectory pipe j in the current frame; Let i be the predicted position of trajectory pipe i in the current frame; Let j be the predicted position of the trajectory pipe in the current frame; The historical correlation between trajectory pipe i and trajectory pipe j in the previous frame; The correlation threshold is 0.5 to 0.7; t is the previous frame, and t+1 is the current frame.

6. The multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 1, characterized in that, The positional error between the two trajectory pipes is calculated using the following formula: ， In the formula, This represents the absolute displacement error of the center point of the candidate trajectory pipe bounding box in the x-direction between two frames. This represents the absolute displacement error of the center point of the candidate trajectory pipe bounding box in the y direction between two frames. The height of the bounding box of the previous frame (frame t); The width of the bounding box of the previous frame (frame t); The shape error of the two trajectory pipelines is calculated using the following formula: ， In the formula, The shape error of the candidate trajectory pipeline formed by the two bounding boxes in frame t and frame t+1; This represents the absolute error in the height of the candidate trajectory pipeline between two frames; This represents the absolute error in the width of the candidate trajectory pipeline between two frames. The height of the bounding box of the previous frame (frame t); The width of the bounding box of the previous frame (frame t).

7. A multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 1, characterized in that, The quality score of the trajectory pipeline is calculated using the following formula: ， In the formula, The quality score for the trajectory pipeline; The raw classification confidence score output by the target detector; and The standard deviation of the Gaussian function; This refers to the positional error of the trajectory pipeline; This refers to the shape error of the trajectory pipeline.

8. A multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 1, characterized in that, The tracking method further includes the following steps: Filter out a set of detection information and trajectory pipelines that are spatially adjacent and need to be matched as candidate targets; The algorithm calculates the size loss, appearance loss, and orientation loss of candidate targets, and then sums the appearance loss and orientation loss by weight; it constructs the association loss function and finds the minimum value of the association loss function; it outputs the optimal identity matching relationship and completes the identity reassignment of candidate targets. A spatially adjacent set of detection information and trajectory pipelines is defined as a detection-trajectory pair. The spatial proximity of the trajectory-detection pair is evaluated by the following expression: ， In the formula: The historical bounding box of the i-th trajectory in frame t; The bounding box of the j-th detection in frame t+1; This represents the area of ​​the intersection of the two bounding boxes. The area of ​​the union of the two bounding boxes; Set a filtering threshold, and define spatial proximity when IoU > the filtering threshold; The following method is used to filter spatially adjacent trajectory-detection pairs: Input trajectory set and detection set The IoU matrix is ​​calculated using the following formula: ， Through formula Filter candidate trajectory-detection pairs and output a set of candidate targets; In the formula, For the set of candidate targets; For the first A single-track pipeline; For the first One detection box; For trajectory pipeline With detection box The intersection-union ratio (IoU) value; This is the filtering threshold.

9. A multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 8, characterized in that, The appearance loss of the candidate target is calculated using the following formula: ， In the formula, Appearance loss for candidate targets; The size loss of the candidate target; The position loss of the candidate target; The formula for calculating the size loss of the candidate target is as follows: ， In the formula, The height of the historical bounding box of the candidate target; The height of the current bounding box of the candidate target; The width of the historical bounding box of the candidate target; The width of the current bounding box of the candidate target; The formula for calculating the position loss of the candidate target is as follows: ， In the formula, This represents the absolute value of the displacement in the x-direction between the center point of the historical bounding box and the current bounding box of the candidate target. This represents the absolute value of the displacement in the y-direction between the center point of the historical bounding box and the current bounding box of the candidate target. The directional loss of the candidate target is calculated using the following formula: ， In the formula, Directional loss for candidate targets; and These are the motion direction vectors of the candidate target in two consecutive time steps.

10. A multi-target tracking method based on trajectory pipeline regression prediction and correlation according to claim 9, characterized in that, The appearance loss and orientation loss of the candidate targets are weighted and summed to construct the final association loss function, which is expressed as follows: In the formula, The weights are used to calculate the minimum value of the association loss function, output the optimal identity matching relationship, complete the identity reassignment of candidate targets, and initialize the unmatched detection boxes as new trajectory pipelines and assign them a new and unique identity identifier.

Images