Multi-target tracking method and system based on self-adaption during testing

Abstract

The invention discloses a multi-target tracking method and system based on self-adaption during testing, and belongs to the field of multi-target tracking. The method comprises the following steps: constructing different types of enhanced views for an input video frame image; a teacher-twin student model double-branch module is established, a spatial entropy method and object-level KL divergence calculation are adopted to obtain a spatial entropy and object-level uncertainty corresponding to a student model, dynamic threshold determination of different layers of the student model is realized based on the object-level uncertainty, and then learning rate dynamic allocation is realized. The enhanced detection consistency loss between the enhanced views is obtained through calculation according to the spatial entropy, the enhanced detection consistency loss and the local patch comparison loss between the enhanced views are added, and a total loss function is obtained; and optimizing the student model according to the total loss function of all the images in the video frame sequence so as to reversely update the teacher model. According to the method, tracking precision, time consistency and calculation efficiency are effectively balanced, and compared with an existing method, all aspects are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of multi-target tracking, and more particularly to a test-time adaptive multi-target tracking method and system. Background Technology

[0002] Multi-target tracking is a core capability in dynamic environment perception systems such as autonomous driving and drone monitoring, and its importance is increasingly evident in complex real-world scenarios. Although existing trackers (such as Quasi-Dense Tracker and MotionTrack) perform excellently in controlled environments, their performance degrades significantly under real-domain offset conditions such as sudden changes in lighting, severe weather, or unknown camera perspectives. The root of this problem lies in the fact that traditional methods rely on a strong assumption: the training and testing domains are statistically identical.

[0003] Existing research on domain adaptation mainly focuses on test-time adaptation techniques. For example, TENT adjusts the classification model through entropy minimization, LAME uses parameterized posterior adaptation, and EATA emphasizes efficient entropy minimization. However, these methods do not fully consider the dual requirements of detection consistency and trajectory coherence in multi-object tracking tasks. While PALM proposes an adaptive learning rate mechanism for persistent domain shifts, its learning rate adjustment strategy ignores the spatiotemporal dependencies unique to multi-object tracking. TTA methods specifically for multi-object tracking (such as DARTH) introduce a teacher-student framework based on detection consistency. However, existing methods have not yet achieved joint optimization of detection, feature extraction, and data association under domain shifts, and suffer from excessive computational overhead. Summary of the Invention

[0004] To address the aforementioned issues, this invention establishes a highly efficient and robust test-time adaptive multi-target tracking unified framework, DapaTTA, to solve the domain offset problem in the field of multi-target tracking.

[0005] In a first aspect, this invention proposes a test-time adaptive multi-target tracking method, comprising: S1, obtain the teacher view, student view and comparison view based on the input video frame image; S2, using pre-trained QDTrack, construct a teacher model for processing the teacher view and a twin student model for processing the student view and the comparison view, respectively. The three models output a first detection box, a first bounding box and a classification probability map for the student view, a second detection box and a second bounding box for the teacher view, and a third detection box for the comparison view, respectively; the classification probability map contains the probability that each first bounding box belongs to a different category. S3, based on the first bounding box and the classification probability map, the spatial entropy value of each pixel in the student view and the object-level uncertainty value of the student view are calculated by the spatial entropy method and the object-level KL divergence, respectively. S4. Calculate the KL divergence loss function for each layer of the student model based on the student view, and further calculate the sensitivity of each layer. The dynamic threshold is calculated using the object-level uncertainty value. The activation state of each layer of the student model is determined based on the dynamic threshold and the binary decision function. If the sensitivity of the layer meets the judgment condition, the layer is activated. The learning rate is then assigned to the activated layer according to the domain offset factor. Otherwise, the learning rate of the corresponding layer is 0. The domain offset factor is calculated based on the spatial entropy value of all pixels in the foreground region of the video frame image. S5: Calculate the local patch contrast loss based on the output results of the teacher view, student view, and contrast view in the corresponding models; calculate the enhancement detection consistency loss based on the first bounding box and the second bounding box; sum the enhancement detection consistency loss and the local patch contrast loss by weight to obtain the total loss function; traverse all video frame images and update the teacher model and twin student model. S6. Input the sequence of video frames to be tested into the trained teacher model, and associate the second bounding box of each frame output by the teacher model with the standard QDTrack association strategy to complete the multi-target tracking task of the sequence of video frames to be tested.

[0006] Furthermore, the teacher view, student view, and comparison view are specifically described as follows: A teacher's view is obtained by applying several geometric enhancement transformations to the video frame image; a student's view is obtained by applying several photometric enhancement transformations to the teacher's view; a comparison view is obtained by randomly applying several combined enhancement transformations to the video frame image; the combined enhancement transformation includes several geometric enhancement transformations and several photometric enhancement transformations, and the geometric transformation and photometric transformation appear mixed in the combined enhancement transformation.

[0007] Furthermore, the local patch contrast loss is: ; in, Embed feature vectors for the appearance of positive samples in the student view. The appearance embedding feature vector corresponding to the third detection box that forms a positive sample pair with the positive samples of the student view. For the appearance embedding feature vector corresponding to the third detection box that forms a negative sample pair with the positive samples in the student's view, exp() is an exponential function. The loss is a local patch contrast loss; the positive sample of the student view is the first detection box with an intersection-union ratio greater than 0.7 with the second detection box.

[0008] Furthermore, the updated teacher model and twin student model specifically include: The average total loss function is obtained by averaging the total loss functions of all video frames. The twin student model optimizes the student model parameters based on the average total loss function; The weights of the teacher model are updated in reverse using the parameters of the twin student model, while other parameters remain frozen.

[0009] Furthermore, the sensitivity of each layer is specifically calculated as follows: based on the KL divergence loss function, the gradient of the corresponding weight parameters of each layer is calculated in reverse, and the L2 norm is used to characterize the gradient of the weight parameters to obtain the sensitivity of each layer. The specific calculation formula for the KL divergence loss function is as follows: ; in, For the detection head of the student model, For the encoder of the student model, These are the student model parameters at the t-th iteration. This is the student view, where T is the temperature parameter. , where is the probability distribution of each object in the student model under ideal conditions; C is the total number of categories.

[0010] Furthermore, the spatial entropy value of each pixel in the student view and the object-level uncertainty value of the student view are specifically calculated using the following steps: Using the spatial entropy method, the spatial entropy value of each pixel in the first bounding box is calculated based on the probability of each first bounding box; Calculate the KL divergence between the probability of each first bounding box and the ideal uniform probability distribution. The formula is as follows: ; in, For the first The predicted probability that the first bounding box corresponding to each object belongs to category c. For the first The KL divergence of an object, where C is the total number of categories; The object-level uncertainty value of the student view is calculated by averaging the KL divergences of all first bounding boxes in the student view.

[0011] Furthermore, the binary decision function is: ; in, This represents the activation state of layer l. This is a dynamic threshold, calculated from the object-level uncertainty value of the student view. The sensitivity of the l-th layer.

[0012] Furthermore, in step S4, the learning rate is assigned to the activated layer according to the domain offset factor, and the specific calculation formula is as follows: ; in, For the first Learning rate of layer The set base learning rate, For the domain offset factor, This represents the number of layers between the current layer and the output layer. is the attenuation coefficient, and m is the total number of layers in the student model.

[0013] Furthermore, the formula for calculating the domain offset factor is as follows: ; in, The total number of pixels in the foreground area. Here, exp() is the sensitivity parameter, and exp() is the exponential function. For pixels Spatial entropy value, This is the domain offset factor.

[0014] Secondly, this invention proposes a test-time adaptive multi-target tracking system to implement the aforementioned test-time adaptive multi-target tracking method.

[0015] The beneficial effects of this invention are as follows: (1) By using dynamic layer selection guided by uncertainty and domain-sensitive learning rate optimization, the computational efficiency is greatly improved while ensuring performance improvement, thus meeting the real-time requirements of autonomous systems.

[0016] (2) Integrating multiple consistency constraints into a unified optimization objective enhances the model’s generalization ability and stability in unknown target domains and effectively alleviates performance degradation caused by domain differences.

[0017] (3) The method of the present invention constructs an efficient and robust test-time adaptive multi-target tracking unified framework, which includes uncertainty-guided dynamic layer selection, domain-sensitive learning rate optimization and multiple consistency formulas, effectively balancing tracking accuracy, time consistency and computational efficiency, and achieving significant improvements in all aspects compared with existing methods. Attached Figure Description

[0018] Figure 1 This is an overall framework diagram of the present invention.

[0019] Figure 2 This is a flowchart illustrating the specific process of constructing the enhanced view according to the present invention. Detailed Implementation

[0020] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0021] like Figure 1 As shown, this invention proposes a test-time adaptive multi-target tracking method, DapaTTA, which includes the following steps: Step 1: Construct multiple types of enhanced views The construction process of different types of augmented views, such as Figure 2 As shown, this invention employs a phased, multi-type enhancement strategy to generate three different enhancement views, each serving a specific learning objective.

[0022] For each frame of a consecutive video sequence, the original image x is subjected to a geometric enhancement transformation. (Including random scaling, cropping, and translation, etc.) to obtain the teacher's view. This view is primarily used to provide relatively stable detection targets.

[0023] In teacher view Based on this, further photometric enhancement transformation is applied. (Including brightness, contrast, color jitter, etc.), to obtain the student view. . from The design, which is derived from rather than independently generated from the original image, is intended to strictly satisfy the basic assumption of the detection consistency loss regarding the geometric alignment between the teacher and student views, ensuring that they are highly consistent in spatial location. This allows the local patch contrast loss calculated from the student view and the contrast view to effectively compare features from the same target in subsequent twin contrast learning tasks.

[0024] Contrast view applies a combined enhancement sequence to the original image x. (This sequence contains multiple geometric transformations and multiple photometric transformations simultaneously, with geometric transformations and photometric transformations appearing mixed in the sequence), resulting in a comparison view. This view, by introducing greater diversity, forces the model to learn feature representations that remain robust to various deformations and appearance changes.

[0025] Step 2: Establish a teacher-twin student network dual-branch module Specifically, it includes: 2.1) Construct the pre-trained student model and teacher model (in this specific embodiment, the teacher model is the appearance-based tracker QDTrack, and the student model is a distilled version of the teacher model; both the teacher and student models contain an encoder, a detector head, and an embedding head). In the subsequent training steps of this invention, the teacher model only dynamically updates its own weights based on the weights of the student model, while other parameters remain frozen.

[0026] The pre-trained teacher model parameters As the initial values ​​for the student model, based on the total loss function obtained in step five. The weights of the teacher model and the parameters of the student model are iteratively optimized; among them, the weights of the student model... Initialize to During each training session, the weights of the teacher model are... The update rule is as follows: The student model parameters are updated using an exponential moving average. ,in The momentum coefficient controls the degree to which historical parameter updates are retained. These are the student model parameters at the t-th iteration.

[0027] 2.2) A Siamese student network is constructed using two identical student models. By introducing local patch contrastive loss, the model learns feature representations with high discriminative power against appearance changes. Local patch contrastive loss. By comparing the detection region features in two enhanced views (i.e., the student view and the contrast view) of the same image, positive sample pairs are brought closer and negative sample pairs are pushed apart in the feature space.

[0028] Specifically, for a given original input image x, the generated teacher view, student view, and comparison view are first input into the teacher model. After passing through the encoder and detector head, multiple candidate boxes and corresponding detection confidence scores are obtained. Only detection confidence scores higher than a threshold are retained. The candidate boxes are selected and used as detection boxes for subsequent matching with the student view and the comparison view.

[0029] Secondly, the student view and the contrast view are respectively processed by the student model encoder and the detector head. The Region Proposal Network (RPN) in the detector head generates multiple candidate boxes (i.e., ROIs) from each level of the Feature Pyramid Network (FPN) of the student view and the contrast view. Since the candidate boxes initially generated by the RPN are of different sizes, they are adaptively assigned to the corresponding level of the FPN according to the scale of each candidate box, and then pooled from the feature map of the corresponding layer into input features of uniform size through ROI Align.

[0030] Finally, the student view candidate boxes and the comparison view candidate boxes, after being standardized in size, are matched sequentially with all detection boxes. If the intersection-over-union (IoU) between the candidate box and the detection box is greater than 0.7, the candidate box is considered a positive sample of the corresponding detection box; if the IoU is less than 0.3, the candidate box is considered a negative sample of the corresponding detection box. Specifically, if the candidate boxes of the student view and the comparison view are both associated with the same detection box, they are considered a positive sample pair; otherwise, they are considered a negative sample pair.

[0031] Assuming it is generated on the student view Using positive RoIs as training samples, a comparison view is proposed. One RoI was used as the comparison target. The calculation formula is:

[0032] in Embed feature vectors for the appearance of positive samples in the student view. and They are respectively with The appearance embedding feature vectors that constitute the comparison view candidate boxes of positive and negative sample pairs are exp(), which is an exponential function.

[0033] Step 3: Dynamic Layer Selection Based on Uncertainty Specifically, it includes: 3.1) To achieve a precise evaluation of the confidence level of model predictions, this invention adopts a dual uncertainty measurement method that combines spatial entropy and object-level KL divergence. These two measures provide complementary uncertainty information from local and global perspectives, respectively.

[0034] Spatial entropy is used to quantify the degree of predictive disorder of a model at each spatial location. For the input student view, after passing through the student model encoder, detector head, and embedding head, the final output is a dimensionless vector vector. The classification probability map and the bounding box regions corresponding one-to-one with all objects detected by the student model in the student view, where h and w are the height and width of the image, respectively. This represents the total number of categories. It's important to note that the candidate boxes used in step two are output from the student model's detection head and represent intermediate results; one object in the student view may correspond to multiple candidate boxes. The bounding boxes, on the other hand, are the final output of the student model, with one object in the student view corresponding to only one bounding box.

[0035] pixel Spatial entropy at the location The calculation formula is as follows:

[0036] in, For pixels The predicted probability of belonging to category c.

[0037] By traversing and calculating the spatial entropy of all pixels in the entire student view, a spatial entropy map can be obtained, which can intuitively visualize the image area where the model feels "confused". The higher the value, the better the model is at the pixel level. The more uniform the distribution of predictions at a given location, the lower the confidence level of the decision and the greater the uncertainty.

[0038] 3.2) Although spatial entropy can effectively reveal local uncertainty, considering that the target detection task is more concerned with the confidence level at the object level, this invention further introduces object-level KL divergence as a measure of global uncertainty.

[0039] For the student model detected the first For each object (such as pedestrians or vehicles in an image), classify it according to its bounding box and the probability distribution of each bounding box belonging to a different category. Through calculation With uniform distribution The KL divergence between the two can measure the student model's performance on the first... The degree to which the predicted distribution of an object deviates from a uniform distribution:

[0040] in, For the first The predicted probability that an object belongs to class c. For the first KL divergence of an object. The larger the value, the more concentrated the prediction distribution, the higher the confidence level of the model's prediction for that object, and the lower the uncertainty; conversely, the smaller the value, the greater the uncertainty.

[0041] Average object-level uncertainty of the entire image It can be calculated using the following formula:

[0042] in, This represents the total number of objects detected in the image.

[0043] As a scalar, it comprehensively reflects the model's overall confidence in predicting the current frame. This establishes a comprehensive uncertainty profile, providing a quantitative basis for the dynamic decision-making process in section 3.4.

[0044] 3.3) After obtaining the prediction results through forward propagation, firstly, the calculation is performed using a uniform distribution... The KL divergence loss function for the objective is calculated as follows:

[0045] in, For the detection head of the student model, For the encoder of the student model, Let T be the student model parameters at the t-th iteration, where T is the temperature parameter.

[0046] Subsequently, while maintaining the integrity of the computational graph, according to In reverse, layer-by-layer computation of the encoder and detector head within the student model, the first... Gradient of weight parameters corresponding to the layer The corresponding sensitivity is characterized by the L2 norm of the gradient of its weight parameters: Gradient norm The magnitude of the norm directly reflects the extent to which the weight parameters of the layer need to be updated to better fit the current data: the larger the norm, the more sensitive the layer is to the current inter-domain differences, and the higher the necessity for it to participate in adaptive updates; the smaller the norm, the more stable the feature representation of the layer is, and it can be temporarily frozen to save computation.

[0047] A significant advantage of this method is its computational efficiency. It only requires an additional recording of the gradient norm for each layer during the regular backpropagation process, introducing almost no additional computational overhead, making it ideal for real-time use during training.

[0048] 3.4) Based on the gradient sensitivity analysis results, this invention designs a dynamic, data-dependent layer freezing strategy. This strategy uses a binary decision function to determine the activation state of each layer. This represents the activation state of layer l. Indicates the first The layer is activated, and its parameters will participate in this update; This indicates that the layer is frozen, and its parameters remain unchanged. The specific formula is as follows:

[0049] in, For dynamic thresholds, , The preset benchmark threshold is (0.2 in this invention). This is an adjustable weighting coefficient used to control the degree of influence of uncertainty on the threshold.

[0050] The dynamic threshold mechanism is the core of this invention. When the model has high uncertainty in predicting the current frame image (i.e., ...), big), This increases accordingly, which means gradient sensitivity The activation threshold is raised, and only highly sensitive layers are activated and updated. This avoids blind and potentially harmful parameter adjustments when the model itself is uncertain; when the model has low uncertainty in predicting the current image (i.e., Small), Approaching the baseline threshold At this point, the system tends to activate more layers with higher gradient sensitivity, using high-confidence data to effectively fine-tune the model and improve performance.

[0051] Step 4: Domain-Sensitive Learning Rate Optimization 4.1) Define the foreground region For a classification probability graph to satisfy the condition that the maximum predicted probability of the class exceeds a threshold (In this invention, the threshold) Let the set of pixels be 0.5, and its mathematical expression be:

[0052] Subsequently, the spatial entropy values ​​of all pixels in the foreground region are averaged. A higher entropy value indicates greater uncertainty in the model's prediction for that region, and a potentially more severe domain shift. Finally, the domain shift factor is calculated. Calculated using the exponential decay function:

[0053] in, The total number of pixels in the foreground area. Sensitivity parameter (in this invention) The default value is 1.5, which controls the intensity of the response to entropy changes; exp() is an exponential function.

[0054] The value range is (0, 1], to ensure that when environmental uncertainty is high (large spatial entropy), The value decreases, thus reducing the overall learning rate and making model updates more conservative; in environments with high determinism (low spatial entropy), A value close to 1 allows for more robust parameter updates.

[0055] 4.2) Obtaining the domain offset factor Then, by further combining network structure depth and gradient information, the learning rate of each frozen layer is set to 0, and an independent learning rate is assigned to each activated layer. This allocation strategy is implemented by the following formula:

[0056] in, For the first Learning rate of layer The base learning rate set for the user. As a depth decay factor, For the first Distance from the output layer (number of layers). The attenuation coefficient (in this invention) The default value is 0.1), where m is the total number of layers in the encoder and embedding head, including frozen and active layers; It is the gradient normalization factor.

[0057] The depth decay factor is used to give shallower networks closer to the output layer a higher learning rate, as they typically learn task-specific features, while deeper networks update more slowly to maintain the stability of the general features they extract; the gradient normalization factor ensures that layers with larger gradients (i.e., layers more sensitive to changes in the loss function) receive a relatively higher learning rate, thereby improving training efficiency.

[0058] Step 5: Calculate multiple consistency constraints to improve the performance of object detection and tracking in the semi-supervised learning framework. 5.1) Enhancing detection consistency loss The definition is as follows:

[0059] in, and The teacher model and the student model are respectively used for the first The bounding box coordinates predicted for each object are, in this invention, taken as the coordinates of the upper left corner and the lower right corner of the bounding box. Let L1 norm be denoted as IoU(), and IoU() be the intersection-union ratio. This is a balancing coefficient used to adjust the weights between the L1 norm and the IoU loss; For the first The average entropy within the bounding box region of an object is used to quantify the uncertainty of that region. The higher the entropy value, the greater the prediction uncertainty in that region. The calculation method is as follows:

[0060] in, For the first The pixel size of the bounding box region of an object.

[0061] By introducing in the enhanced detection consistency loss As a weighting factor, the model can adaptively strengthen its learning in regions of high uncertainty, forcing the student model to align more closely with the teacher's predictions in these regions.

[0062] 5.2) Obtain the enhanced detection consistency loss for each frame of image. Loss compared to partial patches Then, the total loss function of the overall optimization objective is calculated. :

[0063] in, This is a hyperparameter used to control the weights for detection consistency.

[0064] The average total loss function is obtained by iterating through all frames in the continuous input video frames and averaging the total loss function.

[0065] The average total loss function is backpropagated to the student model, which optimizes its parameters based on the total loss function until all iterations are completed, ultimately yielding both the trained student model and the trained teacher model.

[0066] During the model inference stage, the original images of the video frames are input into the trained teacher model to obtain the bounding boxes of different objects. Then, the standard QDTrack association strategy is used to associate the same object in different frames, thereby realizing the multi-object tracking task in the video sequence.

[0067] To verify the effectiveness of the method of this invention, a systematic benchmarking framework was further designed to cover a variety of typical domain transfer scenarios. The student and teacher models were first pre-trained on the source dataset, and then further adaptively trained and their accuracy evaluated on a combined validation set of the target dataset using the method of this invention.

[0068] The experiment mainly selected two types of mainstream datasets: driving scenario datasets and pedestrian scenario datasets, to verify the generalization ability and adaptability of the model in different application domains, including four public datasets: BDD100K dataset, MOT17 dataset, DanceTrack dataset and Vkitti2 dataset.

[0069] The BDD100K dataset is a large-scale, diverse video dataset of driving scenarios, primarily built by the Artificial Intelligence Laboratory (BAIR) at the University of California, Berkeley. This dataset contains 100,000 high-definition video sequences, each approximately 40 seconds long, with a resolution of 1280×720 pixels and a frame rate of 30fps, totaling over 1100 hours. The video data covers six weather conditions (sunny, cloudy, overcast, rain, snow, and fog), six typical driving scenarios (city streets, residential areas, highways, parking lots, gas stations, and tunnels), and three time periods (dawn / dusk, daytime, and nighttime). This multi-dimensional and variable coverage provides an important foundation for testing the generalization capabilities of autonomous driving systems in complex real-world environments.

[0070] The MOT17 dataset is a benchmark dataset for pedestrian multi-object tracking released in 2017 by the MOT Halllenge team. It contains 14 video sequences: 7 training sequences and 7 test sequences, all from real-world surveillance scenes such as city streets and squares. The training set contains approximately 5300 frames, and the test set approximately 5700 frames, totaling over 1100 pedestrian trajectories. Video resolutions range from 640×480 to 1920×1080, and include challenging scenarios such as dense crowds, severe occlusion, varying lighting, and camera shake, making it suitable for testing algorithm performance under complex conditions. Scenes cover indoor and outdoor public places, such as streets, shopping malls, and parks, including both static and moving camera perspectives, as well as footage taken under different weather and lighting conditions (such as nighttime and rainy days).

[0071] The DanceTrack dataset was proposed by Sun et al. in 2022. The total size of the dataset is 100 videos and 100,000 frames (about 10 times that of MOT17). The training, validation and test sets contain 40, 25 and 35 videos respectively. The human movements cover more than 10 types such as street dance, gymnastics and kung fu, and include indoor and outdoor environments, low light and other environments.

[0072] The Vkitti2 dataset is a synthetic driving dataset containing high-quality labeled vehicle and pedestrian targets, suitable for baseline validation of model training and domain transfer studies.

[0073] Based on the above dataset, this invention constructs four types of domain transfer scenarios to systematically analyze the robustness of the DapaTTA method: 1. Shifting from outdoor to indoor scenes: MOT17 DanceTrack, BDD100K DanceTrack was used to verify the model's adaptability from open road environments (outdoors) to closed dance scenes (indoors). The core challenges were the significant differences in background texture, lighting conditions, and target motion patterns. 2. Shifting from indoor to outdoor scenes: DanceTrack MOT17, DanceTrack The BDD100K is used to evaluate the model's ability to transfer from complex indoor scenes to real driving or urban environments, with a focus on examining the cooperative tracking performance of dynamic background interference and multiple types of targets (such as vehicles and pedestrians). 3. Migrating composite scenes to real-world scenes: Vkitti2 BDD100K aims to explore the performance degradation mechanism of models trained on synthetic data in real-world scenarios, providing an interpretable verification basis for cross-domain adaptive algorithms. 4. Migrating from small-scale scenarios to large-scale data: MOT17 BDD100K, DanceTrack BDD100K is used to analyze the transfer potential of models under data-scarce conditions and the effectiveness of adaptive strategies for performance improvement through large-scale unlabeled data. Simultaneously, it explores reverse paths (such as BDD100K). MOT17) validates the generalization ability of the pre-trained driving model in pedestrian scenarios.

[0074] This experiment uses the appearance-based tracker QDTrack as its basic framework. QDTrack learns discriminative instance feature representations by enhancing the object detector and introducing an embedding head. This allows it to effectively utilize appearance similarity when performing data association between frames, mitigating association errors caused by motion model failure or severe occlusion in complex scenes. The detector uses Faster R-CNN as the basic architecture for object detection. This framework generates candidate regions through a Region Proposal Network (RPN) and uses the RoIAlign layer to extract features for final classification and bounding box regression. Backbone network and feature enhancement: The backbone network uses a ResNet-50 pre-trained on ImageNet to extract rich hierarchical features. To further improve the detection capability of multi-scale objects, a Feature Pyramid Network (FPN) is integrated. The FPN constructs a multi-scale feature pyramid with both high semantic information and fine spatial details through top-down and lateral connections, enabling the model to efficiently and accurately handle tracked objects with large scale variations. All experiments were conducted on four NVIDIA 2080Ti GPUs.

[0075] Based on the four established domain migration scenarios, this experiment compares the method of this invention (hereinafter referred to as DapaTTA) with several representative baseline models, namely No Adap., Ten, Vid and DARTH. The experimental results of each method model are evaluated using five indicators: HOTA, MOTA, DetA, IDF1 and AssA.

[0076] For scenarios involving the migration of synthetic scenes to real-world scenes, a systematic evaluation of test-time adaptive strategies under the condition of transitioning from synthetic data to real-world data was conducted in the domain adaptation scenario from the Vkitti2 dataset to the BDD100K dataset. Vkitti2, as a computer-generated driving scene dataset, has rendering characteristics that differ significantly from the real-world images included in BDD100K, including texture details, lighting conditions, and object shapes.

[0077] The experimental results are shown in Table 1. The results indicate that, compared to the four representative baseline models, the DapaTTA method performs best on all four metrics: HOTA, MOTA, DetA, and AssA, achieving scores of 46.7, 34.2, 39.3, and 48.9, respectively. It only differs from the best-performing DARTH model by 0.1 on the IDF1 metric. This result demonstrates that DapaTTA can effectively address the severe domain shift between synthetic and real data, achieving robust transfer from synthetic driving datasets to real-world scenario datasets.

[0078] Table 1: Comparison of DapaTTA results with other baseline TTA methods on the task of adapting the synthetic driving dataset Vkitti2 to the real-world BDD100K dataset.

[0079]

[0080] Table 2 shows the performance of the proposed method and four representative baseline models in the transfer from an outdoor surveillance scene (MOT17) to an indoor dynamic dance scene (DanceTrack). The results indicate that, compared to the four representative baseline models, the DapaTTA method exhibits better adaptability in this transfer direction, with the smallest gap compared to the ideal model trained directly on the target domain (DanceTrack) (Origin: a model trained directly on the target domain with full supervision and access to privileged information provided by the target labels). This further verifies the effectiveness of the proposed method in improving tracking robustness in complex scenes. The DapaTTA method performs best on all four metrics: HOTA, MOTA, DetA, and IDF1, achieving scores of 29.7, 59.3, 53.3, and 27.3, respectively. Only on the AssA metric does the DapaTTA method differ from the best-performing DARTH model by 0.2.

[0081] Table 2: Comparison of DapaTTA results with other baseline TTA methods on the task of adapting the outdoor monitoring scene MOT17 to the indoor dynamic dance scene DanceTrack.

[0082] In addition, this specific embodiment further designed two experiments to migrate from small-scale scenarios to large-scale data to verify the effectiveness of the adaptive strategy of improving performance through unlabeled large-scale data. The specific experimental design is as follows: 1) The model is pre-trained on the MOT17 dataset, and then the method of the present invention is used to migrate it to the BDD100K dataset for adaptive training; 2) The model is pre-trained on the DanceTrack dataset, and then the method of the present invention is used to migrate it to the BDD100K dataset for adaptive training.

[0083] The experimental results are shown in Tables 3 and 4. Compared with the ideal model (Origin) trained directly on BDD100K, the performance gap of the model adapted by DapaTTA is significantly reduced. In the task of adapting MOT17 to BDD100K, DapaTTA even outperformed the Origin model in terms of MOTA index.

[0084] Table 3: Comparison of DapaTTA results with other baseline TTA methods on the task of adapting MOT17 to the large-scale driving dataset BDD100K in outdoor pedestrian monitoring scenarios.

[0085] Table 4: Comparison of DapaTTA results with other baseline TTA methods on the task of adapting the indoor dynamic dance scene DanceTrack to the large-scale driving dataset BDD100K.

[0086] For the task of migrating from indoor to outdoor scenes, this specific embodiment designed an experiment to migrate the DanceTrack dataset to the MOT17 dataset. The experimental results are shown in Table 5. Although the source model (NoAdap.) was fully optimized in all aspects, it still has a large gap with the ideal model (Origin) trained directly on the MOT17 dataset in all aspects, indicating that the domain adaptation challenge in this direction is more complex.

[0087] Table 5: Comparison of DapaTTA results with other baseline TTA methods on the task of adapting the indoor dynamic dance scene DanceTrack to the outdoor monitoring scene MOT17.

[0088] In addition, this study analyzed the impact of different data augmentation settings on DapaTTA through ablation experiments. The experiments were conducted in a specific domain adaptation task scenario (Vkitti2). The experiment was conducted using a BDD100K, applying data augmentation settings including geometry (g) and photometric (p) to the teacher, student, and contrast views. The results are shown in Table 5. The results indicate that the data augmentation method used in this invention performs best across all five evaluation metrics. Applying independent geometric transformation augmentation operations to the teacher, student, and contrast views improves the overall performance of the model; introducing additional photometric augmentation operations during the generation of the student view from the teacher view yields even more significant performance improvements. This design transforms the detection consistency problem into a robustness learning problem to photometric transformations, requiring the model to maintain stable detection capabilities against changes in appearance such as lighting and color. This strategy further improves the multi-object tracking accuracy (MOTA) by 15.1 percentage points, demonstrating the effectiveness of the photometric augmentation strategy.

[0089] Table 6: Ablation experimental results of DapaTTA under different data augmentation settings.

[0090]

[0091] This invention quantitatively evaluates the independent contributions and synergistic effects of each core component in the DapaTTA method through ablation experiments, thereby verifying the rationality and effectiveness of its overall architecture design. The experiments were conducted in a specific domain adaptation task scenario (Vkitti2). The results of the BDD100K test are shown in Table 7.

[0092] Uncertainty-Guided Dynamic Layer Selection (UGDLS) aims to dynamically select the layers in a network for aligning source and target domain features by estimating the uncertainty of model predictions in real time. Experimental results show that when only the UUGDLS component is enabled in the model, the performance gain is extremely limited. This phenomenon indicates that relying solely on uncertainty to dynamically select alignment layers, without the cooperation of other constraints or optimization mechanisms, is insufficient to effectively capture and reduce distributional differences between domains.

[0093] The Multiple Consistency Regularization (MC) module enhances the model's prediction stability across consecutive frames in the target domain video sequence by imposing various consistency constraints, such as temporal consistency and geometric consistency. Analysis shows that introducing the MC component alone delivers a comprehensive and consistent performance improvement, particularly in the MOTA (Motion Over-The-Air) metric. This strongly demonstrates that imposing multiple consistency constraints effectively mitigates model jitter and inconsistencies in the target domain data, thereby significantly improving the robustness and reliability of cross-frame target detection and tracking.

[0094] A significant performance leap was observed when combining the Multiple Consistency Constraint (MC) with the Domain Sensitive Learning Rate Optimization (DSLRO) component. The DSLRO component is designed to adaptively adjust the learning rate based on the differences in the gradient norms of each network layer, guiding the model to better adapt to layers with gentler gradient changes (which typically imply greater domain invariance). The combination of the two demonstrates that the stability constraints provided by MC enable fine-tuning of DSLRO, allowing the model to optimize towards higher prediction confidence.

[0095] Ultimately, when all three core components were integrated, the model achieved optimal overall performance. This result fully verifies the close interdependence among the components, which together form an organic whole. UGDLS is responsible for locating the key adaptation layers, MC ensures the stability of the adaptation process, and DSLRO performs fine-grained parameter update optimization. This integrated design approach is key to achieving optimal adaptive performance from the virtual domain to the real domain.

[0096] Table 7: Ablation experiment results regarding the effects of different components of DapaTTA.

[0097]

[0098] The proposed DapaTTA method is a unified test-time adaptive framework designed to address key challenges in multi-object tracking under domain transfer scenarios. By integrating uncertainty-guided dynamic layer selection, domain-sensitive learning rate optimization, and multiple consistency constraints for detection and association, this method effectively balances tracking accuracy, temporal consistency, and computational efficiency. Experimental results in various domain transfer scenarios demonstrate that DapaTTA outperforms existing test-time adaptive methods for multi-object tracking in terms of robustness and adaptability, while maintaining low inference overhead.

[0099] Based on the same inventive concept, the present invention also provides a test-time adaptive multi-target tracking system, comprising: The enhanced view generation module is used to generate teacher view, student view, and comparison view based on the input video frame images; The bounding box generation module is used to construct a teacher model for processing the teacher view and a twin student model for processing the student view and the comparison view using pre-trained QDTrack. The three models output a first bounding box and a classification probability map for the student view, a second bounding box for the teacher view, and candidate boxes for the comparison view, respectively. The classification probability map contains the probability that each first bounding box belongs to a different category. The spatial entropy and divergence calculation module is used to calculate the spatial entropy value of each pixel in the student view and the object-level uncertainty value of the student view based on the first bounding box and the classification probability map, using the spatial entropy method and object-level KL divergence respectively. The activation layer selection module is used to calculate the KL divergence loss function of each layer of the student model based on the student view, and further calculate the sensitivity of each layer; a dynamic threshold is calculated using the object-level uncertainty value, and the activation state of each layer of the student model is determined based on the dynamic threshold and the binary decision function. If the sensitivity of the layer meets the judgment condition, the layer is activated, and a learning rate is assigned to the activated layer according to the domain offset factor; otherwise, the learning rate corresponding to the layer is 0; the domain offset factor is calculated based on the spatial entropy value of all pixels in the foreground region of the video frame image. The parameter optimization module is used to calculate the local patch contrast loss based on the teacher view, student view, and contrast view; calculate the enhancement detection consistency loss based on the first bounding box and the second bounding box; sum the enhancement detection consistency loss and the local patch contrast loss by weight to obtain the total loss function; and iterate through all video frame images to update the teacher model and twin student model. The video data input module is used to input the sequence of video frames to be tested into the trained teacher model. The second bounding box of each frame output by the teacher model is associated using the standard QDTrack association strategy to complete the multi-target tracking task of the sequence of video frames to be tested.

[0100] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0101] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0102] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A test-time adaptive multi-target tracking method, characterized in that, Includes the following steps: S1, obtain the teacher view, student view and comparison view based on the input video frame image; S2, using pre-trained QDTrack, construct a teacher model for processing the teacher view and a twin student model for processing the student view and the comparison view, respectively. The three models output a first detection box, a first bounding box and a classification probability map for the student view, a second detection box and a second bounding box for the teacher view, and a third detection box for the comparison view, respectively; the classification probability map contains the probability that each first bounding box belongs to a different category. S3, based on the first bounding box and the classification probability map, the spatial entropy value of each pixel in the student view and the object-level uncertainty value of the student view are calculated by the spatial entropy method and the object-level KL divergence, respectively. S4. Calculate the KL divergence loss function for each layer of the student model based on the student view, and further calculate the sensitivity of each layer. The dynamic threshold is calculated using the object-level uncertainty value. The activation state of each layer of the student model is determined based on the dynamic threshold and the binary decision function. If the sensitivity of the layer meets the judgment condition, the layer is activated. The learning rate is then assigned to the activated layer according to the domain offset factor. Otherwise, the learning rate of the corresponding layer is 0. The domain offset factor is calculated based on the spatial entropy value of all pixels in the foreground region of the video frame image. S5: Calculate the local patch contrast loss based on the output results of the teacher view, student view, and contrast view in the corresponding models; calculate the enhancement detection consistency loss based on the first bounding box and the second bounding box; sum the enhancement detection consistency loss and the local patch contrast loss by weight to obtain the total loss function; traverse all video frame images and update the teacher model and twin student model. S6. Input the sequence of video frames to be tested into the trained teacher model, and associate the second bounding box of each frame output by the teacher model with the standard QDTrack association strategy to complete the multi-target tracking task of the sequence of video frames to be tested.

2. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In S1, the teacher view, student view, and comparison view are specifically described as follows: A teacher's view is obtained by applying several geometric enhancement transformations to the video frame image; a student's view is obtained by applying several photometric enhancement transformations to the teacher's view; a comparison view is obtained by randomly applying several combined enhancement transformations to the video frame image; the combined enhancement transformation includes several geometric enhancement transformations and several photometric enhancement transformations, and the geometric transformation and photometric transformation appear mixed in the combined enhancement transformation.

3. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In S6, the local patch contrast loss is: ； in, Embed feature vectors for the appearance of positive samples in the student view. The appearance embedding feature vector corresponding to the third detection box that forms a positive sample pair with the positive samples of the student view. For the appearance embedding feature vector corresponding to the third detection box that forms a negative sample pair with the positive samples in the student's view, exp() is an exponential function. The loss is a local patch contrast loss; the positive sample of the student view is the first detection box with an intersection-union ratio greater than 0.7 with the second detection box.

4. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In S7, the specific content of updating the teacher model and twin student model is as follows: The average total loss function is obtained by averaging the total loss functions of all video frames. The twin student model optimizes the student model parameters based on the average total loss function; The weights of the teacher model are updated in reverse using the parameters of the twin student model, while other parameters remain frozen.

5. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In S4, the sensitivity of each layer is specifically calculated as follows: based on the KL divergence loss function, the gradient of the weight parameters corresponding to each layer is calculated in reverse, and the L2 norm is used to characterize the gradient of the weight parameters to obtain the sensitivity of each layer. The specific calculation formula for the KL divergence loss function is as follows: ； in, For the detection head of the student model, For the encoder of the student model, Let be the student model parameters at the t-th iteration. This is the student view, where T is the temperature parameter. , where is the probability distribution of each object in the student model under ideal conditions; C is the total number of categories.

6. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In S3, the spatial entropy value of each pixel in the student view and the object-level uncertainty value of the student view are calculated using the following steps: Using the spatial entropy method, the spatial entropy value of each pixel in the first bounding box is calculated based on the probability of each first bounding box; Calculate the KL divergence between the probability of each first bounding box and the ideal uniform probability distribution. The formula is as follows: ； in, For the first The predicted probability that the first bounding box corresponding to each object belongs to category c. For the first The KL divergence of an object, where C is the total number of categories; The object-level uncertainty value of the student view is calculated by averaging the KL divergences of all first bounding boxes in the student view.

7. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In S4, the binary decision function is: ； in, This represents the activation state of layer l. This is a dynamic threshold, calculated from the object-level uncertainty value of the student view. The sensitivity of the l-th layer.

8. The test-time adaptive multi-target tracking method according to claim 1, characterized in that, In step S4, the learning rate is assigned to the activated layer according to the domain offset factor, and the specific calculation formula is as follows: ； in, For the first Learning rate of layer The set base learning rate, For the domain offset factor, This represents the number of layers between the current layer and the output layer. is the attenuation coefficient, and m is the total number of layers in the student model.

9. A test-time adaptive multi-target tracking method according to claim 1, characterized in that, The formula for calculating the domain offset factor is: ； in, The total number of pixels in the foreground area. Here, exp() is the sensitivity parameter, and exp() is the exponential function. For pixels Spatial entropy value, This is the domain offset factor.

10. A test-time adaptive multi-target tracking system, used to implement the test-time adaptive multi-target tracking method of claim 1, characterized in that, The system includes: The enhanced view generation module is used to generate teacher view, student view, and comparison view based on the input video frame images; The bounding box generation module is used to construct a teacher model for processing the teacher view and a twin student model for processing the student view and the comparison view using pre-trained QDTrack. The three models output a first bounding box and a classification probability map for the student view, a second bounding box for the teacher view, and candidate boxes for the comparison view, respectively. The classification probability map contains the probability that each first bounding box belongs to a different category. The spatial entropy and divergence calculation module is used to calculate the spatial entropy value of each pixel in the student view and the object-level uncertainty value of the student view based on the first bounding box and the classification probability map, using the spatial entropy method and object-level KL divergence respectively. The activation layer selection module is used to calculate the KL divergence loss function of each layer of the student model based on the student view, and further calculate the sensitivity of each layer; a dynamic threshold is calculated using the object-level uncertainty value, and the activation state of each layer of the student model is determined based on the dynamic threshold and the binary decision function. If the sensitivity of the layer meets the judgment condition, the layer is activated, and a learning rate is assigned to the activated layer according to the domain offset factor; otherwise, the learning rate corresponding to the layer is 0; the domain offset factor is calculated based on the spatial entropy value of all pixels in the foreground region of the video frame image. The parameter optimization module is used to calculate the local patch contrast loss based on the teacher view, student view, and contrast view; calculate the enhancement detection consistency loss based on the first bounding box and the second bounding box; sum the enhancement detection consistency loss and the local patch contrast loss by weight to obtain the total loss function; and iterate through all video frame images to update the teacher model and twin student model. The video data input module is used to input the sequence of video frames to be tested into the trained teacher model. The second bounding box of each frame output by the teacher model is associated using the standard QDTrack association strategy to complete the multi-target tracking task of the sequence of video frames to be tested.

Images