A semi-supervised single target tracking training system

Abstract

This invention provides a semi-supervised single-target tracking training system, comprising: an interactive sampler that collects instance sets based on training videos and several labeled frames; and a trainer that receives the instance sets from the sampler and learns from them using a teacher-student training paradigm. The teacher network explores the training video temporally, tracking from labeled frames to unlabeled frames, generating pseudo-labels based on transitive consistency, and constructing unsupervised template-search pairs. The student network receives supervised template-search pairs from the instance sets and the unsupervised template-search pairs constructed by the teacher network, and optimizes the network based on these pairs. This invention, by providing only a very small amount of manual annotation for each training video, simultaneously mines labeled and unlabeled frames, enabling the tracker trained by this framework to achieve performance comparable to fully supervised trackers.

Description

Technical Field

[0001] This invention relates to the field of visual target tracking technology, and more specifically, to a semi-supervised single-target tracking training system. Background Technology

[0002] Visual object tracking is a fundamental task in computer vision, with numerous applications in fields such as unmanned warehouse management, intelligent broadcasting, robotics, and autonomous driving. Thanks to the development of deep learning, visual object tracking has made significant progress in recent years. To achieve high performance in deep models, recent fully supervised deep trackers require millions of labels on the target objects throughout the video for supervised training. In fact, the number of labels in large-scale video datasets used for training object trackers has grown to 10^ ... 6 This magnitude demonstrates the considerable time and effort required for manual annotation. To reduce the time and cost burden of manual annotation, unsupervised trackers that do not require manual annotation have attracted increasing attention in recent years. However, the performance of current unsupervised trackers lags far behind that of supervised trackers. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a semi-supervised single-target tracking training system.

[0004] According to one aspect of the present invention, a semi-supervised single-target tracking training system is provided, comprising interactive:

[0005] A sampler that collects instance groups based on training videos and several labeled frames;

[0006] A trainer that receives a set of instances of the sampler and learns from them using a teacher-student training paradigm; wherein,

[0007] The teacher network explores the training video temporally, tracking from labeled frames to unlabeled frames, generating pseudo-labels based on transitive consistency, and constructing unsupervised template-search pairs.

[0008] The student network accepts supervised template-search pairs from the instance group and unsupervised template-search pairs constructed by the teacher network, and performs network optimization based on the supervised template-search pairs and unsupervised template-search pairs.

[0009] Preferably, the sampler collects a group of instances, including:

[0010] Supervised instances include supervised template-search pairs acquired from a set of labeled frames, the sampling strategy of which follows the sampling method of the benchmark tracker;

[0011] An unsupervised instance, comprising two labeled frames and two arbitrary video frames, wherein the sampling strategy is stateful and progressive.

[0012] Preferably, the sampling strategy for the unsupervised instances is stateful and asymptotic, and its sampling process includes:

[0013] The sampler maintains a set of spatiotemporal locations of a target object. As its internal state, in which The superscript here indicates the number of sampling steps. This represents the index set of the actual labeled frames. This represents the index set containing real label-annotated frames and pseudo label-annotated frames, where i represents the i-th frame;

[0014] In each iteration, the sampler randomly triggers either a propagation mode or a review mode, loading two labeled frames. and two arbitrary video frames A set of labeled video frames. For training videos;

[0015] in,

[0016] Therefore, the loading process for two arbitrary frames is as follows:

[0017] Sample i2,j2 using the propagation pattern;

[0018] Define the range of sentinel indices Where r represents the frame interval radius;

[0019] Based on spatiotemporal location set The state and sentinel index range stored in the middle Perform the correct search region slicing, explore i2 and j2 in the time domain and generate pseudo-labels;

[0020] Using the review pattern to sample i2 and j2, the sampler follows the classic sampling strategy and performs random sampling. Review snapshots of previously explored target objects;

[0021] The loading process for the two labeled frames is as follows:

[0022] Selected and sampled unlabeled frames The nearest frame is used as the labeled frame.

[0023] The selection of i1 is defined as follows:

[0024]

[0025] The choice of j1 is defined as follows:

[0026]

[0027] Preferably, the sampler maintains a set of spatiotemporal locations. A state machine is used as the basis to record the progress of the teacher network's temporal exploration and outputs a set of instances for training based on the current state; a sampler collects the spatiotemporal location set. The pseudo-tags are used to update the state;

[0028] The trigger probability of the propagation mode is set to l is the training video. The number of frames.

[0029] Preferably, the teacher network explores the training video temporally, tracking from labeled frames to unlabeled frames, generating pseudo-labels based on transitive consistency, and constructing unsupervised template-search pairs, including:

[0030] Each group of unsupervised instances includes two labeled frames. and two arbitrary frames

[0031] The teacher network performs forward or backward tracking;

[0032] Snapshot of the labeled object and Beginning, in frame and Track the target above;

[0033] Output two pseudo-calibration boxes and

[0034] Following transitive consistency, we obtain

[0035] Based on the above and arrive The process involves constructing an unsupervised template-search pair.

[0036] The student network accepts supervised template-search pairs from the instance group and unsupervised template-search pairs constructed by the teacher network, and performs network optimization based on the supervised and unsupervised template-search pairs, including:

[0037] Each group of supervised instances contains a supervised template-search pair (Z). i ,X j ),in

[0038] Based on the unsupervised template-search pair Calculate the unsupervised loss of the student network using a benchmark tracker.

[0039] Based on the supervised template-search pair (Z) i ,X j ), calculate supervised loss of student networks according to benchmark trackers.

[0040] Based on the unsupervised loss and supervised losses Perform network joint optimization to calculate total loss Where λ is a constant.

[0041] Preferably, the network joint optimization is performed only on the student network;

[0042] Based on the joint optimization of the network, following the classic exponential moving average (EMA) paradigm, the model weights W of the teacher network are... tea The weight W of the student network will be regularly determined. stu Refresh at a constant rate α;

[0043] W tea ←α·W tea +(1-α)·W stu .

[0044] Preferably, the transmission consistency is:

[0045] Snapshots of two labeled objects Template matching relationships between them can be achieved through video Forward or backward tracking in the current process is transferred to two new object snapshots. Above, that is

[0046] Preferably, it further includes a tracker, the tracker comprising:

[0047] A classification head, used to classify whether a search area contains a target or background;

[0048] A bounding box regression head, which is used to output the bounding box of the target object on the search area;

[0049] An IoU prediction head, which is parallel to the calibration regression head, predicts the IoU between the calibration box and the ground truth box.

[0050] Preferably, the sampler applies weak and strong data augmentation to the unsupervised instances before sending them to the trainer;

[0051] in,

[0052] The weak data augmentation includes probabilistic image color perturbation, probabilistic image color grayscale conversion, random scaling of the object's size in the search area, and random translation of the object's position in the search area.

[0053] The enhanced data includes image flipping and rotation;

[0054] The image rotation employed an elliptical bounding box transformation strategy.

[0055] Preferably, the trainer further includes a dynamic classification loss, which mines unlabeled instances and dynamically switches the classification loss function between cross-entropy and mean absolute error.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1) The semi-supervised single-target tracking training system in this embodiment of the invention is named SSOT. By providing only a very small amount of manual annotation for each training video, it simultaneously mines both labeled and unlabeled frames, enabling the tracker trained by this framework to achieve performance comparable to a fully supervised tracker.

[0058] 2) The SSOT framework in this embodiment of the invention integrates the teacher-student training paradigm, proposing that the teacher tracker perform step-by-step temporal exploration, while the student tracker is responsible for network optimization; at the same time, by utilizing transitive consistency, it is possible to better mine the implicit information of unlabeled frames.

[0059] 3) The semi-supervised single-target tracking training system in this embodiment of the invention also provides three auxiliary strategies to enhance robustness. First, an IoU filtering module is designed to filter out failed tracking in temporal exploration; second, the framework applies weak and strong data augmentation to the teacher and student respectively, so that the network can converge better; third, the classification loss is dynamically switched between cross-entropy and mean absolute error, thereby focusing on discovering more difficult instances. Attached Figure Description

[0060] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0061] Figure 1 This is for training a target tracker under various supervision settings in one embodiment of the present invention;

[0062] Figure 2 This is another preferred embodiment of the invention for semi-supervised tracker training using transitive consistency;

[0063] Figure 3This is a flowchart of the main steps of the SSOT semi-supervised single-target tracking training framework in another preferred embodiment of the present invention.

[0064] Figure 4 Visualization of the results of SS-Ocean-3 (dashed line) and SS-TransT-3 (solid line) on the LaSOT public test set in another preferred embodiment of the present invention;

[0065] Figure 5 The results of SS-Ocean-3 (dashed line) and SS-TransT-3 (solid line) on the TrackingNet public test set are visualized in another preferred embodiment of the present invention. Detailed Implementation

[0066] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0067] First, let me explain the formal problem definition in all embodiments of this application.

[0068] This invention defines a simple yet effective video annotation metric for training single-object trackers, a strategy that can significantly reduce the number of annotations required for large-scale video datasets. (Define video...) It is a sequence of frames of length l, i.e. Where I t Let t be the t-th frame in the video; in this application, only each video is needed. k-frames are manually labeled in the video. Other frames in the array do not need to be labeled; the index set of labeled frames is represented as All the annotation boxes then form a set of annotation boxes. in Training a semi-supervised single-object tracker means training a tracker using only a set of bounding boxes. and video In this case, a tracker network will be trained from scratch. For convenience, the following text will use... This represents the set of labeled frames.

[0069] It is worth mentioning that the fully supervised target tracker can use a complete set of real-world bounding boxes. and video Training is performed, while unsupervised object tracking can only use a set of bounding boxes. and video Conduct training (see) Figure 1 ).

[0070] This invention provides an embodiment of a semi-supervised single-target tracking training system, comprising an interactive sampler and a trainer; the sampler collects instance sets based on training videos and several labeled frames; the trainer receives the instance sets from the sampler and learns from them using a teacher-student training paradigm; wherein,

[0071] The teacher network explores the temporal progression of training videos, from labeled frames to unlabeled frames, generating pseudo-labels based on transitive consistency and constructing unsupervised template-search pairs;

[0072] The student network accepts supervised template-search pairs from the instance group and unsupervised template-search pairs constructed by the teacher network, and optimizes the network based on the supervised and unsupervised template-search pairs.

[0073] This embodiment strikes a balance between unsupervised and fully supervised target trackers, proposing a semi-supervised single-target tracking training framework, abbreviated as SSOT. It can use only 2-5 manual annotations per video, with a total annotation count of around 10. 4 Under the premise of orders of magnitude, it achieves performance comparable to fully supervised trackers on multiple public single-object tracking benchmark datasets.

[0074] In a preferred embodiment of the invention, the sampler is responsible for providing supervised and unsupervised instances to the trainer. The sampling strategy for supervised instances strictly follows the sampling method of the benchmark tracker (i.e., randomly sampling template-search pairs from the set of labeled frames. This is the same for both Ocean and TransT). Supervised template-search pairs are collected. In contrast, the sampling strategy for unsupervised instances is stateful and asymptotic.

[0075] The preferred embodiment further describes the stateful and progressive sampling strategy for unsupervised instances. Due to the uncertainty of the center and scale of foreground objects in the video, it is impossible to sample from any frame without knowing the location (or at least approximate location) of the target object. Crop the appropriate search area X j Therefore, the sampling strategy for unsupervised instance groups follows a gradual diffusion strategy.

[0076] Specifically, regarding the aforementioned gradual diffusion strategy, the sampler maintains a set of spatiotemporal locations of the target object. As its internal state, in which The superscript here indicates the number of sampling steps. This represents the index set of the actual labeled frames. This represents the index set containing both real-labeled and pseudo-labeled frames, where i represents the i-th frame. It can be seen that the spatiotemporal location set is continuously updated by incorporating the pseudo-labels (those with IoU scores greater than a threshold) from each round. The initial set is the target location set of the labeled frames.

[0077] In each iteration, the sampler randomly triggers either a propagation mode or a review mode, and loads two labeled frames via the two different logics of the propagation mode or review mode. and two arbitrary video frames

[0078] To progressively explore untagged frames across a time span, a propagation mode is used. This mode starts from the current sentinel index range. Mid-sampling of i2,j2. Sentinel index range. Defined as follows.

[0079]

[0080] Sentinel Index Range This includes video frames that were not explored in previous steps, but whose neighboring frames (within the frame interval radius r) have been explored. Thus, it is possible to determine the appropriate frame type based on... The stored state allows for correct search region slicing, making incremental exploration and pseudo-label generation feasible in the temporal domain.

[0081] In review mode, the sampler follows a classic sampling strategy (i.e., randomly sampling template-search pairs from a set of labeled frames. This is the same for both Ocean and TransT), random sampling. The purpose is to review snapshots of previously explored target objects.

[0082] In a preferred embodiment, to maintain an adaptive temporal exploration speed, the trigger probability of the propagation mode is set to... This encourages the exploration of videos with more frames yet to be explored within the temporal domain.

[0083] Based on the above embodiments, two arbitrary frames and two arbitrary video frames are obtained. For the annotation frame They were selected as the unlabeled frames sampled. The nearest frame. The selection of i1 is formally defined as follows, while j1 is calculated using the same strategy. The key idea behind this design is that target tracking within shorter time intervals is often more reliable.

[0084]

[0085] In a preferred embodiment of the invention, the sampler maintains a set based on spatiotemporal location. The state machine records the progress of the teacher's network temporal exploration and outputs a set of instances for training based on the current state. The sampler collects pseudo-labels for state updates. Specifically, it represents the set of spatiotemporal locations of target objects in the sampler's state. The pseudo-labels generated by the teacher network in step t are continuously refreshed during training. They will be merged into the current target object's spatiotemporal location set. For future sampling, refresh it. Corresponding index set That's also true.

[0086] In another preferred embodiment of the invention, the training of the trainer is based on training a semi-supervised target tracker using a transitive consistency method. Transitive consistency depends on two phenomena that hold true for the target tracking task:

[0087] 1) Exploring within timeframes allows trackers to find snapshots of target objects that have never been discovered (annotated) before.

[0088] 2) Tracking trajectories starting from different artificial labels on the same object in the video should still fall on the same object. For example, if there are labels on frames 1 and 100, and it is possible to track from frame 1 to frame 30 and from frame 100 to frame 130 in the video, then frame 30 and frame 130 still refer to the same object.

[0089] Given the two assumptions above, there will be a set of labeled frames. The template matching relationship within the video gradually spreads to the entire video. This helps the tracker better utilize information from a large number of unlabeled frames (see...) Figure 2 ).

[0090] In formal terms, the set of annotation boxes It provides initial template matching relationships from manual annotation. Why not This is denoted as a tracing attempt from B to B'. Therefore, the basic logic of consistent transmission can be expressed as follows. Intuitively, snapshots of two labeled objects... Template matching relationships between them can be achieved through video Forward or backward tracking in the current process is transferred to two new object snapshots. Above.

[0091]

[0092] With new matching correspondence It can be easily based on the corresponding video frames Construct a new template search pair if Template search pair (Z) i ,X j If a pair is a supervised template-search pair, it is a supervised template-search pair; otherwise, it is an unsupervised template-search pair. Combinations of supervised and unsupervised template-search pairs are used to jointly optimize the tracker network.

[0093] In another preferred embodiment of the present invention, based on the aforementioned transitivity, the trainer is trained. The teacher network explores the training video temporally, tracking from labeled frames to unlabeled frames, generating pseudo-labels and constructing unsupervised template-search pairs based on transitivity. The student network receives supervised template-search pairs from the instance group and unsupervised template-search pairs constructed by the teacher network, and optimizes the network based on the supervised and unsupervised template-search pairs. Specifically,

[0094] In each training iteration of this embodiment, the trainer uses N L The group has supervised instances and N U Set up unsupervised training instances to optimize the network.

[0095] Each group of supervised instances contains a supervised template-search pair (Z). i ,X j ),in

[0096] Supervised loss of student networks is calculated strictly according to the baseline tracker.

[0097] In contrast, each group of unsupervised instances consists of two labeled frames. and two arbitrary frames To construct an unsupervised template search pair, the framework uses a teacher network to perform forward or backward tracking from labeled object snapshots. and Beginning, in frame and The target is tracked, resulting in two pseudo-calibration boxes. and Following transitive consistency, we obtain And construct an unsupervised template-search pair

[0098] The unsupervised loss of the student network is calculated again using the baseline tracker. Total loss yes and The weighted sum is shown in the following formula, where λ is a constant.

[0099]

[0100] The joint optimization of the above networks is performed only on the student network. Following the classic exponential moving average (EMA) paradigm, the model weights W for the teacher network are... tea The weight W of the student network will be regularly determined. stu Refresh at a constant rate α.

[0101] W tea ←α·W tea +(1-α)·W stu

[0102] The essence of training a deep neural network is optimizing the model parameters, which are the model weights in this context. Here, the student network is trained using supervised and unsupervised template-search pairs, while the teacher network is updated from the students using EMA (Empirical Modeling).

[0103] To enhance the robustness of semi-supervised training, in a preferred embodiment of the present invention, three strategies are provided: IoU filtering, strong / weak data augmentation, and dynamic classification loss. Specifically,

[0104] Teacher network tracking failures or noisy tracking results can lead to poorly labeled unsupervised instances, the large number of which is detrimental to the effectiveness of semi-supervised learning. Traditional object trackers typically have classification and regression branches, outputting foreground and background classification results in the search region, respectively. Regression results with calibration box use Using classification scores to filter tracking results is a very intuitive solution; however, The fractions in the equation always saturate to their extreme values, which makes it possible to pass through Direct filtering is not feasible. This embodiment proposes adding an IoU prediction head parallel to the calibration box regression head, responsible for outputting a sum... Parallel IoU Predicted Response Plot (See Figure 3 The IoU prediction header will be... Predict IoU score for each position in the dataset. Used to imitate The actual IoU between the regression bounding box and the ground truth bounding box in the graph.

[0105] In each iteration of semi-supervised training, only when the IoU prediction on the unlabeled frame satisfies... So, pseudo-labels Only then will it be considered reliable. Here, τ is a predefined threshold. The trainer will only build unsupervised template-search pairs based on reliable pseudo-labels. It's important to note that the IoU prediction head is trained only using supervised template-search pairs, and from... The backpropagated gradient is cut off before flowing into the baseline tracker (see...). Figure 3 Therefore, adding an IoU prediction head does not affect the network optimization of the original baseline tracker.

[0106] In the basic SSOT, the student network uses snapshots of objects in videos discovered by the teacher network as training instances. However, this approach may encourage the student and teacher networks to converge to the same results. To ensure the student network learns robust deep representations rather than overfitting to the teacher network's tracking results, this application applies weak and strong data augmentations to unsupervised instances before sending them to the teacher and student networks for processing. Weak data augmentations include the data augmentation methods used by the basic tracker TransT, such as probabilistic image color perturbations, probabilistic image color grayscale conversion, random scaling of object size within the search region, and random translation of object position within the search region. Strong data augmentations in the SSOT framework involve only conventional transformations, including image flipping and rotation, to maintain a basic semi-supervised single-tracking baseline. In this application, an elliptical bounding box transformation strategy is employed when rotating the image (see Kalra, Agathya, et al. "Towards Rotation Invariance in Object Detection." Proceedings of the IEEE / CVF International Conference on ComputerVision. 2021). Although this strategy is very simple, it demonstrates its important role in utilizing snapshots of unlabeled targets discovered through temporal exploration.

[0107] Some unlabeled object snapshots are highly correlated with labeled object snapshots, therefore they provide insufficient information to enhance the tracker's discriminative ability. This application proposes that student networks should place greater emphasis on difficult instances. For each generated pseudo-calibration box... Teacher network IoU prediction The higher (note that it must at least meet the following requirements) If the corresponding object snapshot contains a more difficult snapshot, then the snapshot will likely contain a more difficult one. To take advantage of this, inspired by the Generalized Cross Entropy (GCE) loss, this application proposes a dynamic classification loss. This is used for mining unsupervised template-search pairs. Note that the generalized cross-entropy loss states that cross-entropy (CE) loss emphasizes hard instances, while mean absolute error (MAE) treats each instance equally. Therefore, this application... The dynamic classification loss at each spatial point is defined as follows.

[0108]

[0109] Here p i yes The probability of middle school students predicting the target class. It is a linear mapping function from (τ,1] to (0,k], where k<1. When When, the dynamic classification loss becomes CE; when When this happens, the loss switches to MAE. Therefore, difficult instances will use a loss closer to CE, while simple examples will receive a loss relatively closer to MAE. This application uses the total dynamic classification loss within the same batch. Multiply by an adaptive rescalar to ensure Compared with traditional CE loss They have the same scale.

[0110] In other embodiments of the present invention, the SSOT scheme proposed in the above embodiments was verified from multiple perspectives, namely, the effectiveness of the framework, the performance comparison with other trackers, and the visualization of the tracking effect.

[0111] 1. Method effectiveness

[0112] Tables 1 and 2 demonstrate the effectiveness of the SSOT framework in this example under various settings. Specifically, this embodiment trains two heterogeneous trackers, TransT and Ocean (offline version), using two different training methods. The fully supervised variant (Sup) follows the classic sample matching paradigm, learning tracking only from k labeled video frames; while the semi-supervised variant (SS) uses the SSOT framework proposed in this embodiment to train labeled and unlabeled instances from both. The symbol Δ shows the performance improvement compared to the two. Experiments used k = 2, 3, and 5 manual labels in each video to study the impact of label sparsity on SSOT. As shown in Tables 1 and 2, the two single-object trackers, TransT and Ocean, trained with SSOT, outperform their supervised-trained variants with the same number of manual labels, regardless of their model architecture and label sparsity, and are not significantly behind their fully supervised variants (last row). This demonstrates that the SSOT training method and the more advanced tracking models obtained based on it, such as TransT, play a crucial role in improving overall tracking performance.

[0113] 2. Performance comparison with other trackers

[0114] This embodiment tests and evaluates SS-Ocean-3 and SS-TransT-3 on three commonly used public test datasets in the field of target tracking: TrackingNet, LaSOT, and GOT-10k. For TrackingNet and LaSOT datasets, AUC, standard accuracy (Norm.P.), and accuracy (Prec.) are used as evaluation metrics, while AO, SR0.50, and SR0.75 are used for the GOT-10k dataset. For comparison, this embodiment uses several benchmark trackers trained with full supervision (labels for each frame) as performance comparison and reference: SiamRPN++, DiMP, Ocean, and TransT. These trackers all use more than 10 labels. 6 This is on the order of magnitude, while SS-Ocean-3 and SS-TransT-3 use only 3 frames of labels per video, for a total of only about 50,000 labels. It is worth noting that on the GOT-10k dataset, following its testing requirements, only the GOT-10k training dataset was used during training, so only about 28,000 labels were used in total.

[0115] 3. Visualize the tracking effect

[0116] Table 3 shows the test results of each tracker on the TrackingNet and LaSOT datasets, while Table 4 shows the test results of each tracker on the GOT-10k dataset. A visualization is available below. Figure 4 and Figure 5 It can be observed that on the TrackingNet and LaSOT test sets, SS-Ocean-3 and SS-TransT-3, using only a very small number of labels (approximately 50,000 labels), achieved tracking performance close to, and even surpassing, some fully supervised trackers (such as SiamRPN++ and DiMP). On the GOT-10k dataset, the tracking performance of SS-TransT-3 and SS-Ocean-3 (approximately 28,000 labels) even surpassed that of the fully supervised TransT and Ocean (approximately 1.4 million labels), demonstrating the powerful performance of the SSOT framework in this embodiment.

[0117] Table 1. Validation of the SSOT framework on the TransT tracker

[0118]

[0119] Table 2. Validation of the SSOT framework on the Ocean tracker

[0120]

[0121]

[0122] Table 3 shows the performance on the TrackingNet and LaSOT test datasets.

[0123]

[0124] Table 4 shows the performance on the GOT-10k test dataset.

[0125] Model SiamRPN++ DiMP Ocean TransT SS-Ocean-3 SS-TransT-3 AO 51.7 61.1 61.1 67.1 61.3 67.6 <![CDATA[SR 0.50 ]]> 61.6 71.7 72.1 77.7 72.4 78.2 <![CDATA[SR 0.75 ]]> 32.5 49.2 47.3 60.9 51.1 61.1

[0126] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding modules, devices, units, etc. in the system. Those skilled in the art can implement the steps of the method by referring to the technical solution of the system. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0127] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0128] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A semi-supervised single-target tracking training system, characterized in that, Including interactive: A sampler that collects instance groups based on training videos and several labeled frames; A trainer that receives a set of instances of the sampler and learns from them using a teacher-student training paradigm; wherein, The teacher network explores the training video temporally, from labeled frames to unlabeled frames, generates pseudo-labels based on transitive consistency, and constructs unsupervised template-search pairs. The student network accepts supervised template-search pairs from the instance group and unsupervised template-search pairs constructed by the teacher network, and performs network optimization based on the supervised template-search pairs and unsupervised template-search pairs; The sampler collects instance groups including unsupervised instances; the sampling strategy for the unsupervised instances is stateful and asymptotic, and its sampling process includes: The sampler maintains a set of spatiotemporal locations of a target object. As its internal state, in which The superscript here indicates the number of sampling steps. This represents the index set of the actual labeled frames. This represents the index set containing real label-annotated frames and pseudo label-annotated frames, where i represents the i-th frame; In each iteration, the sampler randomly triggers either a propagation mode or a review mode, loading two labeled frames. and two arbitrary video frames A set of labeled video frames. For training videos; in, Therefore, the loading process for two arbitrary frames is as follows: Sample i2,j2 using the propagation pattern; Define the range of sentinel indices Where r represents the frame interval radius; Based on spatiotemporal location set The state and sentinel index range stored in the middle Perform the correct search region slicing, explore i2 and j2 in the time domain and generate pseudo-labels; Using the review pattern to sample i2 and j2, the sampler follows the classic sampling strategy and performs random sampling. Review snapshots of the target objects that have already been explored; The loading process for the two labeled frames is as follows: Selected and sampled unlabeled frames The nearest frame is used as the labeled frame. The selection of i1 is defined as follows: The choice of j1 is defined as follows: The teacher network explores the training video temporally, tracking from labeled frames to unlabeled frames, generating pseudo-labels based on transitive consistency, and constructing unsupervised template-search pairs, including: Each group of unsupervised instances includes two labeled frames. and two arbitrary frames The teacher network performs forward or backward tracking; Snapshot of the labeled object and Beginning, in frame and Track the target above; Output two pseudo-calibration boxes and Following transitive consistency, we obtain Based on the above and arrive The process involves constructing an unsupervised template-search pair. The student network accepts supervised template-search pairs from the instance group and unsupervised template-search pairs constructed by the teacher network, and performs network optimization based on the supervised and unsupervised template-search pairs, including: Each group of supervised instances contains a supervised template-search pair (Z). i ,X j ),in Based on the unsupervised template-search pair Calculate the unsupervised loss of the student network using a benchmark tracker. Based on the supervised template-search pair (Z) i ,X j ), calculate supervised loss of student networks according to benchmark trackers. Based on the unsupervised loss and supervised losses Perform network joint optimization to calculate total loss Where λ is a constant.

2. The semi-supervised single-target tracking training system according to claim 1, characterized in that, The sampler collects a group of instances, including: Supervised instances include supervised template-search pairs acquired from a set of labeled frames, the sampling strategy of which follows the sampling method of the benchmark tracker; An unsupervised instance, comprising two labeled frames and two arbitrary video frames, wherein the sampling strategy is stateful and progressive.

3. The semi-supervised single-target tracking training system according to claim 1, characterized in that, The sampler maintains a set of spatiotemporal locations A state machine is used as the basis to record the progress of the teacher network's temporal exploration and outputs a set of instances for training based on the current state; a sampler collects the spatiotemporal location set. The pseudo-tags are used to update the state; The trigger probability of the propagation mode is set to l is the training video. The number of frames.

4. The semi-supervised single-target tracking training system according to claim 1, characterized in that, The network joint optimization is performed only on the student network; Based on the aforementioned network joint optimization, following the classic exponential moving average paradigm, the model weights W of the teacher network are... tea The weight W of the student network will be regularly determined. stu Refresh at a constant rate α; W tea ←α·W tea +(1-α)·W stu 。 5. The semi-supervised single-target tracking training system according to claim 1, characterized in that, The transitivity consistency refers to: snapshots of two labeled objects Template matching relationships between them can be achieved through video Forward or backward tracking in the current process is transferred to two new object snapshots. Above, that is 6. The semi-supervised single-target tracking training system according to claim 1, characterized in that, It also includes a tracker, which comprises: A classification head, used to classify whether a search area contains a target or background; A bounding box regression head, which is used to output the bounding box of the target object on the search area; An IoU prediction head, which is parallel to the calibration regression head, predicts the IoU between the calibration box and the ground truth box.

7. The semi-supervised single-target tracking training system according to claim 1, characterized in that, The sampler applies weak and strong data augmentation to unsupervised instances before sending them to the trainer. in, The weak data augmentation includes probabilistic image color perturbation, probabilistic image color grayscale conversion, random scaling of the object's size in the search area, and random translation of the object's position in the search area. The enhanced data includes image flipping and rotation; The image rotation employed an elliptical bounding box transformation strategy.

8. The semi-supervised single-target tracking training system according to claim 1, characterized in that, The trainer also includes a dynamic classification loss, which mines unlabeled instances and dynamically switches the classification loss function between cross-entropy and mean absolute error.

Images