A multi-small-target tracker and tracking method based on SAM2

Abstract

The application provides a multi-small-target tracker and tracking method based on SAM2, which divides a video into multiple segments, and overlaps the last frame of a previous video segment with the first frame of a next video segment; performs target detection on the first frame of an initial video segment, and assigns an ID to a detected target object; performs target tracking in each video segment by SAM2, and performs target detection on the overlapping frames between adjacent video segments; and the detected bounding box is matched with a mask output by SAM2 in a previous video segment by a'mask-to-detection' association strategy, so that a new ID can be assigned when a new target appears in a next video segment, and tracking interruption is avoided. The application overlaps video frames to make the mask and the detection in the same space-time position, so that as long as the appearance of the target is still visually distinguishable, the tracking will not be interrupted, and the tracking failure rate can be significantly reduced when the target volume is too small or the camera zooms and moves.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to a multi-small-target tracker and tracking method based on SAM2. BACKGROUND

[0002] With the advent of convolutional neural networks (CNNs) and vision Transformers (ViTs), detection-based multi-object tracking methods have achieved remarkable success in the field of computer vision. This paradigm first detects all objects of interest in each frame, and then combines appearance and motion information to associate these detection results with tracklets by means of motion models such as Kalman filters. However, when there are weak and small targets such as drones in consecutive frames, it still faces great challenges, and the fundamental reason lies in the difficulty of detecting such small targets.

[0003] Specifically, detection-based multi-object tracking methods are usually equipped with motion models, such as Kalman filters, which consist of a dynamic equation and an observation equation. The dynamic equation predicts the position of the tracking box in the next frame according to the speed state of the target; the observation equation updates the motion trajectory using the bounding box provided by the detector, thereby correcting the positioning bias when the target trajectory changes. However, small targets usually have limited feature representation and weak appearance cues, making them undetectable in some frames. This situation can cause the observation equation to fail to successfully update the position of the tracking box, and when the motion trajectory of the target changes abruptly, it is easy to cause tracking failure, as shown in the tracking results of (b) in Figure 1

[0004] In addition, when the camera zooms or moves, as shown in Figure 1 , when the camera zooms or moves between frame 540 and frame 910, the appearance and position of the tracked target may change abruptly and significantly, which often leads to tracking box association errors. Especially in subsequent frames, when the target cannot be detected or is incorrectly associated with an incorrect tracking box, neither the dynamic equation nor the observation equation can be accurately updated, resulting in the corresponding tracking possibly disappearing. As shown in (b) in Figure 1 , the Kalman filter-based tracker BoT-SORT loses the tracking target because it fails to correctly associate the detection results with the existing tracking boxes. SUMMARY

[0005] ​The application is to solve the problem of small target tracking loss in the existing detection-based tracking method, and proposes a multi-small target tracker and tracking method based on SAM2, which realizes automatic multi-target tracking by dividing the video into multiple segments and combining the detector with SAM2; by abandoning Kalman filtering and using a "mask-to-detection" association strategy instead of "detection-to-tracking" association, the problem of small target tracking loss and target ID error addition or switching is effectively solved.

[0006] The application is implemented by adopting the following technical solutions:

[0007] A multi-small target tracker based on SAM2 is proposed, which is composed of a video segmentation unit, a target detector, an ID allocation unit, a SAM2, a mask-to-detection association unit and a tracking unit; wherein:

[0008] The video segmentation unit divides the video containing N x M frames into N segments, each containing M+1 frames; wherein the last frame of each video segment and the first frame of the adjacent next video segment are the same video frame;

[0009] The target detector performs target detection on the first frame of each video segment, generates a bounding box and corresponding confidence data set for the detected target object;

[0010] The ID allocation unit allocates IDs to the target objects detected in the first frame of the initial video segment;

[0011] The SAM2 performs target tracking within each video segment in sequence; wherein for the initial video segment, the detection result with ID allocated by the target detector is used as a prompt to generate a mask with ID;

[0012] The mask-to-detection association unit collects the mask with ID generated by SAM2 in the last frame of each video segment, and uses the mask with ID to match the detection result on the first frame of the next video segment through similarity measurement;

[0013] The tracking unit allocates IDs to new target objects that fail to match by calling the ID allocation unit, and inputs them together with all matched masks with ID as a prompt to SAM2 for further target tracking.

[0014] In some embodiments of the application, the mask-to-detection association unit matches through similarity measurement includes:

[0015] Maximize the posterior probability of association between the mask set { } and the detection result set { }. ; wherein, represents a mask with detection results match the correct probability, including posterior probability based on appearance features , posterior probability based on IoU and posterior probability based on shape ;

[0016] Convert the posterior probability into the association cost:

[0017] ,

[0018] ,

[0019] ;

[0020] wherein, , and represent the cost matrix of appearance, IoU and shape, respectively;

[0021] Convert the optimization objective into minimizing the matching cost of appearance, IOU and shape, respectively:

[0022] ,

[0023] ,

[0024] ;

[0025] Optimize the appearance cost, IOU cost and shape cost in parallel using TOPIC's algorithm.

[0026] A multi-small-target tracking method based on SAM2 is proposed, comprising:

[0027] S1, a video containing N x M frames is divided into N segments, each containing M+1 frames; wherein the last frame of each video segment and the first frame of the adjacent next video segment are the same video frame;

[0028] S2, target monitoring is performed on the first frame of the initial video segment, and a bounding box and a corresponding confidence data set are generated for the detected target object; and the detected target object is assigned an ID;

[0029] S3, the detection result with ID is sent to SAM2 as a prompt, and SAM2 is used to perform target tracking in sequence within the initial video segment, generating a mask with ID;

[0030] S4, collect the masks with IDs generated by SAM2 at the last frame of the video segment; and perform object detection on the first frame of the next video segment;

[0031] S5, match the masks with IDs with the detection results on the first frame of the next video segment by similarity measure;

[0032] S6, assign IDs to the new object instances that fail to match, and input them together with the masks with IDs on all matches to SAM2 as hints to continue object tracking, return to S4 until N video segments are tracked and output the tracking results.

[0033] In some embodiments of the present application, S5 specifically includes:

[0034] maximize the posterior probability of association between the set of masks { } and the set of detection results { }; wherein, P (masks | detection results) represents the probability that the masks correctly match the detection results , which includes the posterior probability based on appearance , the posterior probability based on IoU , and the posterior probability based on shape .

[0035] Convert the posterior probability into association cost:

[0036] ,

[0037] ,

[0038] ;

[0039] wherein, , and represent the cost matrices of appearance, IoU and shape, respectively;

[0040] Convert the optimization objective into minimizing the matching cost of appearance, IOU and shape, respectively:

[0041] ,

[0042] ,

[0043] ;

[0044] Optimize the cost of appearance, the cost of IOU and the cost of shape in parallel using the TOPIC’s algorithm.

[0045] Compared with the prior art, the application has the advantages and positive effects that: in the multi-small-target tracker and tracking method based on SAM2, the video is divided into multiple segments, the last frame of each segment is the same as the first frame of the next segment; for the first frame of each segment, a detector is used to detect targets, a bounding box and a corresponding confidence are generated for the targets, and an ID is assigned to each detected target in the first video segment; in the first video segment, the detection results with the ID generated in the first frame are input into SAM2 as a prompt, so that SAM2 performs target tracking within the segment and generates time-consistent masks, since the detection results already contain the ID, the masks of the corresponding tracked targets also inherit the corresponding ID, then, the masks with the ID generated by SAM2 in the last frame of each video segment are collected, and the masks with the ID are matched with the target detection results on the first frame of the adjacent next video segment through similarity measurement, through the mask-to-detection strategy, the detection results are associated with the masks with the ID, and the detection results that are successfully matched also inherit the ID of the corresponding masks, and when a new target that has not been seen before appears, a new identity can be assigned to the target; the new detection results are input into SAM2 together with the masks with the ID for continuing target tracking. Since SAM2 searches for an object that is most similar in appearance to a tracked target in the previous frame in the next frame through segmentation matching, and assigns an ID based on a mask rather than a detection box, as long as the appearance of the target is still visually distinguishable, the tracking will not be interrupted, so that the tracking failure rate when the target volume is too small or the camera zooms and moves can be significantly reduced.

[0046] Other features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are part of the present application, serve to further understand the present application, and the illustrative embodiments and descriptions of the present application are used to explain the present application, but do not constitute an improper limitation on the present application. Obviously, the drawings described below are only some embodiments, and other drawings can be obtained from the drawings by those skilled in the art without creating labor.

[0048] Figure 1 Comparison of tracking results of the tracker of the present application and BoTSort (a is the original video of the unmanned aerial vehicle flying in the air, b is the tracking result of BoTSort, and c is the tracking result of the tracker of the present application);

[0049] Figure 2 Composition structure of SMA2;

[0050] Figure 3An execution process of the SAM2-based multi-small-target tracker proposed in the present application is shown in the figure;

[0051] Figure 4 An execution step of the SAM2-based multi-small-target tracking method proposed in the present application is shown in the figure. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments will be described clearly and completely below with reference to the drawings in the embodiments of the present application. The following embodiments are used to explain the present application but not to limit the scope of the present application.

[0053] The present application proposes a SAM2-based multi-small-target tracker (hereinafter referred to as SMOTracker), the core idea of which is to associate the masklet generated by SAM2 with the detection result of the detector. Unlike the traditional "detection-to-tracking" association method, the present application performs a "maskle-to-detection" association strategy and uses SAM2 to generate time-consistent masklets. By using the previously tracked bounding box of the previous frame as a prompt, SMOTracker can stably generate the masklets of these targets in the next frame. After the association is completed, the unmatched masklets will retain the identity of the corresponding target, and the unmatched detection results will be initialized as new tracking targets.

[0054] As shown in the figure, Figure 2 SAM2 is composed of an image encoder, a prompt encoder, a mask decoder, a memory attention module, a memory encoder and a memory bank.

[0055] The image encoder is only processed once per video frame and is used to generate a feature embedding representing the frame. In the embodiments of the present application, a MAE pre-trained Hierarchical image encoder is used, which has a hierarchical structure and can generate multi-scale features during the decoding process.

[0056] The memory attention module is stacked by L layers of transformer modules and is followed by an MLP. It first performs self-attention calculation on the image embedding of the current frame, and then performs cross-attention between the image embedding of the current frame and the content of the memory bank. Therefore, it can make the current frame features dependent on the features of the past frames, the prediction results and the new prompts, so as to obtain the context information from the previously output mask, the input prompt and the target pointer.

[0057] The cue encoder defines the extent of a target within a given frame using methods such as points, drawing bounding boxes, or masks. The cue vector output by the cue encoder can be represented as... In target tracking tasks, target cues must be provided in the first frame. SAM2 uses the encoding of the top-left and bottom-right corners of the bounding box of the cued target object as input, and in the remaining frames, it uses the mask predicted in the previous frame as input to the cue encoder.

[0058] The mask decoder combines the embedding vector output by the cue encoder with the image embedding vector with historical information conditions generated by the memory attention module to generate a predicted mask sequence and output the corresponding mask similarity score. and target score That is: M = {( ),( ...}; where, mask similarity score The MAE loss is used to measure the overall confidence of the mask; while the target score... The mask is then learned under the supervision of cross-entropy loss to determine whether it should exist in the current frame. The final output mask is selected based on the mask with the highest similarity score among all predicted masks.

[0059] The memory encoder first downsamples the output mask using a convolutional module, then adds it element-wise to the unconditional frame embedding generated by the image encoder to produce the memory embedding. These memory embeddings are then added to the memory bank.

[0060] The memory bank is a first-in, first-out (FIFO) queue used to store the most recently generated memory information during video decoding. In a video sequence, the memory bank not only saves the prediction results of the target object in past frames, but also maintains a queue with a capacity of... The FIFO queue stores past prediction information of the target object, while also storing the maximum number of prompts received. Frame prompt information. At any point in the sequence. memory bank It can be represented as That is, the memory bank contains the past. The output of the frame is one of its components.

[0061] While SAM2 can provide highly stable masklets, its inherent tracking paradigm processes targets sequentially; that is, it only initializes the tracking of the next target from scratch after the complete tracking of one target is finished. This sequential processing method makes it impossible to generate or integrate new detection results during the tracking process, thus hindering the implementation of correlation operations.

[0062] To overcome the above limitations, the present application divides the video into multiple video segments and performs object detection on the frames between adjacent video segments. The bounding boxes detected are then matched with masklets through a "mask-to-detection" association strategy, so that a new ID can be assigned when a new object appears. Within each video segment, SAM2 still adopts the sequential tracking manner, i.e. it finishes the tracking of one object and then starts processing the next object. When entering the processing of the next segment, the memory bank of SAM2 keeps the information in the previous video segment, so as to maintain the consistency of the tracking of the whole video.

[0063] Through the above concept, the SMOTracker proposed by the present application can completely abandon the Kalman filter, thereby avoiding its inherent limitations, while retaining the core characteristics of the tracking-by-detection method. As shown in (c) of FIG. 1, Figure 1 The tracker of the present application can maintain the tracking of all objects without losing the objects during the zooming and panning of the camera. In the case of sufficient computing resources, the SMOTracker can realize online tracking based on video stream.

[0064] The specific embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. As shown in FIG. 2, Figure 3 The SMOTracker proposed by the present application is composed of a video segmentation unit, an object detector, an ID assignment unit, a SAM2, a mask-to-detection association unit and a tracking unit, wherein:

[0065] The video segmentation unit divides the video containing N x M frames into N segments, each containing M + 1 frames; wherein the last frame of each video segment and the first frame of the adjacent next video segment are the same video frame, so they completely overlap in time.

[0066] The object detector performs object detection on the first frame of each video segment, generates a bounding box for the detected object and a corresponding confidence score set. } as well as its corresponding confidence score set }.

[0067] The ID assignment unit assigns an ID to the detected object in the first frame of the initial video segment.

[0068] SAM2 performs target tracking sequentially within each video segment. For the initial video segment, it uses the target detector's output with assigned IDs (i.e., bounding boxes) as cues to perform target tracking sequentially within the segment, generating masks with IDs. SAM2 completes tracking of one target before starting from the beginning again, re-initializing and tracking the next target, until all targets within the current video segment have been tracked.

[0069] Mask to detection association unit: Collect the mask with ID generated by SAM2 in the last frame of each video segment { }, using a mask with an ID and the detection result from the first frame of the next video segment { Matching is performed using similarity metrics.

[0070] The tracking unit, for any new target object that fails to match, calls the ID allocation unit to assign an ID to it, and then provides this ID, along with all matched masks bearing the ID, as input to SAM2 to continue target tracking. Since the last frame of each video segment is the same as the first frame of the next video segment, the mask and the detection result are in the same spatiotemporal location, making the matching process more reliable.

[0071] When using SMOTracker to perform multi-small-target tracking, such as Figure 4 As shown, follow these steps:

[0072] S1: Divide the video containing N×M frames into N segments, each segment containing M+1 frames; wherein the last frame of each video segment is the same video frame as the first frame of the next adjacent video segment.

[0073] S2: Perform target detection on the first frame of the initial video segment, generate bounding boxes and corresponding confidence datasets for the detected target objects; and assign IDs to the detected target objects.

[0074] S3: The detection results with IDs are sent to SAM2 as a cue. SAM2 is then used to perform target tracking sequentially within the initial video segment, generating a mask with IDs.

[0075] S4: Collect a mask with an ID generated by SAM2 in the last frame of the video segment; and perform object detection in the first frame of the next video segment.

[0076] S5: Use a mask with ID to match the detection results on the first frame of the next video segment using a similarity metric.

[0077] S6: Assign IDs to unmatched new target objects and input them together with all the matched IDed masks as cues to SAM2 for continuing target tracking, return to S4 until N video segments are tracked and output the tracking results.

[0078] The present application retains the existing detection-based tracking architecture, replaces the Kalman filter for predicting target position with SAM2, and replaces the tracking box originally generated by the Kalman filter with the mask generated by SAM2. The Kalman filter predicts the position of the target in the next frame based on the inertia of the target motion. When the direction of the target motion changes, the prediction result will often deviate. SAM2 alleviates the limitations of the Kalman filter and solves the problem of prediction deviation caused by changes in the direction of target motion by segmenting and matching to find the object in the next frame that is most similar in appearance to the tracked target. In the second aspect, the present application uses a "mask-to-detection" association strategy to solve the problem of tracking interruption caused by failure to detect small targets. Since SAM2 searches for the object in the next frame that is most similar in appearance to the tracked target in the previous frame by segmenting and matching, and assigns IDs based on masks rather than detection boxes, as long as the target appearance is still visually distinguishable, the tracking will not be interrupted.

[0079] In the present application, for the first segment of the video, target detection is performed on the first frame, and a unique identity (ID) is assigned to each target object in the detection result. Then, the IDed detection result is input as a cue to SAM2 to achieve target tracking. Since the detection result already contains the ID, the mask of the corresponding tracked target also inherits the corresponding ID. In each subsequent video segment, the IDed mask can be used to associate the detection results in the first frame of the video segment. The associated detection results also inherit the corresponding ID, while the unassociated detection results are defined as:

[0080] { } = Asso({ },{ }) (1)

[0081] Wherein, Asso represents the association operation, and the set { } represents the unassociated l detection results.

[0082] For these l unassociated detection results, the system assigns them new identities ID and inputs them together with all the n IDed masks as cues to SAM2 for continuing target tracking.

[0083] The association method is crucial because a good association strategy can alleviate the problem of false ID and ID switching; on the contrary, an improper association method will exacerbate the problem. In the embodiment of the application, a double-wheel parallel matching mechanism based on three kinds of measurement methods of appearance similarity, IoU similarity and shape similarity is adopted. The mechanism processes the three similarity measurements in a probabilistic optimization manner in parallel, can effectively fuse and integrate the information of each measurement, avoid mutual conflict, and thus obtain the optimal association result.

[0084] Specifically, the posterior probability of the association between the mask set { } and the detection result set { } is maximized as the target:

[0085] (2)

[0086] wherein, represents the probability of the mask and the detection result matching correctly, including the posterior probability based on the appearance feature , the posterior probability based on the IoU and the posterior probability based on the shape ; these posterior probabilities are further converted into association costs:

[0087] (3)

[0088] (4)

[0089] (5)

[0090] wherein, , and represent the cost matrices of the appearance, the IoU and the shape respectively. Thus, the optimization target can be converted into minimizing the matching costs of the appearance, the IOU and the shape respectively:

[0091] (6)

[0092] (7)

[0093] (8)

[0094] Finally, the TOPIC’s algorithm is adopted to optimize the appearance cost, the IoU cost and the shape cost in parallel.

[0095] In the embodiments of the present application, when Grounding DINO is combined as a detector, SMOTracker can become an unsupervised tracker that can process targets of any category, thanks to the multi-modal open set detection capability of Grounding DINO.

[0096] The experimental training process and reasoning process of SMOTracker of the present application are described in detail below, and are compared with a variety of existing methods in the field, including DeepOCSort, BoTSort, StrongSort, OCSort, ByteTrack, Imprassoc, BoostTrack, and BoostTrack++. The above-mentioned methods all use YOLOv10x model as a detector.

[0097] Training stage: In the embodiments of the present application, several publicly available unmanned aerial vehicle target detection datasets are collected to construct a large-scale training set, and a YOLO detector is trained based on these datasets. The YOLO detector is integrated into SMOTracker, which can achieve better performance in unmanned aerial vehicle tracking tasks compared with Grounding DINO.

[0098] Specifically, a pre-trained YOLOv10x model is used, and fine-tuning is performed on the self-built unmanned aerial vehicle detection dataset to use it as the detector model in the framework. The unmanned aerial vehicle detection training dataset is constructed from 13 public unmanned aerial vehicle datasets collected from the Internet. During training, the resolution of all images is adjusted to 1024 1024. Training is performed on 8 NVIDIA GeForce RTX 4090 GPUs, with a total of 180 training rounds, each round containing more than 800,000 images, and a batch size of 20. The optimizer uses AdamW, with an initial learning rate of a weight decay coefficient of The first 3 rounds are the warm-up phase, with a momentum of 0.8, and then the momentum is adjusted to 0.937.

[0099] Inference stage: all inferences are completed on NVIDIA RTX 4090 GPUs, and the same input settings as training are maintained. In the inference process of SMOTracker, only detection results with a confidence greater than 0.05 are retained.

[0100] Performance evaluation: The performance of the YOLO model is evaluated using two metrics, mAP50 and mAP50-95. The YOLOv10 model used has a performance on the validation set of mAP50 = 0.93 and mAP50-95 = 0.643. In the performance evaluation of multi-object tracking, five metrics are used, including: high-order tracking accuracy (HOTA), ID F1 score (IDF1), total identity number (IDs), total identity switch number (IDSW), and total number of missed targets (IDFN). Among them, the HOTA index balances the evaluation of detection and association performance by explicitly combining detection accuracy score and association accuracy score.

[0101] Comparison with the state-of-the-art method:

[0102] (1) DOHA Anti-UAV dataset: This dataset is a multi-UAV tracking benchmark dataset containing more than 100 videos, with a total of more than 180,000 bounding boxes and identity labels annotated. The embodiments of the present application evaluate each model on its test set. As shown in Table 1, the Y-SMOTracker of the YOLOv10x detector of the present application achieves 75.109 in the HOTA index and 77.951 in the IDF1 index, and only produces 32 total identity numbers, with an overall performance improvement of 90.7% (HOTA) and 98.21% (IDF1) compared to BoTSort. The biggest advantage of SMOTracker is its strong ability in small target tracking. In contrast, tracking models based on Kalman filter perform poorly in such scenarios due to their difficulty in handling extremely small targets.

[0103] Table 1 Comparison table on the DOHA Anti-UAV dataset

[0104] Method HOTA IDF1 IDs ID IDSW IDFN DeepOCSORT 38.553 35.799 487 486 17262 BoTSort 39.386 39.326 843 827 17268 StrongSORT 33.295 40.15 811 832 17268 OCSORT 35.843 46.3 128 120 17262 ByteTrack 38.223 49.156 147 139 17220 Imprassoc 33.412 48.158 1459 1620 17250 Boosttrack 11.215 7.629 67 48 22680 Boosttrack++ 13.127 8.569 75 54 22026 G-SMOTracker 61.109 ]]> ​ 65.685 ]]> ​ 36 ]]> 6243 ​ 36 6243 ]]> ​ Y-SMOTracker 75.109 77.951 32 40 ]]> ​ 3957

[0105] Table 1 also compares the performance of different detector configurations. Y-SMOTracker refers to the SMOTracker variant trained with YOLO as the detector, while G-SMOTracker uses GroundingDINO as the detector. Although the GroundingDINO series of detectors have stronger generalization ability, they are not as good as Y-SMOTracker in extremely small target tracking performance, which further illustrates that detector specialization for UAV datasets is crucial for achieving robust small target tracking.

[0106] (2) Anti-UAV dataset: This dataset is a large-scale infrared and visible tracking dataset, containing more than 300 video pairs, with a total of more than 580,000 manually annotated bounding boxes. As shown in Table 2, SMOTracker achieved the best performance of 61.578 in HOTA, 81.571 in IDF1 score, and only 94 in IDs, all of which are better than the current state-of-the-art tracking methods. From the scores of IDs and IDSW, the main optimization goal of BoostTrack and oostTrack++ methods is to reduce false positive ID assignment and identity switching, so they achieve very low values in both IDs and IDSW indicators. However, this improvement comes at the cost of significantly reducing other evaluation indicators, and the performance drops sharply in small target scenarios. In contrast, Y-SMOTracker also significantly alleviates the problem of false positive ID assignment and identity switching during tracking, not only achieving lower values than BoostTrack series in IDs and IDSW, but also achieving the current best (SOTA) performance in other evaluation indicators.

[0107] Table 2 Comparison table on Anti-UAV dataset

[0108] Method HOTA IDF1 IDs IDSW IDFN DeepocSORT 55.481 54.934 ]]> ​ 579 577 23229 BoTSort 54.981 52.957 652 656 23521 StrongSORT 43.359 38.838 809 1004 30486 OCSORT 52.06 49.696 713 729 26002 Bytetrack 55.575 ]]> ​ 53.842 700 683 23138 Imprasso 39.971 31.895 1137 1181 33187 BoostTrack 24.092 19.705 450 426 42202 BoostTrack++ 24.176 19.803 452 425 42157 G-SMOTracker 43.286 53.582 150 ]]> ​ 21 13038 ]]> ​ Y-SMOTracker 61.578 81.571 94 33 ]]> ​ 7981

[0109] (3) DUT Anti-UAV dataset: This dataset contains 20 videos, covering short and long sequences. Unlike other datasets, the DUT Anti-UAV dataset has a higher proportion of small unmanned aerial vehicle targets, with a small proportion of multi-target scenarios. The background scenes of this dataset are extremely diverse, including open sky and cloudy environments, urban environments (high-rise buildings and residential areas), and rural scenes (farmland and playgrounds). In addition, this dataset covers a variety of lighting conditions (such as daytime, nighttime, dawn, and dusk) and weather conditions (sunny, cloudy, and snowy). This complex background and significant outdoor lighting changes are crucial for training a robust and high-performance unmanned aerial vehicle detection model. As shown in Table 3, Y-SMOTracker achieved SOTA results of HOTA score 53.098, IDSW 16, and IDFN 5890 on this dataset.

[0110] Table 3 Comparison table on DUT Anti-UAV dataset

[0111] Method HOTA IDF1 IDs IDSW IDFN DeepocSORT 42.401 ]]> ​ 46.262 ]]> ​ 62 47 8171 BoTSort 40.822 41.737 60 45 8506 StrongSORT 39.379 41.241 87 76 8686 OCSORT 41.078 44.894 82 68 8437 ByteTrack 39.934 41.128 76 63 8563 Imprasso 32.791 30.082 268 243 9720 BoostTrack 12.698 9.4363 31 21 12428 BoostTrack++ 13.75 10.473 34 24 12345 G-SMOTracker 36.566 39.531 33 8 7557 ]]> ​ Y-SMOTracker 53.098 55.931 32 ]]> ​ 16 ]]> ​ 5890

[0112] From the above comparison, it can be seen that the tracker and tracking method of the present application have achieved the current best performance in multiple scenarios.

[0113] It should be pointed out that the above description is not a limitation of the present application, and the present application is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the spirit and scope of the present application should also be within the protection scope of the present application.

Claims

1. A SAM2-based multi-small-target tracker, characterized in that, Consists of a video segmentation unit, a target detector, an ID assignment unit, a SAM2, a mask-to-detection association unit and a tracking unit; wherein: The video segmentation unit segments a video containing N x M frames into N segments, each containing M+1 frames; wherein the last frame of each video segment is the same video frame as the first frame of the next adjacent video segment; The target detector performs target detection on the first frame of each video segment, and generates a bounding box and a corresponding confidence data set for the detected target object; The ID assignment unit assigns an ID to the target object detected in the first frame of the initial video segment; The SAM2 performs target tracking within each video segment in sequence; wherein for the initial video segment, the detection result output by the target detector with an assigned ID is used as a prompt to generate a mask with an ID; The mask-to-detection association unit collects the mask with an ID generated by the SAM2 in the last frame of each video segment, and matches the mask with an ID with the detection result on the first frame of the next video segment using a similarity measure; The tracking unit assigns an ID to new target objects that fail to match, and inputs them together with all the matched masks with an ID as a prompt to the SAM2 for continued target tracking; The matching by the mask-to-detection association unit using a similarity measure includes: To maximize the posterior probability of the association between the set of masks { } and the set of detection results { }: ; where denotes the probability that mask is correctly matched with detection result , which includes the appearance-based posterior probability , the IoU-based posterior probability , and the shape-based posterior probability ; Converting the posterior probability into an association cost: ， ， ； wherein, , and represent the cost matrices for appearance, IoU and shape, respectively; Converting the optimization objective into a matching cost that respectively minimizes the appearance, IOU and shape: ， ， ； Optimizing the appearance cost, IOU cost and shape cost in parallel using the TOPIC's algorithm.

2. A method of multi-small-target tracking based on SAM2, characterized in that, Comprises: S1, segmenting a video containing N x M frames into N segments, each containing M+1 frames; wherein the last frame of each video segment is the same video frame as the first frame of the next adjacent video segment; S2, performing target monitoring on the first frame of the initial video segment, generating a bounding box and a corresponding confidence data set for the detected target object; and assigning an ID to the detected target object; S3, inputting the detection result with an ID into the SAM2 as a prompt, and using the SAM2 to perform target tracking within the initial video segment in sequence to generate a mask with an ID; S4, collecting the mask with an ID generated by the SAM2 in the last frame of the video segment; and performing target detection on the first frame of the next video segment; S5, matching the mask with an ID with the detection result on the first frame of the next video segment using a similarity measure; S6, assigning an ID to new target objects that fail to match, and inputting them together with all the matched masks with an ID as a prompt to the SAM2 for continued target tracking, returning to S4 until the N video segments are tracked to output the tracking result; S5 specifically includes: To maximize the posterior probability of the association between the set of masks { } and the set of detection results { }: ; where denotes the probability that mask is correctly matched with detection result , which includes the appearance-based posterior probability , the IoU-based posterior probability , and the shape-based posterior probability ; Converting the posterior probability into an association cost: ， ， ； wherein, , and represent the appearance, IoU and shape cost matrices, respectively; Converting the optimization objective into a matching cost that respectively minimizes the appearance, IOU and shape: ， ， ； Optimizing the appearance cost, IOU cost and shape cost in parallel using the TOPIC's algorithm.

Images