Single target tracking method from the perspective of a drone

Abstract

This invention relates to a single-target tracking method from the perspective of unmanned aerial vehicles (UAVs), belonging to the fields of computer vision and UAV application technology. Addressing the challenges of drastic target scale changes, significant background interference, and susceptibility to drift during long-term tracking in maritime UAV tracking, a single-target tracking method based on a dynamic multi-scale fusion Transformer (DMFTrack) is proposed. This method comprises two core modules: a dynamic weighted multi-scale feature fusion module (DWMS-FFM) and an adaptive template update mechanism (ATUM). DWMS-FFM adaptively fuses multi-scale features through a learnable gating mechanism, enhancing the model's adaptability to scale changes; ATUM, based on a confidence-aware two-stage attention mechanism, achieves stable and reliable long-term template updates, preventing tracking drift. This invention achieves excellent tracking accuracy and robustness on multiple UAV tracking benchmarks and is suitable for applications such as maritime surveillance and target state estimation.

Description

Technical Field

[0001] This invention relates to a single-target tracking method from the perspective of an unmanned aerial vehicle (UAV). This method is based on a dynamic multi-scale fusion Transformer to achieve single-target tracking from the perspective of an UAV, and is particularly suitable for single-target tracking of maritime vessels. It belongs to the field of computer vision and UAV application technology. Background Technology

[0002] With the rapid development of UAV technology, its applications in fields such as maritime monitoring, patrol and law enforcement, emergency rescue, target reconnaissance, and intelligent inspection are becoming increasingly widespread. UAVs typically rely on onboard visual sensors to continuously perceive and analyze targets on the ground or sea surface. Among these, single-target tracking, as one of the key technologies in UAV visual perception systems, directly affects the UAV's ability to continuously monitor targets and the effectiveness of mission execution.

[0003] However, compared to traditional fixed cameras or ground monitoring systems, single-target tracking from a UAV perspective faces more complex and demanding application environments. On one hand, the UAV's attitude and altitude constantly change during flight, causing significant dynamic changes in the target's scale, shape, and perspective within the image. On the other hand, images acquired by UAVs typically contain extensive background information, with the target occupying a relatively small proportion of the frame and susceptible to interference from complex backgrounds, lighting variations, and occlusion. Furthermore, in open environments such as at sea or in open spaces, the high similarity of background textures and numerous interfering targets further increase the difficulty of distinguishing the target from the background.

[0004] Most existing single-target tracking methods are based on deep learning models, obtaining the target's appearance representation through offline training and updating the target model frame by frame during the online tracking phase. However, these methods still have certain shortcomings in the context of UAVs: First, some methods rely excessively on the target features of the initial frame, making them prone to tracking drift when the target's appearance changes significantly; second, the online update strategy lacks effective constraints on tracking confidence, easily introducing erroneous information in cases of target occlusion or mismatch, leading to gradual model degradation; third, relying on fixed-scale backbone features or simple linear fusion strategies cannot adaptively handle the extreme scale changes commonly found in UAV videos, making it difficult to maintain stable tracking under complex motion and rapid viewpoint changes.

[0005] Therefore, designing a tracking method that can adaptively fuse multi-scale information and has a stable and reliable template update mechanism is key to improving the robustness and accuracy of long-term UAV tracking of ships at sea. Summary of the Invention

[0006] This invention provides a single-target tracking method from an UAV perspective, aiming to address the shortcomings of existing technologies in terms of tracking robustness and accuracy when facing problems such as drastic target scale changes, complex unstructured background interference, and template drift that easily occurs during long-term tracking in maritime UAV tracking scenarios. This invention constructs a unified and adaptive Transformer tracking framework by introducing a dynamic multi-scale feature fusion and confidence-aware adaptive template update mechanism, significantly improving the long-term stable tracking capability of ship targets under complex sea conditions.

[0007] A single-target tracking method from the perspective of an unmanned aerial vehicle (UAV), characterized by the following steps:

[0008] (1) Obtain the video sequence captured by the drone, extract the position of the target in the initial frame, crop the initial template image with the position as the center, and use the subsequent frames as the search image;

[0009] (2) Construct a single-target tracking network based on Transformer, the network including a feature extraction backbone, a dynamic weighted multi-scale feature fusion module (DWMS-FFM), an adaptive template update mechanism (ATUM), and a prediction head;

[0010] (3) Starting from the second frame, with the target position predicted in the previous frame as the center, a search area with a fixed size is cropped in the next frame. The template image and the search area are simultaneously input into the feature extraction backbone and jointly encoded to generate template features and search area features.

[0011] (4) Dynamic multi-scale feature weighting fusion is performed on the search region features extracted from the backbone using DWMS-FFM. The weights of each scale feature are generated through a learnable gating network to achieve dynamic scale-aware feature representation. The scale selection is at least 3 types. The steps are as follows:

[0012] 1) Perform multi-scale scaling on the input search region features to obtain feature representations at different scales;

[0013] 2) Extract features at various scales through lightweight convolutional branches;

[0014] 3) Use a gated network to generate adaptive weights for each scale based on global context information;

[0015] 4) The features at each scale are weighted and summed to obtain the fused multi-scale features;

[0016] (5) Input the fused multi-scale features and template features into the prediction head, and output the position of the target in the search area;

[0017] (6) Using the position of the prediction head output as the center, crop the template image from the original search image of the frame, and input the position of the prediction head output together with the original search image of the frame into ATUM. Use ATUM to update the template image of the frame with confidence awareness: evaluate the reliability of the current template image through a two-stage cross-attention mechanism, and update the template image only when the confidence is high to prevent drift in long-term tracking. The steps are as follows:

[0018] 1) Maintain a template pool of fixed capacity to store historical templates with high confidence levels and their confidence scores;

[0019] 2) Evaluate the confidence level of the current template image using the score head;

[0020] 3) The current template image is added to the template pool only when the update interval is reached and the confidence level is higher than the threshold;

[0021] 4) Randomly select a template image from the template pool and perform a weighted fusion with the current template image to update the current template image;

[0022] (7) Repeat steps (3) to (6) to achieve frame-by-frame tracking of maritime targets.

[0023] The single-target tracking network uses Vision Transformer (ViT) or its variants as the feature extraction backbone.

[0024] The single-target tracking network uses DeiT-tiny as the feature extraction backbone.

[0025] The prediction head uses a center-based prediction method to output the target center position offset, scale, and classification confidence, and finally obtains the target bounding box result in corner format based on the above data.

[0026] The method described is applicable to single-target tracking in videos captured by various drones and can effectively cope with complex environmental challenges such as scale changes, lighting changes, and background interference. In particular, the method is applicable to single-target tracking of ships at sea.

[0027] This invention proposes a single-target tracking method and its implementation device from the perspective of an unmanned aerial vehicle (UAV). By organically integrating multi-scale feature modeling, temporal correlation analysis, and an adaptive optimization learning mechanism, a stable target tracking framework suitable for complex dynamic scenarios is constructed. This technical solution can effectively address practical problems such as drastic changes in target scale, frequent perspective switching, and severe background interference during UAV flight, significantly improving the robustness and stability of the system while ensuring tracking accuracy. Compared with traditional tracking methods that rely on static features or fixed update strategies, this invention introduces a joint modeling mechanism of target historical state and current observation information, enabling the target localization process to have stronger temporal continuity and predictive ability, thus maintaining stable tracking even under rapid movement and complex maneuvering conditions.

[0028] Furthermore, this invention employs an adaptive optimization learning strategy to dynamically update the target model, adjusting the model update intensity based on tracking confidence. This effectively avoids the accumulation of erroneous information under conditions such as occlusion and mismatch, significantly reducing the risk of model drift. In engineering applications, this method does not rely on specific scenarios or manual rule settings, exhibiting good versatility and scalability. It can be widely applied to scenarios such as maritime surveillance, UAV inspection, target reconnaissance, and intelligent security, providing a highly reliable and adaptable single-target tracking technology solution for UAV visual perception systems.

[0029] Dynamic multi-scale adaptive capability: Through the DWMS-FFM module, the model can dynamically adjust the contribution of features at each level according to the current appearance and scale of the target, effectively overcoming the problem of drastic scale changes caused by changes in altitude and viewpoint under the perspective of UAVs, and enhancing the discriminative power of feature representation.

[0030] Stable and reliable long-term tracking: The ATUM mechanism uses a confidence-aware template update strategy to update the template only when tracking is reliable. This avoids template drift caused by occlusion, rapid movement or background interference, thus ensuring the stability of long-sequence tracking.

[0031] Efficient architecture design: The entire DMFTrack framework maintains high performance while having low model complexity and fast inference speed, meeting the real-time computing needs of UAV platforms and possessing high engineering application value.

[0032] Strong generalization performance: The method of this invention has achieved leading performance in multiple public UAV tracking benchmark tests, especially in challenging scenarios such as scale changes, occlusion and background clutter, demonstrating its strong generalization ability and robustness.

[0033] This invention provides an efficient and reliable single-target tracking solution for visual perception tasks in fields such as UAV maritime surveillance, maritime safety, and intelligent shipping. Attached Figure Description

[0034] Figure 1 This is a complete flowchart of the present invention.

[0035] Figure 2 This is a diagram of the deep learning model architecture of the present invention.

[0036] Figure 3 This is a schematic diagram of the structure of the Dynamic Weighted Multi-Scale Feature Fusion Module (DWMS-FFM) in this invention.

[0037] Figure 4 This is a flowchart of the Adaptive Template Update Mechanism (ATUM) algorithm in this invention.

[0038] Figure 5 This is the success rate result of the present invention tested and verified on the UAV123 drone tracking benchmark dataset.

[0039] Figure 6 This is the accuracy result of the present invention tested and verified on the UAV123 drone tracking benchmark dataset.

[0040] Figure 7 This is the normalized accuracy result of the present invention tested and verified on the UAV123 drone tracking benchmark dataset.

[0041] Figure 8 This invention compares the results of single-target tracking tests of ships at sea with those of other models. Detailed Implementation

[0042] The complete process framework of this invention is as follows: Figure 1 As shown.

[0043] One of the features of this invention is its applicability to single-target tracking of ships at sea. Therefore, the method for single-target tracking of ships at sea from the perspective of an unmanned aerial vehicle (UAV) includes:

[0044] Step 1. Acquire video sequences of ships at sea captured by drones, extract the bounding box of the specified target in the initial frame as the initial template image, and use subsequent frames as search images.

[0045] Step 2. Construct a single-target tracking network based on Vision Transformer (ViT). Using the predicted target location from the previous frame as the center, a search region with the same size as the initial template image is cropped in the next frame. This network uses a one-stream backbone and processes the template image... and search images Perform joint encoding (where 3 represents a channel RGB image, and (These represent the dimensions of the template image and the search region, respectively), generating a feature token sequence. The encoding process is represented as:

[0046] (1)

[0047] in, This represents the image block embedding and Transformer encoding process. and These are the features of the template image and the search region, respectively.

[0048] Step 3. Input the search region features obtained in Step 2 into the Dynamically Weighted Multi-Scale Feature Fusion (DWMS-FFM) module to adaptively aggregate multi-scale contextual information. The structure of this module is as follows: Figure 2 As shown, it specifically includes:

[0049] (1) Features of the input search region (in For batch size, For the number of channels, (Image size), a set of predefined scales is generated through bilinear interpolation. Feature maps at at least 3 scales.

[0050] (2) The features at each scale are processed by a lightweight convolutional branch to extract scale-specific features. :

[0051] (2)

[0052] in, This represents the i-th convolutional branch.

[0053] (3) Resample the features at all scales to the same spatial resolution and stack them to form a multi-scale feature tensor. .

[0054] (4) Attention weights at each scale are dynamically generated based on the global context of the input features through a gating network consisting of fully connected layers. :

[0055] (3)

[0056] in, Represents the normalization function. Indicates global average pooling. Represents the ReLU (Modified Linear Unit) activation function. and These represent the weight matrices of the two fully connected layers, and These represent the bias vectors of the two fully connected layers, respectively.

[0057] The obtained weights are used to perform weighted fusion of multi-scale features, and then connected to the original input features via residual connections. Adding them together yields the enhanced fusion features. :

[0058] (4)

[0059] in, This represents multi-scale features stacked together. This indicates a weighted fusion of features.

[0060] Step 4. Input the fused features output from Step 3 into the prediction head for target localization and classification, and output the target's position in the search area. The prediction head uses a center-point-based design and outputs the target center offset. Normalized bounding box size and classification confidence plot .

[0061] Step 5. Using the position of the prediction head output as the center, crop a template image with the same size as the initial template image from the original search image of this frame. Input the position of the prediction head output along with the original search image of this frame into ATUM. Use the Adaptive Template Update Mechanism (ATUM) to maintain and update the tracking template. The algorithm flow is as follows: Figure 3 As shown, it specifically includes:

[0062] (1) Maintain a template pool with a fixed capacity. ,in For template feature vectors, The confidence score is derived from the score head.

[0063] (2) When the set update interval is reached The confidence level of the tracking result in the current frame is evaluated using a fractional head. The fractional head calculates the matching confidence level between the current template and the features of the search region through a two-stage cross-attention mechanism. .

[0064] (3) Only when confidence level When the value exceeds a preset threshold, the template features extracted from the current frame will be used. Its confidence level is stored in the template pool. .

[0065] (4) Sort the template pool in descending order of confidence and retain the top ones. (At least 3) high-quality templates.

[0066] (5) Randomly select a historical template from the template pool. , with the current online template Weighted fusion is performed to achieve stable updates:

[0067] (5)

[0068] in, To balance the weighting coefficients of the old and new templates, This is a random selection operation.

[0069] Step 6. Calculate the loss using the set loss function based on the obtained prediction results. Repeat steps 2 to 4 continuously to iteratively optimize the model parameters by minimizing the loss.

[0070] In the first stage of training, excluding the score header, the total loss function consists of classification loss and regression loss:

[0071] (6)

[0072] in, This is the weighted focus loss. For GIoU loss, For L1 loss, and To balance the weights.

[0073] In the second phase, everything except the scorehead is frozen, and only the scorehead is trained. The total loss is as follows:

[0074] (7)

[0075] in, For confidence loss, To balance the weights.

[0076] Step 7. For each frame of the video sequence, repeat steps 2 through 5 to achieve continuous and robust tracking of the target from the UAV's perspective. The tracker ultimately outputs the precise bounding box of the target vessel in each frame.

[0077] Example

[0078] The single-target tracking method from the perspective of an unmanned aerial vehicle (UAV) described in this invention has the following overall process framework: Figure 1 As shown, the model is mainly implemented through the collaborative work of the Dynamically Weighted Multi-Scale Feature Fusion Module (DWMS-FFM) and the Adaptive Template Update Mechanism (ATUM) embedded in the Transformer backbone network. The overall model architecture is as follows: Figure 2As shown. Taking single-target tracking of a maritime vessel as an example, in this embodiment, the UAV platform is equipped with a visual acquisition device to continuously acquire video sequences of the area where the target vessel is located during flight. In the target tracking initialization phase, the initial position of the target vessel is determined in the initial frame of the video sequence. This initial position can be obtained through manual annotation or automatically generated by a target detection algorithm, and is used as a reference starting state for subsequent tracking processes. After obtaining the initial target position, the system constructs a target search region centered on this position and performs cropping and scale normalization on the search region to reduce the interference of background redundancy information on target feature extraction.

[0079] The video stream is input to the system frame by frame. In each frame, the search region is cropped centered on the prediction result of the previous frame and input into the tracking network along with the currently maintained template image. The backbone of the network adopts a DeiT-tiny Transformer architecture, with the template image and search region having fixed sizes of 128×128 pixels and 256×256 pixels, respectively. After block embedding and Transformer encoding, a fused feature representation of the template and search region is obtained.

[0080] After identifying the search region features, DWMS-FFM is then used for enhancement. The structure of this module is as follows: Figure 3 As shown. In the specific implementation, n scale factors are preset (at least 3, for example 0.75, 1.0, 1.25) for the input feature map. Perform bilinear interpolation scaling. For each scale... The features are processed through a separate, channel-reduced 3×3 convolutional layer.

[0081] (2) The features at each scale are processed by a lightweight convolutional branch to extract scale-specific features. :

[0082] (2)

[0083] in, Indicates the first Each convolutional branch.

[0084] The gating network consists of two fully connected layers, which process the input features. After performing global average pooling, normalized weights corresponding to the three scales are calculated.

[0085] (3)

[0086] in, Represents the normalization function. Indicates global average pooling. Represents the ReLU (Modified Linear Unit) activation function. and These represent the weight matrices of the two fully connected layers, and These represent the bias vectors of the two fully connected layers. These weights are multiplied by the features at each scale, summed, and then passed through a 1×1 convolutional layer to adjust the number of channels. Finally, the residuals are added to the original input features.

[0087] (4)

[0088] in, This represents multi-scale features stacked together. This represents feature weighted fusion. This process enables the network to dynamically prioritize detailed features that aid in localization or semantic features that provide contextual information, based on the current image content, thereby adapting to the drastic scale changes of the target vessel caused by the drone's movement.

[0089] During the tracking process, the maintenance and updates of the template are handled by ATUM, and the process is as follows: Figure 4 As shown. The system maintains a maximum capacity. A template pool of at least 3 is used to store historical high-confidence templates and their corresponding confidence scores. Confidence is calculated by a dedicated score header, which receives the current search feature and template features, evaluates the reliability of their match through a two-stage cross-attention mechanism, and outputs a score between 0 and 1. The tracker does not update the template every frame, but rather sets an update interval. (e.g., every 50 frames). Tracking confidence only in the current frame. Only when the confidence score exceeds a preset threshold τ (e.g., 0.7) will the features extracted from the current frame after cropping and preprocessing based on the predicted bounding box be added to the template pool as candidate templates. The template pool is sorted in descending order of confidence score, and the highest-scoring templates are retained. There are several templates. When a template update is needed, the system randomly selects a historical high-quality template from the pool and compares it with the currently tracked online template according to the formula:

[0090] (5)

[0091] Perform linear interpolation fusion, where To balance the weighting coefficients of the old and new templates, they are typically set to 0.9 to achieve a balance between adapting to changes in the target appearance and maintaining template stability. This is a random selection operation. This confidence-based update strategy effectively prevents the introduction of erroneous template information when the target is briefly occluded, undergoes drastic deformation, or is similar to background interference objects, thereby suppressing drift in long-term tracking.

[0092] Features enhanced by DWMS-FFM are fed into the prediction head. The prediction head consists of four convolutional layers (each containing convolution, batch normalization, and ReLU activation), whose output is reinterpreted as a two-dimensional spatial feature map and predicts three parts: a classification confidence map indicating the probability that each location is the target center; a center offset map for precise localization; and a bounding box size map. During inference, a Hanning window is first applied to the classification confidence map to introduce a location prior and suppress responses from regions far from the target location in the previous frame. Then, the point with the highest response is selected as the prediction center, and the final target bounding box is calculated by combining the predicted offset and size of that point.

[0093] To ensure the tracking model possesses strong feature extraction and discrimination capabilities, it needs to be thoroughly trained offline. The training data utilizes large-scale, general-purpose single-object tracking datasets, such as a mixture of LaSOT, TrackingNet, GOT-10k, and COCO datasets.

[0094] The training is divided into two phases:

[0095] In the first phase, the parameters of the scorehead are fixed, and the AdamW optimizer is used to train the rest of the network for 300 epochs, with an initial learning rate set to... And after the 240th round, it dropped to The loss function is the total loss of the prediction head. .

[0096] In the second stage, the parameters of the backbone network and modules such as DWMS-FFM were fixed, and only the scorehead underwent 50 rounds of fine-tuning training to enable it to accurately evaluate template reliability. Data augmentation techniques such as random horizontal flipping and brightness jitter were used during training to improve the model's generalization ability. The batch size was set to 32.

[0097] In actual UAV maritime tracking missions, after system initialization, it enters an autonomous operation loop. For each newly arrived image frame, the system automatically executes the above process: cropping the search area, extracting features, dynamic multi-scale fusion, predicting the target state, assessing confidence, and deciding whether to update the template. The entire process runs in real-time or near real-time, continuously providing operators with accurate target vessel position information.

[0098] The method of this invention has been tested and verified on multiple UAV tracking benchmark datasets such as UAV123 and DTB70. It demonstrates significant advantages, particularly in handling challenging scenarios such as scale variations, background clutter, and partial occlusion, while maintaining low model complexity and computational overhead. This fully proves its effectiveness and efficiency in practical engineering applications. Specific success rate and accuracy metrics are shown in Table 1 below:

[0099] Table 1

[0100] Dataset Success rate (%) Accuracy (%) DTB70 65.5 84.23 UAV123 67.82 86.12 UAV123@10fps 67.01 84.66 UAVTrack112 67.33 83.13 UAVTrack112L 66.05 82.65 .

[0101] Detailed results on the success rate, accuracy, and normalized accuracy of this invention tested and verified on the UAV123 unmanned aerial vehicle tracking benchmark dataset can be found in [the respective tables]. Figure 5 , Figure 6 , Figure 7 Specifically, the comparison of the results of this invention in single-target tracking tests of ships at sea with other models shows that... Figure 8 .

Claims

1. A single-target tracking method from the perspective of an unmanned aerial vehicle (UAV), characterized in that... Includes the following steps: (1) Obtain the video sequence captured by the drone, extract the position of the target in the initial frame, crop the initial template image with the position as the center, and use the subsequent frames as the search image; (2) Construct a single-target tracking network based on Transformer, the network including a feature extraction backbone, a dynamic weighted multi-scale feature fusion module DWMS-FFM, an adaptive template update mechanism ATUM, and a prediction head; (3) Starting from the second frame, with the target position predicted in the previous frame as the center, a search area with a fixed size is cropped in the next frame. The template image and the search area are simultaneously input into the feature extraction backbone and jointly encoded to generate template features and search area features. (4) Dynamic multi-scale feature weighting fusion is performed on the search region features extracted from the backbone using DWMS-FFM. The weights of each scale feature are generated through a learnable gating network to achieve dynamic scale-aware feature representation. The scale selection is at least 3 types. The steps are as follows: 1) Perform multi-scale scaling on the input search region features to obtain feature representations at different scales; 2) Extract features at various scales through lightweight convolutional branches; 3) Use a gated network to generate adaptive weights for each scale based on global context information; 4) The features at each scale are weighted and summed to obtain the fused multi-scale features; (5) Input the fused multi-scale features and template features into the prediction head, and output the position of the target in the search area; (6) Using the position of the prediction head output as the center, crop the template image from the original search image of the frame, and input the position of the prediction head output together with the original search image of the frame into ATUM. Use ATUM to update the template image of the frame with confidence awareness: evaluate the reliability of the current template image through a two-stage cross-attention mechanism, and update the template image only when the confidence is high to prevent drift in long-term tracking. The steps are as follows: 1) Maintain a template pool of fixed capacity to store historical templates with high confidence levels and their confidence scores; 2) Evaluate the confidence level of the current template image using the score head; 3) The current template image is added to the template pool only when the update interval is reached and the confidence level is higher than the threshold; 4) Randomly select a template image from the template pool and perform a weighted fusion with the current template image to update the current template image; (7) Repeat steps (3) to (6) to achieve frame-by-frame single target tracking.

2. The method of claim 1, wherein, The single-target tracking network uses VisionTransformer or its variants as the feature extraction backbone.

3. The method of claim 1, wherein, The single-target tracking network uses DeiT-tiny as the feature extraction backbone.

4. The method of claim 1, wherein, The prediction head uses a center-based prediction method to output the target center position offset, scale, and classification confidence, and finally obtains the target bounding box result in corner format based on the above data.

5. Use of the method according to claim 1 or 2 or 3 or 4, characterized in that, The method is applied to single-target tracking of ships at sea.

Images