Target tracking method based on local and global attention modeling

Abstract

The invention discloses a target tracking method based on local and global attention modeling. Comprising the following steps: constructing a backbone network to preliminarily extract features, and pre-training on a basic classification image set; establishing a tracking neck module, wherein the tracking neck module comprises an attention structure with global and local attention capturing capabilities and a double-flow single-target tracking network which is used for tracking and accurate positioning and is based on a twin architecture; defining an optimization method, optimization parameters and a loss function; inputting a fixed initial frame image into a template branch of the backbone network, and extracting an invariant template feature map by tracking the feature backbone network; and inputting subsequent frames into a search area branch of the backbone network in sequence, thereby realizing stable tracking and accurate positioning of given unmanned aerial vehicle data. The method is excellent in performance under the scene of high visual angle and scale change of the unmanned aerial vehicle, real-time tracking can be carried out at the speed of 25 frames per second, and the real-time requirement in practical application is met.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a target tracking method based on local and global attention modeling. Background Art

[0002] Single target tracking algorithm is a key research direction in the field of computer vision. The core of this task is to accurately predict the position of the object in subsequent frames based on the initial appearance and position of the object, using methods such as bounding rectangles. This function has wide application value in many fields, such as human-computer interaction, autonomous navigation, and satellite monitoring. Although current research has made significant progress in dealing with challenges such as object scale changes and occlusion, existing trackers still face many difficulties in dealing with drone targets. This is mainly due to the particularity of drone targets, such as small size, fast speed, and complex and changing background. Therefore, it is necessary to deeply explore and design specialized network structures to more effectively solve specific challenges in drone target tracking. By continuously optimizing algorithms and improving performance, it is expected to contribute more to the development of drone visual tracking.

[0003] The Siamese tracking framework aims to treat the tracking task as a similarity matching problem, adopting a dual-branch structure containing a template and a search region. The template region mainly focuses on capturing the intrinsic features of the object, while the search region aims to distinguish the object from the background. Early works such as SiamFC, SiamRPN, and SiamRPN++ laid the foundation for the Siamese tracking framework, which usually consists of a tracking backbone, a tracking neck, and a tracking head. In SiamFC, a correlation-based tracking neck is introduced to define the tracking task and add multi-scale applications. SiamRPN integrates the region proposal network (RPN) in Faster R-CNN into the tracking head to estimate the center point of the object using a rectangle defined by its width and height. SiamRPN++ further enhances the tracking performance by combining deep backbone networks such as ResNet and implementing a novel sampling strategy to prevent the sampled points from being overly clustered around the center. In addition, DaSiamRPN solves the problem of distractor interference by introducing a learnable distractor-aware block to effectively suppress similarity interference. For anchor-free structures such as SiamBAN, SiamCAR, and SiamFC++, the traditional RPN module is eliminated and the estimated center point, width, and height of the object bounding rectangle are directly predicted.

[0004] Siamese trackers further adopt the Transformer architecture, which greatly enhances the performance of the model. For example, TransT introduces self-attention blocks and cross-attention blocks, which significantly enhance the interaction between the two branches. KeepTrack introduces a learning association network to solve similarity inference. At the same time, online trackers such as discrete cosine transform (DCF), kernel correlation filter (KCF), ATOM, and DiMP use dynamic modeling techniques to adaptively learn object features through continuous online learning. In order to effectively utilize temporal information, ToMP integrates Transformer encoder and decoder modules with DCF to dynamically capture temporal changes. SparseTT adopts the Top-K algorithm to enhance the discriminative ability, while IDGtrack integrates linear attention through the graph model to improve the inference speed. Despite these advances, traditional Transformer-based trackers still lack unique designs for drone scenarios to address the two key factors mentioned in the introduction. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a target tracking method based on local and global attention modeling, the core of which is to propose an attention model, called (Macaron Attention). This method aims to solve the problems of perspective change and scale change existing in drone data that were ignored by previous trackers.

[0006] Two key factors are emphasized when dealing with UAV object tracking tasks: limited UAV viewpoint and motion blur, and drastic changes in object scale. To address these challenges, a dual-stream tracker is chosen as a solution because it is highly flexible and allows the tracking neck to be modified to accommodate these specific factors. To further improve the tracking performance, a new attention mechanism (Macaron Attention) is proposed in this paper. This mechanism is cleverly integrated into the current tracking framework, aiming to hierarchically consider the scale changes of UAVs and significantly enhance the tracker's ability to distinguish UAVs from background distractors. The innovation of the Macaron attention mechanism is that it integrates a fixed window attention module (FWA), a local squeezed global attention module (LSGA), and a conventional global attention module (CGA). The FWA module effectively solves the multi-scale problem in UAV tracking by cropping and labeling within a fixed window scale in the spatial domain. This enables the tracker to focus on the target object more accurately and reduce the interference of background information. At the same time, the LSGA module adopts a label aggregation strategy based on adaptive clustering, which enables the tracker to better adapt to scale changes. By designing a "window-to-window" alignment strategy, LSGA combines global attention with local attention, thereby achieving more accurate target positioning and tracking. Finally, by applying these modules, the present invention retains the original global modeling capability of Transformer. This means that the tracker can still maintain the perception and understanding of global information while paying attention to local details. This combination of global and local attention mechanism makes the tracker of the present invention more robust and accurate when dealing with drone object tracking tasks.

[0007] To achieve the above object, the present invention adopts the following technical solution, a target tracking method based on local and global attention modeling, comprising the following steps: Step 1: Build a backbone network to initially extract features and perform pre-training on a basic classification image set; Step 2: Build a tracking module, including an attention structure with global and local attention capture capabilities and a twin-architecture-based two-stream single target tracking network for tracking and precise positioning; Step 3: Define the optimization method, optimization parameters and loss function; Step 4: Input the fixed initial frame image into the template branch, and extract the invariant template feature map through the backbone network; Step 5: Input the subsequent frames into the search area branch in sequence to achieve stable tracking and precise positioning of the given drone data.

[0008] The beneficial effects of the present invention are as follows: Experiments on drone data show that the proposed tracker integrated with Macaron attention has the most advanced tracking performance in the face of scale change and perspective change challenges. The scale change is adapted by using an aggregation strategy of feature vectors based on adaptive clustering, and a "window-to-window" alignment method is used to seamlessly integrate global and local attention, while retaining the original global modeling capability of attention, effectively enhancing the accuracy and robustness of the tracker in drone tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is an overall flow chart of a target tracking method based on local and global attention of the present invention; Figure 2 Schematic diagram of attention module divided into fixed areas; Figure 3 Schematic diagram of attention module divided into adaptive regions; Figure 4 Qualitative schematic diagram of drone data for the proposed attention implementation of the present invention; Figure 5 Quantitative results of drone data for the proposed attention implementation of the present invention. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above-mentioned purpose, the present invention adopts the following technical scheme.

[0011] like Figure 1 As shown, it is a flow chart of a target tracking method based on local and global attention modeling of the present invention, which specifically includes: Step 1: Build a backbone network to initially extract features and pre-train on a basic classification image set; the process of building a backbone network and pre-training on a classification image set is as follows: Step 1.1: Select the lightweight pure convolution network EfficientNet as the backbone network for feature extraction. The backbone network is divided into 5 different stages: module-1, module-2, module-3, module-4, module-5. The first three stages, module-1, module-2, and module-3, are composed of fused lightweight convolution blocks (Fused MB-Conv) with depthwise separable convolution (DW-Conv). The last two stages, module-4 and module-5, only have lightweight convolution blocks (MB-Conv). The number of corresponding modules in the 5 different stages, block-1 to block-5, is 1:2:2:2:1. Among them, Fused MB-Conv includes 2 residual modules, and each residual module is internally constructed with 3 convolution layers and one SE-Net layer. Figure 1 As shown, the input size in the first stage Starting from (C is the channel dimension, H is the height dimension, and W is the width dimension), the dimension range of the output features of the subsequent stages is as follows: , and Then, the features of the last two stages are fused together. And a convolution operation (Conv) is applied to unify the features of the last two stages. and The channel dimension is ( The default number is 256).

[0012] Step 1.2: Pre-train the backbone network on the classification dataset ImageNet-1K, use the ImageNet-1K dataset to train 500 epochs, and add the Random Erasing data augmentation method; Step 1.3: For the pre-trained backbone network, retain its pre-trained network model parameters and use its optimal model weights as the starting point for fine-tuning. At the same time, adopt a learning rate strategy with a rate decay according to a given step size to ensure the feasibility and efficiency of the network convergence during the fine-tuning process. Step 2: Build a tracking module, which includes an attention module with global and local attention capture capabilities and a twin-architecture-based two-stream single target tracking network for precise tracking and positioning. Figure 1 It can be seen that the features output by the backbone network will be assigned to the template branch (the upper branch of the dual-stream architecture) and the search area branch (the lower branch of the dual-stream architecture). The features of the template branch will pass through two layers of traditional attention, and the features of the search area branch will pass through a layer of attention module based on fixed area division, a layer of attention module based on adaptive area division and a layer of traditional attention. The traditional attention is implemented in the same way as the softmax in step 2.1. The construction process of the dual-stream single target tracking network based on the twin architecture is as follows: Step 2.1: Establish an attention module for fixed region partitioning, such as Figure 2 As shown, the backbone network is first used to extract the network feature map, and then the feature map is divided into non-overlapping square areas according to a predetermined fixed square area, and feature mapping is realized through a linear layer. For each small square area that is divided and mapped, the correlation between the feature vector itself and other feature vectors is calculated through the attention mechanism to generate a similarity attention map, and then the softmax normalization function is used to amplify the difference in similarity, and the difference is used as a similarity measure to calculate the aggregation result of the feature vector. The specific formula is as follows:

[0013] Among them, Q, K, V are linear transformations of the input, The channel dimension is used as the scaling weight, and softmax is the normalization function. is the output result. It is worth noting that this attention is only performed within the divided blocks, such as Figure 2 As shown, the non-overlapping partitioning divides the input into regions of the same size, and the above attention is implemented separately in each region; Step 2.2: Establish an attention module based on adaptive region division. Considering that fixed region division is not capable of extracting information about scale transformation of UAV, an adaptive aggregation algorithm based on attention is adopted. There is no need to pre-define and learn overlapping region blocks of different sizes. Similarly, for each divided region block, the attention mechanism is used again to generate a similarity attention map, and it is restored to the scale of the global general attention size according to the division index. The Hadamard product is applied to the global general attention and the adaptive local attention to obtain a scale-adapted similarity attention map. Finally, the construction of the tracking neck module is completed. The calculation of its attention is consistent with the formula in step 2.1, and the input method is different, such as Figure 3As shown, an effective mechanism is formulated to extract rich semantic details from global attention, thereby giving different clustering regions the ability to effectively address the above obstacles. To illustrate this process, the attention map of each cluster is first spread to a size that matches the global attention map. This step results in the generation of N cluster centers, each of which is able to produce an attention map of the same size. Subsequently, through the cropping operation, a rectangular spread of attention maps is created, which will be mapped into feature vectors of the same size, each of which will use the same softmax attention algorithm as step 2.1, except that the resulting attention map will be expanded, and the spread attention map of each cluster region is filled with the value of the corresponding index, while zero padding is applied to the position where the index is missing. Then, the Hadamard product between each scattered attention map and the global attention map is calculated, and this result will be restored to the size of the unexpanded attention map, followed by subsequent attention operations. This operation illustrates how a unified global attention map guides different clustering regions, and different local attention values ​​squeeze the global attention together at different locations. This global perspective is dynamically integrated into the local region, which helps to improve the tracking performance of the UAV target.

[0014] Step 3: Define the loss function and optimization method to ensure the convergence of network training; The specific settings of the loss function and optimization method are as follows: Step 3.1: For the optimization design of the loss function of the tracking network, the joint optimization of classification loss and regression offset loss is adopted, which is specifically achieved through cross entropy classification loss, intersection-over-union positioning loss and L1 distance metric loss:

[0015] in is the true label, n is the number of categories, is the learned probability.

[0016]

[0017] in is the bounding box predicted by the model, is the true bounding box.

[0018]

[0019] Where N is the number of samples, is the predicted value, is the true value.

[0020] During the optimization process, the AdamW optimizer was used to improve the stability and convergence of the loss function learning, thereby ensuring that the network can identify and locate the target position during the training process; Step 3.2: During the training process, the tracking network selected four datasets, namely COCO, TrackingNet, LaSOT and GOT10k, as training datasets, aiming to improve the generalization and adaptability of the model. When generating training labels, the feature vectors corresponding to the internal pixels in the bounding-box are selected as positive samples, that is, these feature vectors represent the possible positions of the target objects. The feature vectors corresponding to the pixels outside the bounding-box are regarded as negative samples, that is, these feature vectors do not represent the target objects, but the background or other interference information. Through such a division of positive and negative samples, the tracking network can be trained more accurately, enabling it to better identify and track the target objects; Step 3.3: In order to ensure the convergence effect and training efficiency of the network, adjust the initial value and decay strategy (1e-4) of the learning rate according to the size and complexity of the data set. At the same time, through multiple experiments and verifications, the appropriate number of training iterations (200 rounds) was determined to ensure that the network can fully learn and reach a stable convergence state.

[0021] Step 4: Given the reasoning process of the initial frame; the tracking reasoning process in the initial state is specifically as follows: Step 4.1: Input the 0th frame image with a given initial frame and the center position coordinates and width and height parameters of its target, and input it into the template branch of the entire tracking network. After the feature extraction backbone network and the tracking neck module, the invariant features of the 0th frame are extracted; Step 4.2: In the subsequent target tracking task, save the current output template features and use them as the template branch input of the tracking network. In the subsequent tracking process, the feature map of the template branch input will remain unchanged to ensure that the target can be stably tracked in consecutive frames, which helps to improve the stability and accuracy of tracking and avoid tracking loss or drift problems caused by feature changes; Step 5: Reasoning process of subsequent continuous frames; the tracking reasoning process of the subsequent continuous frames is specifically as follows: Step 5.1: Input subsequent continuous frames into the entire tracking network to obtain the feature map corresponding to the branch in the search area; Step 5.2: Measure the similarity between the template branch invariant feature map and the search area variable feature map, and then use the maximum response position in the measurement result as the predicted position of the current output, and restore its position to the original image size to obtain the specific position coordinates; Step 5.3: According to its predicted position coordinates, a square frame is captured at the corresponding position in the next frame, and the captured result is used as the subsequent input image, and this process is repeated until the tracking is completed.

[0022] In this example, the results on the UAV-based dataset demonstrate the state-of-the-art performance of our tracker, notably when faced with scale variations, such as Figure 4 As shown in the first and second rows, the present invention compares the algorithm and the true value (Bounding Box) of the present invention with the existing high-performance algorithms (Transt, TrDiMP, SiamCAR). The attention proposed by the present invention dynamically combines local and global attention, thereby enhancing its ability to adapt to such scale changes. In addition, when faced with the challenging drone perspectives depicted in the third and fourth rows, the tracker of the present invention can accurately track these perspectives, which shows that the integrated fusion of fixed window attention, local squeezed global attention, and traditional global attention modules in the Macaron attention mechanism enables it to effectively cope with various challenges encountered in drone tracking scenarios. At the same time, as Figure 5 As shown in the figure, it describes its tracking effect with high-performance algorithms (ECO, SiamAttn, SiamBAN, TrDiMP, TrSiam, Transt, ATOM, SiamPT) in challenging scenes of similar interference, partial occlusion, background interference and viewpoint change. In particular, in the scene of background interference, the tracker of the present invention shows a significant performance lead. Compared with the second best algorithm TrSiam, the tracker of the present invention has a 10.7% increase in success rate and an 8.4% increase in accuracy. Similarly, when facing partial occlusion, the tracker of the present invention is also better than TrSiam, with a 2.2% increase in success rate and a 1.9% increase in accuracy. In terms of handling viewpoint changes, the tracker of the present invention also performs well, with a 1.1% increase in success rate and a 0.8% increase in accuracy compared to TrSiam.

[0023] The above is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. As long as the drone tracking is performed by using local and global attention, and then the single target tracking implementation method based on attention falls within the protection scope of the present invention.

Claims

1. A target tracking method based on local and global attention modeling, characterized in that: The method comprises the following steps: Step 1: Build a backbone network to initially extract features and perform pre-training on a basic classification image set; Step 2: Build a tracking module, including an attention structure with global and local attention capture capabilities and a twin-architecture-based two-stream single target tracking network for tracking and precise positioning; Step 3: Define the optimization method, optimization parameters and loss function; Step 4: Input the fixed initial frame image into the template branch, and extract the invariant template feature map through the backbone network; Step 5: Input the subsequent frames into the search area branch in sequence to achieve stable tracking and precise positioning of the given drone data.

2. The target tracking method based on local and global attention modeling according to claim 1, characterized in that: In step 1, building the backbone network includes: Step 1.1: Select the lightweight pure convolutional network EfficientNet as the backbone network for feature extraction. The backbone network is divided into 5 different stages: module-1, module-2, module-3, module-4, module-5. The first three stages are composed of fused lightweight convolution blocks with depthwise separable convolutions, and the last two stages have only lightweight convolution blocks. The number of corresponding modules is 1:2:2:2:1; Step 1.2: Pre-train the backbone network on the classification dataset ImageNe-1K; Step 1.3: For the pre-trained network, retain its pre-trained network model parameters and use its optimal model weights as the starting point for fine-tuning. At the same time, adopt a learning rate strategy with a rate decay according to a given step size to ensure the feasibility and efficiency of the network convergence during the fine-tuning process.

3. A target tracking method based on local and global attention modeling according to claim 2, characterized in that: In step 2, building a dual-stream single target tracking network based on a twin architecture for tracking and precise positioning includes: Step 2.1: Establish an attention module based on fixed area division, use the backbone network to extract feature maps, divide the feature maps into non-overlapping square areas according to pre-given fixed square areas, and calculate the correlation between the feature vector itself and other feature vectors in each divided small square area through the attention mechanism to generate a similarity attention map. Then use the softmax normalization function to amplify the difference in similarity, and use the difference as a similarity measure to calculate the aggregation result of the feature vector. The specific formula is as follows: ， Among them, Q, K, V are linear transformations of the input, The channel dimension is used as the scaling weight, and softmax is the normalization function. is the output result; Step 2.2: Establish an attention module based on adaptive region partitioning. First, spread the attention map of each cluster to a size that matches the global attention map to generate N cluster centers. Each cluster center produces an attention map of the same size. Then, through the cropping operation, a rectangular scattered attention map is created. These rectangular scattered attention maps will be mapped into feature vectors of the same size. Each feature vector will use the same attention mechanism calculation method as step 2.

1. The obtained attention map will be expanded, and the scattered attention map of each cluster area is filled with the value of the corresponding index, while zero padding is applied to the position where the index is missing. Then, the Hadamard product between each scattered attention map and the global attention map is calculated. This result will be restored to the size of the unexpanded attention map, followed by subsequent attention operations.

4. The target tracking method based on local and global attention modeling according to claim 3, characterized in that: Step 3 includes: Step 3.1: Loss function optimization design adopts the joint optimization of classification loss and regression offset loss, which is achieved through cross entropy classification loss, intersection-over-union positioning loss and L1 distance metric loss: ； in is the true label, n is the number of categories, is the learned probability; ； in is the bounding box predicted by the model, is the true bounding box; ； Where N is the number of samples, is the predicted value, is the true value; Step 3.2: During the training process, four datasets, COCO, TrackingNet, LaSOT, and GOT10k, were selected as training datasets. When generating training labels, the feature vectors corresponding to the internal pixels in the bounding-box were selected as positive samples, that is, these feature vectors represent the possible positions of the target objects, while the feature vectors corresponding to the external pixels of the bounding-box were regarded as negative samples, that is, these feature vectors do not represent the target objects, but the background or other interference information. Through such positive and negative sample division, the network is trained; Step 3.3: According to the size and complexity of the data set, adjust the initial value and decay strategy of the learning rate and determine the appropriate number of training iterations to ensure that the network can fully learn and reach a stable convergence state.

5. The target tracking method based on local and global attention modeling according to claim 4, characterized in that: The step 4 comprises: Step 4.1: Input the 0th frame image with a given initial frame and the center position coordinates and width and height parameters of its target, input it into the template branch, and extract the invariant features of the 0th frame through the feature extraction backbone network and the tracking neck module; Step 4.2: In the subsequent target tracking task, save the current output template features and use them as template branch input. In the subsequent tracking process, the feature map of the template branch input will remain unchanged.

6. The target tracking method based on local and global attention modeling according to claim 5, characterized in that: The step 5 comprises: Step 5.1: Input subsequent continuous frames into the search area branch to obtain the variable feature map corresponding to the search area branch; Step 5.2: Measure the similarity between the template branch invariant feature map and the search area variable feature map, and then use the maximum response position in the measurement result as the predicted position of the current output, and restore its position to the original image size to obtain the specific position coordinates; Step 5.3: According to its predicted position coordinates, a square frame is captured at the corresponding position in the next frame, and the captured result is used as the subsequent input image, and this process is repeated until the tracking is completed.

Images