Robust target tracking method and system based on graph attention transformer network

Abstract

The application relates to a robust target tracking method and system based on a graph attention Transformer network, which comprises the following steps: constructing a graph attention Transformer network model for a target tracking task and training the same, wherein the network model comprises a feature extraction network, an adaptive graph attention module and a feature prediction head; sampling video sequence frames at different sampling ratios, and taking the sampling samples obtained from a first frame and subsequent frames as templates and search regions respectively; inputting the templates and the search regions into the trained network model, extracting template and search features through the feature extraction network as inputs of the adaptive graph attention module; performing an aggregation operation on the input template and search features through the adaptive graph attention module, then performing feature fusion through a hybrid attention to obtain more robust adaptive features; and finally estimating the target in the search region through the feature prediction head to obtain a tracking result. The method and system are beneficial to obtaining more stable, robust and accurate tracking results.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a robust target tracking method and system based on graph attention Transformer networks. Background Technology

[0002] Visual tracking plays a crucial role in computer vision, aiming to estimate the state of any object in a video frame based on a given initial bounding box. In recent years, object tracking has seen widespread application in fields such as intelligent transportation and video surveillance. However, the performance of existing trackers is affected by various challenging factors, including lighting variations, deformation, motion blur, and background clutter.

[0003] Current mainstream trackers include twin-based trackers and Transformer-based trackers, both of which have achieved good results in terms of efficiency and accuracy. Trackers based on Siamese networks [Bertinetto, L., Valmadre, J., Henriques, JF, Vedaldi, A., Torr, PH: Fullyconvolutionalsiamese networks for object tracking. In: Proceedings of the European Conference on Computer Vision Workshops (ECCVW). pp. 850–865 (2016), Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., Yan, J.: SiamRPN++: Evolution of siamese visual tracking with very deep networks. In: Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPR). pp. 4282–4291(2019), . Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: [Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 8971–8980 (2018)] utilizes cross-correlation to embed information between the template and the search branch. A Transformer-based tracker [X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, H. Lu, Transformer tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 8126–8135.] utilizes global and dynamic modeling capabilities to establish long-distance correlations between the extracted template and the search features.For example, STARK [Yan, B., Peng, H., Fu, J., Wang, D., Lu, H.: Learning spatio-temporal transformer for visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 10428–10437 (2021).] proposed an encoder-decoder transformer architecture to model the global spatio-temporal feature dependencies between the target object and the search region.

[0004] Despite their significant success, they still have some unavoidable drawbacks. Transformer-based trackers can compute global and rich contextual dependencies between the template and the search region. However, the extracted features lack part-level embedding information, making them difficult to adapt to complex tracking scenarios. Furthermore, the template features extracted by traditional trackers may contain too much redundant information, thus accumulating tracking errors. Summary of the Invention

[0005] The purpose of this invention is to provide a robust target tracking method and system based on graph attention Transformer networks, which is beneficial for obtaining more stable, robust and accurate tracking results.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a robust target tracking method based on graph attention Transformer networks, comprising:

[0007] A graph attention Transformer network model for target tracking tasks is constructed and trained. This network model includes a feature extraction network, an adaptive graph attention module, and a feature prediction head.

[0008] The video sequence frames are sampled at different sampling ratios, and the sampled samples obtained from the first frame and subsequent frames are used as templates and search regions, respectively.

[0009] The obtained template and search region are input into the trained network model. The template and search features are extracted by the feature extraction network and used as input to the adaptive graph attention module. The adaptive graph attention module performs aggregation operations on the input template and search features, and then uses hybrid attention to perform feature fusion to obtain more robust adaptive features. Finally, the target in the search region is estimated by the feature prediction head to obtain the tracking result.

[0010] Furthermore, the feature extraction network consists of three stages, each consisting of 21 blocks, and each block contains attention operations. The adaptive graph attention module includes aggregation operations on template nodes and search nodes, and FocusedDropout operations on aggregated search nodes. The feature prediction head is a localization head based on fully convolutional corner points, which uses Conv-BN-ReLU layers to predict the top left and bottom right corners of the target, thereby estimating the bounding box of the tracked target.

[0011] The method for training the graph attention Transformer network model is as follows: training samples are selected from four datasets: LaSOT, TrackingNet, COCO, and GOT-10k, and a dataset loader is established; IOU and L1 are used as training loss functions, and ADAM is used as the optimizer; a training period is set, and the network model is trained using the training samples to obtain the trained network model.

[0012] Furthermore, the specific methods for obtaining the template and search area are as follows:

[0013] For the first labeled frame in the test video sequence, a sampling factor of 2 is set to obtain a template image of size 128*128. Then, the data type is converted from an array to a tensor and used as the input to the network model. For subsequent frames in the test video sequence, a sampling factor of 5 is set to obtain a search region image of size 320*320. Then, the data type is converted from an array to a tensor and used as the subsequent input to the network model.

[0014] Furthermore, the method for extracting templates and searching features using a feature extraction network is as follows: the feature extraction network first processes the input template (H... t *W t *3) and search area (H) s *W s *3) Perform convolution operations to extract features; then perform attention fusion operations on the extracted template and search features to obtain (H t *W t +H s *W s The fusion features of )*C, where H t W t These represent the height and width of the template, respectively, with a size of 128*128, H s W s These represent the height and width of the search region, respectively, with a size of 320*320. C represents the feature channel.

[0015] Furthermore, the specific method for feature fusion using hybrid attention after aggregating the input template and search features through an adaptive graph attention module is as follows:

[0016] First, the template features and search features extracted by the feature extraction network are transformed into two-dimensional feature vectors. Then, an adaptive graph attention module converts the extracted features into feature nodes. The initial template node of the first frame, the dynamically updated template nodes in subsequent frames, and the search node are aggregated. A FocusedDropout operation is then performed on the aggregated search node to obtain adaptive features. Different update thresholds are set for different training datasets; when the set update threshold is reached, the sample is used as a dynamic template to obtain a dynamic template node. To obtain a more accurate template, graph attention aggregation is performed on the initial template node and the dynamic template node to obtain an aggregated template node, helping the network to more accurately find the target in the search region. Next, each feature of the target and the search region is flattened, and a query, key, and value matrix is ​​generated through linear projection. To achieve local spatial context modeling capabilities, hybrid attention is performed on each feature through separate depth-wise convolutional layers to obtain the fused features of the template and the search region. Hybrid attention also performs downsampling operations on the key and value matrices, thereby improving tracking performance.

[0017] Furthermore, the specific method for estimating the target in the search region using the feature prediction head is as follows:

[0018] First, search features are extracted from the output features. Then, the similarity between the search region features and the template features is calculated. The similarity score is multiplied by the search region features to enhance important target regions and weaken regions with poor discriminative power. The resulting adaptive features are input into a fully convolutional network, which consists of stacked Conv-BN-ReLU layers and outputs two probability values, representing the top-left and bottom-right corners of the target bounding box, respectively. Finally, the expectation of the corner probability distribution is calculated to obtain the predicted bounding box coordinates, resulting in more accurate and robust predictions for target tracking.

[0019] Furthermore, the weight of the IOU loss function is 2.0, the weight of the L1 loss function is 5.0, the learning rate is 0.0001, and the weight decay value is 0.0001; during the tracking process, the sampling factor for the template is set to 2.0, and the sampling factor for the search region is set to 5.0.

[0020] The present invention also provides a robust target tracking system based on a graph attention Transformer network, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0021] Compared with the prior art, the present invention has the following beneficial effects: It provides a robust target tracking method and system based on graph attention Transformer network. The method and system construct a graph attention Transformer network. After extracting templates and search features through a feature extraction network, the input templates and search features are aggregated through an adaptive graph attention module. Then, feature fusion is performed using hybrid attention to obtain more robust adaptive features, making the tracker more robust to challenging scenarios, thereby obtaining more stable, robust and accurate tracking results. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the method implementation of an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the adaptive graph attention module in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] like Figure 1 As shown, this embodiment provides a robust target tracking method based on a graph attention Transformer network, including the following steps:

[0028] 1) Construct and train a graph attention Transformer network model for target tracking tasks.

[0029] This network model includes a feature extraction network, an adaptive graph attention module, and a feature prediction head. The feature extraction network consists of a patch embedding operation, which transforms the input template and search features into a token vector. It comprises three stages, each with 21 blocks, and each block includes an attention operation. Figure 2 As shown, the adaptive graph attention module includes aggregation operations on template nodes and search nodes, and a FocusedDropout operation on aggregated search nodes. Its main function is to convert input features into node features, which are then aggregated to generate adaptive features. The feature prediction head is a fully convolutional corner-based localization head that uses a Conv-BN-ReLU layer to predict the top-left and bottom-right corners of the target, thereby estimating the bounding box of the tracked target.

[0030] The method for training the graph attention Transformer network model is as follows: training samples are selected from four datasets: LaSOT, TrackingNet, COCO, and GOT-10k, and a dataset loader is established; IOU and L1 are used as training loss functions, and ADAM is used as the optimizer; 200 training epochs are set, and the network model is trained using training samples to obtain the trained network model.

[0031] In this embodiment, the weight of the IOU loss function is 2.0, the weight of the L1 loss function is 5.0, the learning rate is 0.0001, and the weight decay value is 0.0001; during the tracking process, the sampling factor for the template is set to 2.0, and the sampling factor for the search region is set to 5.0.

[0032] 2) Sample the video sequence frames using different sampling ratios, using the samples obtained from the first frame and subsequent frames as templates and search regions, respectively. The specific method is as follows:

[0033] For the first labeled frame in the test video sequence, a sampling factor of 2 is set to obtain a template image of size 128*128. Then, the data type is converted from an array to a tensor and used as the input to the network model. For subsequent frames in the test video sequence, a sampling factor of 5 is set to obtain a search region image of size 320*320. Then, the data type is converted from an array to a tensor and used as the subsequent input to the network model.

[0034] 3) Input the obtained template and search region into the trained network model, and use the feature extraction network to extract the template and search features as input to the adaptive graph attention module; use the adaptive graph attention module to aggregate the input template and search features, and then use hybrid attention to perform feature fusion to obtain more robust adaptive features; finally, use the feature prediction head to estimate the target in the search region to obtain the tracking result.

[0035] The method for extracting templates and searching features using a feature extraction network is as follows: The feature extraction network first processes the input template (H)... t *W t *3) and search area (H) s *Ws *3) Perform convolution operations to extract features; then perform attention fusion operations on the extracted template and search features to obtain (H t *W t +H s *W s The fusion features of )*C, where H t W t These represent the height and width of the template, respectively, with a size of 128*128, H s W s These represent the height and width of the search region, respectively, with a size of 320*320. C represents the feature channel.

[0036] The specific method for aggregating the input template and search features using an adaptive graph attention module, and then using hybrid attention for feature fusion, is as follows:

[0037] First, the template features and search features extracted by the feature extraction network are transformed (reshaped) into two-dimensional feature vectors. Then, an adaptive graph attention module converts the extracted features into feature nodes. The initial template node of the first frame, the dynamically updated template nodes in subsequent frames, and the search node are aggregated. A FocusedDropout operation is then performed on the aggregated search node to obtain adaptive features. Different update thresholds are set for different training datasets; when the set update threshold is reached, the sample is used as a dynamic template to obtain dynamic template nodes. To obtain more accurate templates, graph attention aggregation is performed on the initial template nodes and dynamic template nodes to obtain aggregated template nodes, helping the network to more accurately find targets in the search region. Next, each feature of the target and search region is flattened, generating query, key, and value matrices through linear projection. To achieve local spatial context modeling capabilities, hybrid attention is performed on each feature through separate depth-wise convolutional layers to obtain the fused features of the template and search region. Hybrid attention also performs downsampling operations on the key and value matrices, thereby improving tracking performance.

[0038] The specific method for estimating the target in the search region using a feature prediction head is as follows:

[0039] First, search features are extracted from the output features. Then, the similarity between the search region features and the template features is calculated. The similarity score is multiplied by the search region features to enhance important target regions and weaken regions with poor discriminative power. The resulting adaptive features are input into a fully convolutional network, which consists of stacked Conv-BN-ReLU layers and outputs two probability values, representing the top-left and bottom-right corners of the target bounding box, respectively. Finally, the expectation of the corner probability distribution is calculated to obtain the predicted bounding box coordinates, resulting in more accurate and robust predictions for target tracking.

[0040] This embodiment also provides a robust target tracking system based on a graph attention Transformer network, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the above-described method steps.

[0041] In this embodiment, the OTB100 dataset was used for comparative verification. Table 1 shows the comparison results between the proposed method and other target tracking methods on the OTB100 dataset. As can be seen from Table 1, the method of this invention has higher accuracy and robustness compared to other target tracking methods, specifically in terms of optimal accuracy and success rate.

[0042] Table 1

[0043]

[0044] In Table 1, STARK-S corresponds to the method proposed by B. Yan et al. (Yan, B., Peng, H., Fu, J., Wang, D., Lu, H.: Learning spatio-temporal transformer for visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 10428–10437 (2021).).

[0045] DaSiamRPN corresponds to the method proposed by Z. Zhu et al. (Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, W. Hu, Distractor-aware siamese networks for visual object tracking, in: Proceedings of the European Conference on Computer Vision, Vol. 11213, 2018, pp. 103–119.).

[0046] SiamRPN corresponds to the method proposed by B. Li et al. (B. Li, J. Yan, W. Wu, Z. Zhu, X. Hu, High performance visual tracking with siamese region proposal network, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 8971–8980.).

[0047] GradNet corresponds to the method proposed by P. Li et al. (P. Li, B. Chen, W. Ouyang, D. Wang, X. Yang, H. Lu, Gradnet: Gradientguided network for visual object tracking, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 6162–6171.);

[0048] DeepSRDCF corresponds to the method proposed by M. Danelljan et al. (M. Danelljan, G. Hager, F. Shahbaz Khan, M. Felsberg, Convolutional features for correlation filter-based visual tracking, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2015, pp. 58–66.).

[0049] SiamDW corresponds to the method proposed by Z. Zhang et al. (Z. Zhang, H. Peng, Deeper and wider siamese networks for real-time visual tracking, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4591–4600.).

[0050] SiamFC corresponds to the method proposed by L. Bertinetto et al. (L. Bertinetto, J. Valmadre, JF Henriques, A. Vedaldi, PH Torr, Fully-convolutional siamese networks for object tracking, in: Proceedings of the European Conference on ComputerVision Workshops, 2016, pp. 850–865.).

[0051] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0052] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0053] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0054] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0055] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A robust target tracking method based on graph attention Transformer networks, characterized in that, include: A graph attention Transformer network model for target tracking is constructed and trained. This network model includes a feature extraction network, an adaptive graph attention module, and a feature prediction head. The feature extraction network consists of a patch embedding operation, which transforms the input template and search features into token vectors. It consists of three stages, with each block in each stage containing an attention operation. The adaptive graph attention module includes aggregation operations on template nodes and search nodes, and a FocusedDropout operation on aggregated search nodes. It mainly transforms the input features into node features, and then generates adaptive features through aggregation operations. The feature prediction head is a fully convolutional corner-based localization head, which uses Conv-BN-ReLU layers to predict the top-left and bottom-right corners of the target, thereby estimating the bounding box of the tracked target. The video sequence frames are sampled at different sampling ratios, and the sampled samples obtained from the first frame and subsequent frames are used as templates and search regions, respectively. The obtained template and search region are input into the trained network model. The template and search features are extracted by the feature extraction network and used as input to the adaptive graph attention module. The adaptive graph attention module performs aggregation operations on the input template and search features, and then uses hybrid attention to perform feature fusion to obtain more robust adaptive features. Finally, the target in the search region is estimated by the feature prediction head to obtain the tracking result.

2. The robust target tracking method based on graph attention Transformer network according to claim 1, characterized in that, The feature extraction network consists of three stages, each consisting of 21 blocks. Each block contains attention operations. The adaptive graph attention module includes aggregation operations on template nodes and search nodes, and FocusedDropout operations on aggregated search nodes. The feature prediction head is a localization head based on fully convolutional corner points, which uses Conv-BN-ReLU layers to predict the top left and bottom right corners of the target, thereby estimating the bounding box of the tracked target. The method for training the graph attention Transformer network model is as follows: training samples are selected from four datasets: LaSOT, TrackingNet, COCO, and GOT-10k, and a dataset loader is established; IOU and L1 are used as training loss functions, and ADAM is used as the optimizer. Set a training period, train the network model using training samples, and obtain a trained network model.

3. The robust target tracking method based on graph attention Transformer network according to claim 1, characterized in that, The specific methods for obtaining the template and search area are as follows: For the first labeled frame in the test video sequence, a sampling factor of 2 is set to obtain a template image of size 128*128. Then, the data type is converted from an array to a tensor and used as the input to the network model. For subsequent frames in the test video sequence, a sampling factor of 5 is set to obtain a search region image of size 320*320. Then, the data type is converted from an array to a tensor and used as the subsequent input to the network model.

4. The robust target tracking method based on graph attention Transformer network according to claim 2, characterized in that, The method for extracting templates and searching features using a feature extraction network is as follows: The feature extraction network first processes the input template (H... t *W t *3) and search area (H) s *W s *3) Perform convolution operations to extract features; then perform attention fusion operations on the extracted template and search features to obtain (H t *W t +H s *W s The fusion features of )*C, where H t W t These represent the height and width of the template, respectively, with a size of 128*128, H s W s These represent the height and width of the search region, respectively, with a size of 320*320. C represents the feature channel.

5. The robust target tracking method based on graph attention Transformer network according to claim 2, characterized in that, The specific method for aggregating the input template and search features using an adaptive graph attention module, and then using hybrid attention for feature fusion, is as follows: First, the template features and search features extracted by the feature extraction network are transformed into two-dimensional feature vectors. Then, an adaptive graph attention module converts the extracted features into feature nodes. The initial template node of the first frame, the dynamically updated template nodes in subsequent frames, and the search node are aggregated. A FocusedDropout operation is then performed on the aggregated search node to obtain adaptive features. Different update thresholds are set for different training datasets. When the set update threshold is reached, the sample is used as a dynamic template to obtain dynamic template nodes. To obtain more accurate templates, graph attention aggregation is performed on the initial template nodes and dynamic template nodes to obtain aggregated template nodes, helping the network to more accurately find targets in the search region. Then, each feature of the target and search region is flattened, and a query, key, and value matrix is ​​generated through linear projection. To achieve the ability to model local spatial context, hybrid attention is performed on each feature through separate depth-wise convolutional layers to obtain the fused features of the template and search region. Hybrid attention also performs downsampling on the key and value matrices, thereby improving tracking performance.

6. The robust target tracking method based on graph attention Transformer network according to claim 2, characterized in that, The specific method for estimating the target in the search region using a feature prediction head is as follows: First, search region features are extracted from adaptive features. Then, the similarity between search region features and template features is calculated. The similarity score is multiplied by the search region features to enhance important target regions and weaken regions with poor discriminative power. The obtained adaptive features are input into a fully convolutional network, which consists of stacked Conv-BN-ReLU layers and outputs two probability values, representing the top left and bottom right corners of the target bounding box, respectively. Finally, the expectation of the corner probability distribution is calculated to obtain the predicted bounding box coordinates, thus generating more accurate and robust predictions for target tracking.

7. The robust target tracking method based on graph attention Transformer network according to claim 2, characterized in that, The IOU loss function has a weight of 2.0, the L1 loss function has a weight of 5.0, the learning rate is 0.0001, and the weight decay value is 0.0001. During the tracking process, the sampling factor for the template is set to 2.0, and the sampling factor for the search region is set to 5.

0.

8. A robust target tracking system based on a graph attention Transformer network, characterized in that, It includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor, which, when executed by the processor, enable the implementation of the method as described in any one of claims 1-7.

Images