A target tracking method based on adaptive hierarchical graph attention feature fusion

Abstract

The application aims to provide a target tracking method based on adaptive hierarchical graph attention feature fusion, and the specific method is as follows: on the basis of a light network ShuffleNet V2, an adaptive feature map refinement module is adopted, the expression ability of different scale features is improved by learning the linear combination of multiple convolution kernels and the attention thereof; a hierarchical graph attention module is adopted, the accurate transmission of corresponding local information between different feature maps is realized, so as to cope with the situations such as target partial occlusion and target deformation, thereby guaranteeing the accuracy and real-time performance of tracking. The application can balance the relationship between tracking accuracy and speed, and reduce the influence of situations such as target partial occlusion and target deformation on the target tracking effect.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and object tracking, and relates to an object tracking method based on adaptive hierarchical graph attention feature fusion. Background Technology

[0002] Object tracking, as one of the fundamental problems in computer vision research, has received increasing attention. Currently, existing deep learning-based object tracking techniques are susceptible to factors such as partial object occlusion and object deformation, leading to a decrease in tracking accuracy. Furthermore, as networks become increasingly large, the time and space complexity of tracking algorithms also increases, making it more difficult to deploy tracking algorithms on mobile devices. Therefore, finding a high-precision, lightweight tracking method is one of the urgent problems to be solved in practical applications.

[0003] Existing Siamese network tracking algorithms employ classic convolutional neural networks such as AlexNet and ResNet50 as backbones to extract robust visual features and effectively improve tracking accuracy by introducing different functional modules. However, this leads to a problem: as the network size increases, the algorithm's running speed decreases significantly. Using only lightweight networks as the backbone, while improving speed, fails to guarantee tracking accuracy. Furthermore, Siamese network tracking algorithms use the same static convolutional kernels for all data, ignoring differences between data sets. Additionally, these algorithms treat the features of the search region and template region as a whole for similarity matching, ignoring the correspondence between their local features, making the target tracking results susceptible to effects such as target occlusion and deformation. To further improve the model's robustness, more convolutional layers or an increase in the number of input / output channels in convolutional layers are needed, which undoubtedly increases the complexity of the network model, leading to a decrease in tracking speed.

[0004] In recent years, the Transformer model has been widely used in object tracking, showing great potential in improving tracking performance. The Transformer model's self-attention mechanism can effectively learn contextual information in images, thus enabling more accurate tracking. However, the Transformer model's self-attention mechanism requires significant computational resources, has a large number of model parameters, and results in a relatively slow algorithm execution speed.

[0005] Current research findings both domestically and internationally suffer from an imbalance between tracking accuracy and speed, making it difficult to meet the real-time requirements of mobile devices. This invention, based on the lightweight network ShuffleNet V2, employs an adaptive feature map refinement module. By learning linear combinations of multiple convolutional kernels and their attention mechanisms, it enhances the expressive power of features at different scales. Furthermore, it utilizes a hierarchical map attention module to accurately transfer corresponding local information between different feature maps, addressing situations such as target occlusion and deformation, thereby ensuring both tracking accuracy and real-time performance. Summary of the Invention

[0006] The purpose of this invention is to provide a target tracking method based on adaptive hierarchical graph attention feature fusion, which aims to balance the relationship between tracking accuracy and speed, while reducing the impact of target partial occlusion and target deformation on the target tracking effect.

[0007] To achieve the above objectives, this method, based on the lightweight network ShuffleNet V2, employs an adaptive feature map refinement module. By learning a linear combination of multiple convolutional kernels and their attention, it enhances the expressive power of features at different scales. A hierarchical map attention module is used to accurately transfer corresponding local information between different feature maps, addressing situations such as target occlusion and target deformation, thereby ensuring tracking accuracy and real-time performance. Specifically, the method includes the following steps: S1: For the input video sequence, the size of the template region in its first frame is set through processing. And the size of the search region in subsequent frames is set through processing. The template feature map is input into the ShuffleNet V2 backbone network of the template branch and the search branch in chronological order for feature extraction. Then, the template feature map output from any three stages of ShuffleNet V2 in the template branch and the search feature map output from the three stages of ShuffleNet V2 corresponding to the template feature map in the search branch are respectively input into the adaptive feature map refinement module.

[0008] S2: This step is the adaptive feature map refinement module proposed in this invention.

[0009] For the template branch, take any three template feature maps extracted from stages 2, 3, 4, and 5 of the ShuffleNet V2 network. P ti The inputs are fed into the adaptive feature map refinement module, where the attention weights for the input and output channels of each convolutional kernel, as well as the attention weights assigned to that kernel, are calculated in parallel using an attention mechanism. The input template feature map is then enhanced using the following formula to obtain the enhanced template feature map:

[0010] in, For the first branch in the template The first stage used One convolutional kernel, This represents the number of convolution kernels. In the template branch... stage, For the first Dynamic attention for each convolutional kernel, For the first Attention weights for the input channels of each convolutional kernel. For the first Attention weights for the output channels of each convolutional kernel. The representative is the first Attention weights assigned to each convolutional kernel, For dynamic convolution kernels, This represents multiplication operations along different dimensions of the kernel space. This represents the convolution operation.

[0011] For the search branch, three search feature maps are extracted from the three stages in the ShuffleNet V2 network that are the same as the stages selected in the template branch mentioned above. P si The inputs are fed into the adaptive feature map refinement module, where the attention weights for the input and output channels of each convolutional kernel, as well as the attention weights assigned to that kernel, are calculated in parallel using an attention mechanism. The input search feature map is then enhanced using the following formula to obtain the enhanced search feature map:

[0012] in, For the first branch in the search The first stage used One convolutional kernel, This represents the number of convolution kernels. In the template branch... stage, For the first Dynamic attention for each convolutional kernel, For the first Attention weights for the input channels of each convolutional kernel. For the first Attention weights for the output channels of each convolutional kernel. The representative is the first Attention weights assigned to each convolutional kernel, For dynamic convolution kernels, This represents multiplication operations along different dimensions of the kernel space. This indicates a convolution operation.

[0013] S3: Combine the enhanced template feature map and the enhanced search feature map obtained in S2 at the same stage and input them into the hierarchical graph attention module. The specific steps are as follows: Assuming the template branch and the search branch are selected as follows: a , b , c Three stages, among which a < b <c The enhanced template feature map and the enhanced search feature map from the same stage are combined into three groups. Each group is then processed by a graph attention module to obtain three feature maps fused from the same stage. ,in Then, the three feature maps Each The grid is a node, in which Indicates the number of feature channels, let V p for The corresponding set of nodes, and using a complete bipartite graph. To model feature maps The hierarchical relationship between nodes, where , For each pair of nodes ,use express Middle node and Middle node The cross-correlation score is calculated using the following formula:

[0014] in, and They are Middle node and Middle node To adaptively learn the relationships between nodes, the node features are first linearly transformed, and then the inner product of the transformed feature vectors is used to calculate the correlation score. The specific formula is as follows:

[0015] in, and These are all linear transformation matrices. To facilitate comparison of coefficients between different nodes, the Softmax function is used. Normalization was performed, and the formula is as follows:

[0016] definition subgraph, , , ,use All nodes are passed to The first in Attention to each node, calculate the node The aggregation representation is shown in the following formula:

[0017] in, It is a linear transformation matrix. It combines aggregate features with node features. The fusion process yields a more robust feature representation, as shown in the following formula:

[0018] in, This represents vector concatenation, followed by parallel computation of all... and The resulting feature map, obtained by concatenating the two vectors, is denoted as follows: , Then , and The three feature maps are concatenated and connected through a grid of size [size missing]. The convolution kernels are used to obtain the final fused feature map. ; S4: Convert the feature map obtained in step S3 into... Input two branches: classification and regression. The classification branch is used to predict the target location, and the regression branch is used to predict the target bounding box information corresponding to that location. The specific steps are as follows: feature map After input, the classification branch outputs a classification confidence plot. This is used to predict the score for each position, and the position with the highest score is selected as the target position; the regression branch outputs a regression response plot. This is used to predict the target bounding box information corresponding to each location in the confidence map. At this point, target tracking in the current frame ends. S5: Repeat steps S1 to S4 until every frame of the video sequence has been processed, and the entire tracking task is completed.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention, based on the lightweight network ShuffleNet V2, employs an adaptive feature map refinement module. By learning a linear combination of multiple convolutional kernels and their attention, it enhances the expressive power of features at different scales, achieving high tracking accuracy while ensuring tracking speed, and also enhancing the robustness of the model.

[0020] This invention employs an improved layered graph attention module, which enables the transfer of corresponding local information between different feature maps, overcoming the difficulty of global information in dealing with target occlusion and target deformation, and further improving tracking accuracy. Attached Figure Description

[0021] Figure 1 This is a flowchart of a target tracking method based on adaptive hierarchical graph attention feature fusion proposed in this invention.

[0022] Figure 2 This is a schematic diagram of the adaptive feature map refinement module proposed in this invention.

[0023] Figure 3 This is a flowchart illustrating the tracking process of the method of this invention.

[0024] Table 1 shows a performance comparison of the method of this invention with other target tracking algorithms on the LaSOT dataset.

[0025] Table 2 is a comparison table of the model parameter quantity, number of floating-point operations, and frame rate of the method of the present invention with other target tracking algorithms.

[0026] Table 3 is a performance comparison table of the method of the present invention and other target tracking algorithms under the condition of partial target occlusion in the LaSOT dataset.

[0027] Table 4 compares the performance of the method of this invention with other target tracking algorithms under target deformation conditions on the LaSOT dataset. Specific Implementation To enable those skilled in the art to better understand this invention, the specific implementation details and methods of this invention will be described in detail. However, it should be clarified that this invention is not limited to the specific embodiments described; any creations that benefit from the inventive concept are within the scope of legal protection, provided that some technical changes are within the scope of the claims.

[0029] like Figure 1 As shown, a target tracking method based on adaptive hierarchical graph attention feature fusion is implemented as follows: First, input a video sequence, and then process the template region size in its first frame to set it to... And the size of the search region in subsequent frames is set through processing. Feature extraction is performed on the ShuffleNet V2 backbone network, which is input into the template branch and search branch in chronological order. In the template branch, after any three stages of ShuffleNet V2, this embodiment selects three adjacent stages, namely stages 2, 3, and 4, whose output template feature maps have sizes of [sizes missing]. , and In the search branch, the same stages as the template branch are selected, namely stages 2, 3, and 4. The sizes of the search feature maps output from stages 2, 3, and 4 of ShuffleNet V2 are respectively... , and Then, the above six feature maps are input into the adaptive feature map refinement module.

[0030] The adaptive feature map thinning module proposed in this invention, such as Figure 2 As shown.

[0031] For the template branch, the three feature maps extracted from stages 2, 3, and 4 of the ShuffleNet V2 network are used. The inputs are fed into the adaptive feature map refinement module, where the attention weights of the input and output channels of each convolutional kernel are calculated in parallel using an attention mechanism, along with the attention weights assigned to that kernel. The input feature map is then enhanced using the following formula.

[0032]

[0033] in, For the first branch in the template The first stage used One convolutional kernel, This represents the number of convolution kernels. In the template branch... stage, For the first Dynamic attention for each convolutional kernel, For the first Attention weights for the input channels of each convolutional kernel. For the first Attention weights for the output channels of each convolutional kernel. The representative is the first Attention weights assigned to each convolutional kernel, For dynamic convolution kernels, This represents multiplication operations along different dimensions of the kernel space. This represents the convolution operation. In the experiment, we take... .

[0034] After the adaptive feature map refinement module, this invention obtains three enhanced feature maps of the template branch. The sizes of the three enhanced template feature maps are respectively , and This enhances the expressive power of features.

[0035] For the search branch, the three feature maps extracted from stages 2, 3, and 4 of the ShuffleNet V2 network are used. The inputs are fed into the adaptive feature map refinement module, where the attention weights for the input and output channels of each convolutional kernel are calculated in parallel using an attention mechanism, along with the attention weights assigned to that kernel. The input feature map is then enhanced using the following formula:

[0036] in, For the first branch in the search The first stage used One convolutional kernel, This represents the number of convolutional kernels. In the template branch... stage, For the first Dynamic attention for each convolutional kernel, For the first Attention weights for the input channels of each convolutional kernel. For the first Attention weights for the output channels of each convolutional kernel. The representative is the first Attention weights assigned to each convolutional kernel, For dynamic convolution kernels, This represents multiplication operations along different dimensions of the kernel space. This represents the convolution operation. In the experiment, we take... .

[0037] After the adaptive feature map refinement module, the present invention obtains three feature maps for the search branch as follows: The sizes of the three enhanced search feature maps are respectively , and .

[0038] The hierarchical graph attention module proposed in this invention enhances the feature map with template branch and search branch. and , and , and The inputs are then fed into the hierarchical graph attention module. The specific steps are as follows: Enhanced feature maps of template branch and search branch and , and , and After passing through the two-graph attention module in "Guo D, Shao Y, Cui Y, et al. Graph attention tracking[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 9543-9552.", the feature map after feature fusion is obtained as follows. The sizes of the three fused feature maps are respectively , and .

[0039] Then, the three fused feature maps Each The grid is a node, in which This represents the number of feature channels. Let... They are respectively The corresponding set of all nodes, and using a complete bipartite graph. This is used to model the hierarchical relationship of nodes between two feature maps. , , For each pair of nodes The present invention uses express Middle node and Middle node The cross-correlation score is calculated using the following formula:

[0040] in and They are Middle node and Middle node To adaptively learn better representations between nodes, this invention first applies a linear transformation to the node features, and then calculates the correlation score by taking the inner product of the transformed feature vectors. The specific operation formula is as follows:

[0041] in, and These are all linear transformation matrices. To facilitate comparison of coefficients between different nodes, the Softmax function is used. The formula has been normalized and is as follows:

[0042] definition subgraph, , , ,use All nodes are passed to The first in Attention to each node, calculate the node The aggregation representation is shown in the following formula:

[0043] in, It is a linear transformation matrix. It combines aggregate features with node features. The fusion process yields a more robust feature representation, as shown in the following formula:

[0044] in, This represents vector concatenation. Then, all computations are performed in parallel. and Two sizes are obtained. The feature maps are denoted as follows: , Then , and The three feature maps are concatenated and connected through a grid of size [size missing]. The convolution kernels are used to obtain the final fused feature map. Its size is .

[0045] Finally, the feature map The input consists of two branches: classification and regression. The classification branch predicts the target location, and the regression branch predicts the target bounding box information corresponding to that location. The classification and regression branches used in this invention are the same as those in "GuoD, Wang J, Cui Y, et al. SiamCAR: Siamese fully convolutional classification and regression for visual tracking[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition. 2020: 6269-6277." The specific steps are as follows: feature map After input, the classification branch outputs a classification confidence plot. This is used to predict the score for each position, and the position with the highest score is selected as the target position; the regression branch outputs a regression response plot. This is used to predict the target bounding box information corresponding to each location. At this point, target tracking in the current frame ends.

[0046] Then repeat the above steps until every frame of the video sequence has been processed, at which point the entire tracking task is complete.

[0047] To verify the reliability and effectiveness of the present invention, the present invention will be further described below with reference to examples.

[0048] This experiment is based on the Ubuntu 18.04LTS operating system, with a computer configuration of Intel i9 CPU, 64GB RAM, and an RTX 4090 graphics card. The algorithm platform is the PyTorch 1.7.1 deep learning framework, programmed using Python 3.7. Taking the test results on the LaSOT dataset as an example, the tracking success rate (AUC), normalized precision, and precision are evaluated using the overall situation, the target being partially occluded, and the target being deformed.

[0049] Figure 3This diagram illustrates the tracking process of the method of the present invention. Even with interference from numerous objects similar to the target, the method of the present invention can stably track the selected target and is minimally affected by target deformation and scale changes.Table 1 shows the results of the method of the present invention and "Chen X, Yan B, Zhu J, et al. IEEE / CVF conference on computer vision and pattern recognition. 2019: 4282-4291." (SiamRPN++ for short), "Bhat G, Danelljan M, Gool LV, et al. Learning discriminative model prediction for tracking[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision. 2019: 6182-6191." (DiMP50 for short), "Guo D, Shao Y, Cui Y, et al. Graph attention Performance comparison of "tracking[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 9543-9552." (abbreviated as SiamGAT) and "Chen Z, Zhong B, Li G, et al. SiamBAN: Target-awaretracking with Siamese box adaptive network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 5158-5173." (abbreviated as SiamBAN) on the LaSOT dataset.Table 1 below compares the performance of the method of this invention with other target tracking algorithms on the LaSOT dataset, and Table 2 compares the model parameter count, number of floating-point operations, and frame rate of the method of this invention with other target tracking algorithms. Combining Tables 1 and 2, the TransT algorithm has the highest tracking success rate, normalized accuracy, and accuracy, but its model parameter count and computational load are very large, resulting in a slower tracking speed. The present invention can better balance the relationship between tracking accuracy and speed.

[0050]

[0051]

[0052] The method of this invention uses the lightweight network ShuffleNet V2 as the backbone network, thus the number of model parameters and computational cost is very small, and the running speed is very fast, enabling it to run on mobile devices. Table 3 below compares the performance of the method of this invention with other target tracking algorithms under the condition of partial target occlusion on the LaSOT dataset, and Table 4 compares the performance of the method of this invention with other target tracking algorithms under the condition of target deformation on the LaSOT dataset.

[0053]

[0054]

[0055] As shown in Tables 3 and 4, the tracking success rate, normalized accuracy, and precision of the method of this invention are relatively good when the target is occluded or deformed. This is because, when the target is occluded or deformed, the adaptive feature map refinement module can improve the expressive ability of the target's features at different scales by learning the linear combination of multiple convolutional kernels and attention. At the same time, the hierarchical map attention module realizes the transfer of local information between different feature maps, overcoming the difficulty of global information transfer in dealing with scenarios where the target scale and shape change. When tracking occluded or deformed targets, it can accurately obtain the target's features, thus achieving accurate tracking. However, other algorithms besides TransT have poor tracking accuracy and tracking success rate when tracking occluded or deformed targets. Meanwhile, as shown in Table 2, the method of this invention has the fastest tracking speed and the smallest number of model parameters and model computation among all compared algorithms, indicating that the method of this invention achieves a good balance between tracking accuracy and tracking speed.

[0056] The embodiments described above are merely illustrative of specific implementations of the present invention and should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A target tracking method based on adaptive hierarchical graph attention feature fusion, characterized in that, The method is based on the lightweight network ShuffleNet V2, employing an adaptive feature map refinement module. By learning a linear combination of multiple convolutional kernels and their attention, it enhances the expressive power of features at different scales. A hierarchical map attention module is used to accurately transfer corresponding local information between different feature maps, addressing situations such as target occlusion and target deformation, thereby ensuring tracking accuracy and real-time performance. Specifically, it includes the following steps: S1: For the input video sequence, the template region in the first frame and the search region in the subsequent frames are input into the ShuffleNet V2 backbone network of the template branch and the search branch respectively in chronological order for feature extraction. Then, the template feature map output from any three stages of ShuffleNet V2 in the template branch and the search feature map output from the three stages of ShuffleNet V2 corresponding to the template feature map in the search branch are respectively input into the adaptive feature map refinement module. S2: The adaptive feature map refinement module mentioned in step S1 is specifically as follows: For the template branch, extract three template feature maps from any three stages (2, 3, 4, and 5) of the ShuffleNet V2 network. P ti The inputs are fed into the adaptive feature map refinement module, where the attention weights for the input and output channels of each convolutional kernel, as well as the attention weights assigned to that kernel, are calculated in parallel using an attention mechanism. The input template feature map is then enhanced using the following formula to obtain the enhanced template feature map: ; in, For the first branch in the template The first stage used One convolutional kernel, Represents the number of convolution kernels, in the template branch. stage, For the first Dynamic attention for each convolutional kernel, For the first Attention weights for the input channels of each convolutional kernel. For the first Attention weights for the output channels of each convolutional kernel. The representative is the first Attention weights assigned to each convolutional kernel, For dynamic convolution kernels, This represents multiplication operations along different dimensions of the kernel space. Indicates the convolution operation; For the search branch, three search feature maps are extracted from the three stages in the ShuffleNet V2 network that are the same as the selected stages in the template branch mentioned above. P si The inputs are fed into the adaptive feature map refinement module, where the attention weights for the input and output channels of each convolutional kernel, as well as the attention weights assigned to that kernel, are calculated in parallel using an attention mechanism. The input search feature map is then enhanced using the following formula to obtain the enhanced search feature map: ; in, For the first branch in the search The first stage used One convolutional kernel, Represents the number of convolution kernels, in the template branch. stage, For the first Dynamic attention for each convolutional kernel, Representing the Attention weights for the input channels of each convolutional kernel. Representing the Attention weights for the output channels of each convolutional kernel. The representative is the first Attention weights assigned to each convolutional kernel, For dynamic convolution kernels, This represents multiplication operations along different dimensions of the kernel space. Indicates the convolution operation; S3: Combine the enhanced template feature map and the enhanced search feature map obtained in S2 at the same stage and input them into the hierarchical graph attention module. The specific steps are as follows: Assuming the template branch and the search branch are selected as follows: a , b , c Three stages, among which a < b <c The enhanced template feature map and the enhanced search feature map from the same stage are combined into three groups. Each group is then processed by a graph attention module to obtain three feature maps fused from the same stage. ,in Then, the three feature maps Each The grid is a node, in which Indicates the number of feature channels, let V p for The corresponding set of nodes, and using a complete bipartite graph. To model feature maps The hierarchical relationship between nodes, where , For each pair of nodes ,use express Middle node and Middle node The cross-correlation score is calculated using the following formula: ; in and They are Middle node and Middle node To adaptively learn the relationships between nodes, the node features are first linearly transformed, and then the inner product of the transformed feature vectors is used to calculate the correlation score. The specific formula is as follows: ; in, and These are all linear transformation matrices. To facilitate comparison of coefficients between different nodes, the Softmax function is used. The normalization operation is performed using the following formula: ; definition subgraph, , , ,use All nodes are passed to The first in Attention to each node, calculate the node The aggregation representation is shown in the following formula: ; in, It is a linear transformation matrix that combines aggregate features with node features. The fusion process yields a more robust feature representation, as shown in the following formula: ; in, This represents vector concatenation, followed by parallel computation of all... and The resulting feature map, obtained by concatenating the two vectors, is denoted as follows: , Then , and The three feature maps are concatenated and connected through a grid of size [size missing]. The convolution kernels are used to obtain the final fused feature map. ; S4: Convert the feature map obtained in step S3 into... Input two branches: classification and regression. The classification branch is used to predict the target location, and the regression branch is used to predict the target bounding box information corresponding to that location. The specific steps are as follows: feature map After input, the classification branch outputs a classification confidence plot. This is used to predict the score for each position, and the position with the highest score is selected as the target position; the regression branch outputs a regression response plot. This is used to predict the target bounding box information corresponding to each location. At this point, target tracking in the current frame ends. S5: Repeat steps S1 to S4 until every frame of the video sequence has been processed, and the entire tracking task is completed.

Images