Cascading type single-target visual tracking method based on spatio-temporal information fusion

Abstract

The invention relates to the technical field of computer vision, in particular to a cascading type single-target visual tracking method based on spatio-temporal information fusion, which comprises the following steps of: firstly, respectively carrying out feature extraction on a static template, a dynamic template and a search area of a tracking target by using a ResNet network to obtain feature mapping graphs of three branches; and converting the two-dimensional feature map into a one-dimensional feature sequence. And then fusing and interacting the feature sequences of the three branches by using a space-time information fusion network based on an encoder-decoder cascade structure. And finally, respectively sending the feature map to a bounding box prediction network and a result scoring network for target positioning and dynamic template updating. According to the method, the space-time information of the static template, the dynamic template and the target search area can be effectively fused, good tracking performance is achieved, and real-time operation under the GPU can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cascaded single-target visual tracking method based on spatiotemporal information fusion. Background Art

[0002] With the rapid development of science and technology, computer vision is gaining increasing attention. Visual tracking is a crucial component of computer vision. Its core function is to analyze and model a specific target at the initial moment of a video sequence, accurately predicting the target's spatial position at any subsequent moment, laying the foundation for further analysis and understanding of the target's motion trends and behavior. Consequently, it has found widespread and in-depth applications in video surveillance, intelligent transportation, autonomous driving, visual navigation, human-computer interaction, medical diagnosis, and military reconnaissance and strike.

[0003] Due to the excellent representation learning capabilities and flexible generalization capabilities of deep learning technology, visual object tracking has entered a new era of development. Object tracking methods based on deep learning have significantly surpassed other methods in terms of success rate, accuracy, and robustness, becoming a current research hotspot.

[0004] Currently, pure Transformer architectures and "CNN+Transformer" architectures have achieved competitive results in visual tracking tasks. These two architectures are similar in that they both use the Transformer architecture to enhance feature interactions between the template and the search region, replacing the traditional cross-correlation operation. Trackers based on the pure Transformer architecture do not include CNNs in the feature extraction and fusion stages. They both design a Transformer-based full-attention tracking framework. Unlike pure Transformer architectures, "CNN+Transformer" trackers incorporate convolutional neural networks to learn the local connectivity of image features. Both use a CNN as the backbone network to extract features and then utilize the Transformer architecture to effectively integrate the features of the template and search region. However, these trackers still have two shortcomings. First, the utilization of temporal information of the target requires further development. Visual tracking is a continuous task, so temporal information is crucial. However, previous Siamese network trackers only utilize spatial information for tracking, using only the initial template for comparison with the search region, which is not robust to changes in the appearance of the tracked target. Although subsequent multi-template methods use historical information about the target to handle appearance changes, this information does not interact well between the initial feature expression, the time-varying feature expression, and the feature expression of the current frame, resulting in an inability to effectively utilize information about the target and its surroundings. Therefore, how to better fuse and interact the target's initial static information and dynamic information with the search area remains an open problem in the target tracking task. Secondly, the utilization of spatial information needs further exploration. Existing trackers based on the Transformer encoder-decoder framework have the problem of insufficient interaction between the target template and the search area information. Between different encoder-decoder layers, the encoder and decoder are just a simple multi-layer stack, without in-depth interactive propagation of spatial information, which also limits further improvement in tracking performance. Summary of the Invention

[0005] The present invention provides a cascaded single-target visual tracking method based on spatiotemporal information fusion, which overcomes the shortcomings of the above-mentioned prior art, combines the spatiotemporal information of the target more effectively, and uses the encoder-decoder cascade structure for feature interaction. In order to establish the correlation of spatiotemporal features, a spatiotemporal information fusion network is introduced to fully fuse the static template, dynamic template and global nonlinear spatiotemporal information of the target search area, which can be used for more extensive fusion propagation. In order to mine rich spatial semantic information, a feature cascade interaction structure based on encoder-decoder is designed, which can perform interactive propagation more effectively. The results show that this method achieves good tracking performance and can run in real time under GPU.

[0006] To achieve the above object, the present invention provides the following technical solution: a cascaded single-target visual tracking method based on spatiotemporal information fusion, the method comprising the following steps: S1. Feature extraction: Use the ResNet network to extract features from the static template, dynamic template and search area of ​​the tracking target respectively, obtain the feature maps of the three branches, and convert the two-dimensional feature maps into a one-dimensional feature sequence; S2. Spatiotemporal Information Fusion: A spatiotemporal information fusion network based on an encoder-decoder cascade structure is used to fuse and interact the feature sequences of the three branches, fully integrating the global nonlinear spatiotemporal information of the static template, dynamic template, and target search area, facilitating information propagation among the three. The search branch uses the decoder's multi-head attention mechanism to fuse spatiotemporal information with the static template branch and the dynamic template branch respectively. The fused search area features are spliced ​​in the channel dimension and sent to the search area fusion module for secondary fusion to accurately identify the target in the search area. S3. Feature Cascade Interaction: In the spatiotemporal information fusion network, a Transformer-based encoder-decoder cascade structure is designed. The encoder-decoder pipeline is rearranged, and three branches interact with the features of the target template and search area through different layers of the encoder-decoder cascade structure. The encoder and decoder of each layer are interconnected to form a continuous multi-layer cascade feature, which more effectively interacts with features from different layers and further mines the spatial information of the target template and search area. S4. Result prediction: Use the bounding box prediction network to accurately locate the tracked target; S5. Dynamic template update: Use the result scoring network to update the dynamic template.

[0007] Preferably, in step S1, the image features of the three branches are extracted, and 16-fold downsampling features are calculated, and the channel dimension is changed from the initial 3 to 256.

[0008] Preferably, in step S2, the search area fusion module includes a 1x1 convolutional neural network.

[0009] Preferably, in step S3, the encoder-decoder cascade structure adopts a multi-head attention mechanism based on Transformer, including a three-layer encoder based on multi-head self-attention and a three-layer decoder based on multi-head cross-attention, and uses residual connections to directly add the input information of the encoder or decoder to the output end of each layer.

[0010] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a cascaded single-target visual tracking method based on spatiotemporal information fusion. This method cleverly combines the spatiotemporal information of the tracked target and uses the encoder-decoder cascade structure for feature interaction. In order to establish the correlation between spatiotemporal features, a spatiotemporal information fusion network is introduced to fully fuse the initial static template, dynamic template and the global nonlinear spatiotemporal information of the target search area, which can be used for more extensive fusion propagation. In order to mine rich spatial semantic information, a feature cascade interaction structure based on encoder-decoder is designed, which can perform interactive propagation more effectively. Experiments show that this method achieves good tracking performance and can run in real time under GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a framework diagram of the target tracking method of the present invention; Figure 2 This is a diagram of the encoder-decoder cascade structure of the present invention; Figure 3 This is a structural diagram of the encoder of the present invention; Figure 4 This is a structural diagram of the decoder of the present invention. DETAILED DESCRIPTION

[0012] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0013] The present invention provides a cascaded single-target visual tracking method based on spatiotemporal information fusion, which overcomes the problems of the existing technology, cleverly combines the spatiotemporal information of the tracked target, and uses the encoder-decoder cascade structure for feature interaction. In order to establish the correlation of spatiotemporal features, a spatiotemporal information fusion network is introduced to fully fuse the initial static template, dynamic template and the global nonlinear spatiotemporal information of the target search area, which can be used for more extensive fusion propagation. In order to mine rich spatial semantic information, a feature cascade interaction structure based on encoder-decoder is designed, which can perform interactive propagation more effectively. The results show that this method achieves good tracking performance and can run in real time under GPU.

[0014] A cascaded single-target visual tracking method based on spatiotemporal information fusion, the method comprising the following steps: S1. Feature extraction: Use the ResNet network to extract features from the static template, dynamic template and search area of ​​the tracking target respectively, obtain the feature maps of the three branches, and convert the two-dimensional feature maps into a one-dimensional feature sequence; S2. Spatiotemporal Information Fusion: A spatiotemporal information fusion network based on an encoder-decoder cascade structure is used to fuse and interact the feature sequences of the three branches, fully integrating the global nonlinear spatiotemporal information of the static template, dynamic template, and target search area, facilitating information propagation among the three. The search branch uses the decoder's multi-head attention mechanism to fuse spatiotemporal information with the static template branch and the dynamic template branch respectively. The fused search area features are spliced ​​in the channel dimension and sent to the search area fusion module for secondary fusion to accurately identify the target in the search area. S3. Feature Cascade Interaction: In the spatiotemporal information fusion network, a Transformer-based encoder-decoder cascade structure is designed. The encoder-decoder pipeline is rearranged, and three branches interact with the features of the target template and search area through different layers of the encoder-decoder cascade structure. The encoder and decoder of each layer are interconnected to form a continuous multi-layer cascade feature, which more effectively interacts with features from different layers and further mines the spatial information of the target template and search area. S4. Result prediction: Use the bounding box prediction network to accurately locate the tracked target; S5. Dynamic template update: Use the result scoring network to update the dynamic template.

[0015] Specifically, such as Figure 1 As shown, an embodiment of a cascaded single-target visual tracking method based on spatiotemporal information fusion includes the following steps: S101: The present invention designs three branches. In addition to the conventional first frame initial static template branch and the current frame search area branch, a dynamic template sampled from the intermediate frame is introduced as an additional input branch. The ResNet-50 network is used to extract features of the tracking target static template, dynamic template and search area respectively. The target static template, dynamic template and search area images are cropped and scaled to 320×320 pixel image patches, which constitute the input of the feature extraction network ResNet-50. The shapes are as follows: the current frame is the search area image patch , the initial frame is the target static template image patch , a collection of multiple intermediate frames is a dynamic template .in H s 、W s 、H st 、W st 、H dt 、W dt are the pixel height and width of the search area image, target static image, and dynamic image respectively.

[0016] ResNet-50 downsamples the input by a factor of 16 to obtain a feature map, and the channel dimension changes from 3 to 256. The ResNet network is a prior art in this field and will not be described in detail in this invention.

[0017] In order to input the feature map into the attention module, the two-dimensional feature map is converted into a one-dimensional feature sequence.

[0018] S102: Use the spatiotemporal information fusion network to fuse and interact the feature maps of the three branches, such as Figure 1 As shown in Figure 2, the three branches interact with the spatial features of the target template and the search area through different layers in the encoder-decoder cascade structure. First, the search branch fuses the spatiotemporal information with the static template branch and the dynamic template branch through the multi-head attention mechanism of the decoder. Then, the fused search area features are spliced ​​in the channel dimension and sent to the search area fusion module for secondary fusion to obtain the feature map f. out ∈R d×Ns ,in N s = H s × W s Finally, the feature maps are fed into the bounding box prediction network and the result scoring network for target localization and dynamic template updating. Therefore, the present invention designs two levels of information dissemination: one is to interactively disseminate spatial information using a feature cascade interaction structure; the other is to fusion-disseminate spatiotemporal information using a spatiotemporal information fusion network.

[0019] S103: In the spatiotemporal information fusion network, first, each encoder-decoder cascade structure in the three branches simultaneously receives the feature sequence from the branch and the other branch and fuses them. Subsequently, the features fused with the static template in the search area and the features fused with the dynamic template in the search area are aggregated and spliced ​​in the channel dimension to simultaneously utilize the initial static information and dynamic information of the target. Finally, the spliced ​​result is sent to the search area feature fusion module, the feature is convolved, and the number of channels is projected back to the original dimension, and the global relationship between all elements is modeled in space and time to generate a discriminative spatiotemporal feature map. Specifically, the feature S∈R fused with the initial static template d ×H×W and the feature C∈R fused with the dynamic template d×H×W Proceed as follows: (1) , (2) Where X∈R 2d×H×Wis the aggregate feature, Conv(·) is the convolution operation for learning the interaction between features, and generates the fusion feature f∈R d×H×W , as the input of the bounding box prediction network and the scoring network, the original channel dimension d =256.

[0020] S104: In the spatiotemporal information fusion network, the present invention designs a Transformer-based encoder-decoder feature cascade structure. Figure 2 As shown in Figure 1, unlike other structures, the encoder-decoder pipeline is rearranged. This structure contains a three-layer Transformer encoder based on multi-head self-attention and a three-layer Transformer decoder based on multi-head cross attention. The specific structures of the encoder and decoder are as follows: Figure 3 and Figure 4 Specifically, the first encoder layer of the cascade structure receives the feature sequence f from the feature extraction network in ∈R d×N , the features are enhanced and sent to the next layer of encoder. At the same time, they are also sent to the corresponding decoder as the query Q sequence of cross attention, while the key K and value V sequence come from other branches. The connection method of the second and third layers is similar, but the key K and value V sequence of each layer of cross attention comes from the sum of the decoder output of the previous layer and the query Q sequence. The above operations constitute a stacked feature interaction structure, which further improves the degree of interaction of spatial information. At the same time, residual connections are introduced in the encoder and decoder to directly add the input information to the output end of this layer, thereby improving the feature expression capability. This structure does not simply stack the encoder and decoder, but rearranges the pipeline. The encoder and decoder of each layer are interconnected to form a continuous multi-layer cascade feature. This structure makes information interaction between the encoder and decoder layers possible, further mining the spatial information of the target template and the search area. The multi-head self-attention module and the multi-head cross-attention module of the Transformer are prior arts in this field, and the present invention will not go into details about the internal specific structure.

[0021] S105: The bounding box prediction network accurately locates the tracked target. A corner-based prediction head is used to estimate the bounding box of the target object in the current search frame. Specifically, the prediction head uses a two-branch convolutional neural network to generate two probability maps for the top-left and bottom-right corners of the target object. The final target bounding box is obtained by calculating the expected probability distribution of the two corner points. Corner-based prediction heads are state-of-the-art in this field, and the detailed internal structure is not described in detail in this disclosure.

[0022] The result scoring network updates the dynamic template. For each bounding box, the result scoring network estimates its Intersection over Union (IoU) using the ground truth. The predicted IoU score is then compared with a set threshold to determine whether to use the current search frame as the new dynamic template. Specifically, the IoU scoring network consists of three Conv-Bn-Relu layers, one PrPool layer, and two fully connected layers. IoU calculation is well-known in the art and will not be further described in this document.

[0023] In terms of loss functions, the result scoring network uses mean squared error loss, while the bounding box prediction network uses a linear combination of L1 loss and GIoU loss, with a linear combination ratio of L1 and GIoU of 5:2.

[0024] S106: When the tracker performs online tracking, it first gives the initial frame of the video sequence with true value annotations, extracts its features and saves them as the initial static template and the original dynamic template in the template pool. The multi-template scheme records the temporal information and appearance changes of the target. Five frames are selected as templates, including one static template and four dynamic templates. Each subsequent frame of the video sequence is a search frame. The backbone network is used to extract their features and interact with the static template and dynamic template in the spatiotemporal information fusion network to obtain the bounding box result. When the IoU score of the predicted bounding box is greater than the set threshold, this frame is used as a new dynamic template to update the template pool. When the IoU score of the predicted bounding box is less than the set threshold, this frame is discarded and not put into the template pool. Here, the IoU threshold is set to 0.7, and the template pool is set to 100 frames by default. When the template pool capacity reaches the upper limit, the earliest dynamic template will be ejected.

[0025] Experimental details: The hardware environment of the experiment of the present invention is a 4x12GB Titan V GPU server, which can run at a speed of up to 40FPS. The software environment of the present invention is Python 3.6, PyTorch 1.8 and Cuda 10.1. This experiment uses the training sets of LaSOT, TrackingNet, GOT-10k, and COCO for offline training, using a distributed data parallel training strategy. Common data enhancement methods such as translation and brightness jitter are used to expand the data volume of the training set. The entire training process requires 500 rounds. In terms of optimizer and learning rate, the AdamW optimizer is used to train the model, and the weight decay is 1*e -4 . Set the initial learning rate of the backbone network parameters to 1*e -5 , the remaining components are set to 1*e -4 During the training process, one frame is randomly selected as the search frame, one frame as the initial static template, and one frame as the dynamic template, ensuring that the index of the search frame is greater than the indexes of the two template frames.

[0026] Comparative experiment: This method is compared with the current cutting-edge single target tracking algorithm. The experimental results are shown in the following table:

[0027]

[0028] In summary, the present invention provides a cascaded single-target visual tracking method based on spatiotemporal information fusion. This method cleverly combines the spatiotemporal information of the tracked target to overcome the robustness and accuracy issues encountered by existing tracking networks. The method includes two levels of information propagation. The first is based on fused spatiotemporal information propagation, which utilizes a spatiotemporal information fusion network to enable the tracker to fully establish spatiotemporal feature associations. The second is based on interactive spatial information propagation, which utilizes an encoder-decoder cascade structure to enable the tracker to effectively mine rich spatial semantic information. The method proposed in the present invention enables more effective interaction between the target template and the search area, thereby obtaining more robust tracking results.

[0029] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0030] The present invention has been described in detail above with reference to the accompanying drawings. These embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Anyone skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. The details in the examples should not be construed as limiting the present invention. The scope of protection of the present invention shall be determined by the scope defined in the appended claims.

Claims

1. A cascaded single target visual tracking method based on spatiotemporal information fusion, characterized in that: The method comprises the following steps: S1. Feature extraction: Use the ResNet network to extract features from the static template, dynamic template and search area of ​​the tracking target, obtain feature maps of the three branches, and convert the two-dimensional feature map into a one-dimensional feature sequence; S2, spatiotemporal information fusion: The spatiotemporal information fusion network based on the encoder-decoder cascade structure is used to fuse and interact the feature sequences of the three branches, fully integrating the global nonlinear spatiotemporal information of the static template, dynamic template and target search area, so as to facilitate information propagation among the three. The search branch uses the multi-head attention mechanism of the decoder to fuse the spatiotemporal information with the static template branch and the dynamic template branch respectively, splices the fused search area features in the channel dimension, and sends them to the search area fusion module for secondary fusion, so as to accurately identify the target in the search area; S3, feature cascade interaction: In the spatiotemporal information fusion network, a Transformer-based encoder-decoder cascade structure is designed, and the encoder-decoder pipeline is rearranged. The three branches interact with the features of the target template and the search area through different layers in the encoder-decoder cascade structure. The encoder and decoder of each layer are interconnected to form a continuous multi-layer cascade feature, so as to more effectively interact with the features from different layers and further mine the spatial information of the target template and the search area; S4, Result Prediction: Use the bounding box prediction network to accurately locate the tracked target; S5. Dynamic template update: Use the result scoring network to update the dynamic template.

2. The cascaded single target visual tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: In the step S1, the image features of the three branches are extracted, and 16-fold down-sampling features are calculated, and the channel dimension is changed from the initial 3 to 256.

3. The cascaded single target visual tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: In step S2, the search area fusion module includes a 1x1 convolutional neural network.

4. The cascaded single target visual tracking method based on spatiotemporal information fusion according to claim 1, characterized in that: In step S3, the encoder-decoder cascade structure adopts a multi-head attention mechanism based on Transformer, including a three-layer encoder based on multi-head self-attention and a three-layer decoder based on multi-head cross-attention, and uses residual connection to directly add the input information of the encoder or decoder to the output end of each layer.

Images