A spatio-temporal attention-based visual object tracking system and method

Abstract

The application relates to a visual target tracking system and method based on space-time attention, the system of the application comprises an image acquisition module, a feature extraction module, a space-time feature enhancement module and a classification regression positioning module connected in sequence, the space-time feature enhancement module comprises a space-time excitation module, a motion excitation module, a channel excitation module and a sparse Transformer, the sparse Transformer module comprises a sparse Transformer encoder and a sparse Transformer decoder connected with each other, the feature extraction module is connected with the sparse Transformer encoder through the space-time excitation module, the motion excitation module and the channel excitation module respectively, and the sparse Transformer encoder and the sparse Transformer decoder are connected with the classification regression positioning module respectively. Through exploration of the space-time context relationship of continuous frames, the application can learn discriminative space-time, motion and spatial information, and effectively improves the performance of a tracker.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to the field of visual tracking technology for digital images, and particularly to a visual target tracking system and method based on spatiotemporal attention. Background Technology

[0002] Visual object tracking is an important research area in computer vision, with wide applications in video understanding, human-computer interaction, visual surveillance, and autonomous driving. In recent years, object tracking based on Siamese networks has become increasingly popular. The SiamFC algorithm combines CNN features with the Siamese framework to achieve a fully convolutional Siamese network tracking algorithm, and subsequently, a series of improved algorithms based on Siamese network trackers have emerged. Compared with SiamFC's multi-scale estimation, SiamRPN and SiamFC++, based on anchored or anchorless bounding box estimation mechanisms, effectively improve the accuracy of object localization. Meanwhile, SiamRPN++ and Ocean use the powerful ResNet50 as a backbone to enhance feature representation capabilities. Furthermore, algorithms such as ATOM and DiMP combine online updates with Siamese networks, resulting in better tracking performance.

[0003] While CNN-based visual feature extraction has achieved success in various image tasks, extracting robust spatiotemporal features from video sequences remains challenging due to the variability and complexity of video content. Spatiotemporal feature extraction networks are widely used in action recognition and action understanding. In tracking tasks, given the strong spatiotemporal dynamic correlations in videos, these networks can be considered. However, existing tracking algorithms mostly rely on image-level features in videos, performing two-dimensional convolutions on individual frames. This approach does not effectively utilize the spatiotemporal dynamic correlations of videos. For example... Figure 1 As shown in the image, the foreground and background of the target are poorly distinguishable; the target occupies 20%–30% of the bounding box, and interference and background clutter appear in the response maps of both DiMP and TrDiMP algorithms. This indicates that in such complex backgrounds, relying solely on the target's apparent features in individual frames is prone to deviation during tracking. Therefore, it is essential to utilize networks to learn the spatiotemporal dynamic changes of targets in videos. To address this issue, this paper characterizes the temporal and spatial information of targets in videos by dividing video feature information into three parts: spatial information flow, temporal information flow, and motion information flow.

[0004] Transformer networks are widely used in various video tasks and have good learning and generalization ability in different scenarios, and algorithms such as TrDiMP and TrTr successfully apply the Transformer structure in visual target tracking and achieve good results. However, due to the limitation of computational complexity, the Transformer is not good at processing long context. SUMMARY

[0005] In order to solve the above problems, the application provides a visual target tracking method based on spatio-temporal attention, which can learn discriminative spatio-temporal, motion and spatial information by exploring the spatio-temporal context relationship of continuous frames, and effectively improves the performance of the tracker.

[0006] The technical solution of the application is: the application is a visual target tracking system based on spatio-temporal attention, which is characterized in that: the visual target tracking system based on spatio-temporal attention includes image acquisition module, feature extraction module, spatio-temporal feature enhancement module and classification regression positioning module connected in turn, the image acquisition module is used for acquiring multiple images in a video sequence as training images and test images respectively; the feature extraction module is used for inputting the training images and test images into a visual tracking network for feature extraction to obtain feature representation of the video sequence; the spatio-temporal feature enhancement module includes a spatio-temporal excitation module, a motion excitation module, a channel excitation module and a sparse Transformer, the sparse Transformer module includes a sparse Transformer encoder and a sparse Transformer decoder connected with each other, the feature extraction module is connected with the sparse Transformer encoder through the spatio-temporal excitation module, the motion excitation module and the channel excitation module, the sparse Transformer encoder and the sparse Transformer decoder are connected with the classification regression positioning module, the spatio-temporal information in the video sequence is modeled through the spatio-temporal excitation module, the motion excitation module and the channel excitation module, then the three kinds of mixed complementary features are fused, and the integrated feature information is input into the spatio-temporal feature fusion network enhanced by the sparse Transformer module to model the spatio-temporal information; the classification regression positioning module includes a model predictor f and a conv calculation, the model predictor f iteratively calculates and optimizes the sparse Transformer decoder aggregation features, and then performs conv calculation with the decoder output to output weights, which are used for target classification operation of the feature map; finally, the bounding box is obtained by the offset predicted in the classification map and the coordinates of the best prediction point.

[0007] Further, the spatio-temporal excitation module uses single-channel 3D convolution to represent spatio-temporal information; compared with traditional 3D convolution operation, the calculation efficiency of STE is higher, and each channel of the input feature can perceive spatio-temporal information from a fine feature excitation.

[0008] The motion excitation module calculates the time difference between adjacent frames, and then uses the time difference to excite the motion-sensitive channel; the motion between every two adjacent frames is mainly described, which is obtained by subtracting the previous frame from the latter frame and splicing in the dimension, and the motion features of adjacent frames are calculated mainly according to the frame difference method.

[0009] The channel excitation module adaptively recalibrates the channel feature response by explicitly modeling the interdependence between channels in time. The video sequence in the channel excitation module contains time sequence information, and a 1D convolution in the time domain is inserted between the Squeeze and Unsqueeze of the feature channel to enhance the degree of interdependence of the channel in the time domain. The excitation space information of the input features is obtained by point multiplication of the attention matrix.

[0010] The application also provides a method for realizing a visual target tracking system based on spatiotemporal attention, and the speciality of the method is that the method comprises the following steps:

[0011] 1) obtaining multiple frames of images in a video sequence as training images and test images respectively through image acquisition;

[0012] 2) inputting the training images and test images into a visual tracking network through a feature extraction module for feature extraction, so as to obtain feature representation of the video sequence;

[0013] 3) modeling the spatiotemporal information in the video sequence through three modules of a spatiotemporal excitation module, a motion excitation module and a channel excitation module, then fusing three kinds of mixed complementary features, and inputting the integrated feature information into a sparsely Transformer-enhanced spatiotemporal feature fusion network to model the spatiotemporal information;

[0014] 4) performing target classification operation on the integrated feature map through a classification regression positioning module, and finally obtaining a bounding box through the predicted offset in the classification map and the coordinates of the best prediction point.

[0015] Further, the spatiotemporal excitation module in step 3) mainly uses single-channel 3D convolution to represent spatiotemporal information, and the interaction between time and space is split, the interaction is calculated only in space, and then the Attention is calculated again in time, which enables a spatiotemporal MaskM(N×T×1×H×W) to be obtained with very small calculation amount, and the corresponding features excited by the spatiotemporal information are obtained by point multiplication of the attention map and the input features.

[0016] Further, the motion excitation module in step 3) describes the movement of the action between each adjacent two frames, wherein the input is a 5D tensor (N, T, C, H, W); N corresponds to the batch, T is the number of video frames, C is the number of channels, H and W correspond to the height and width of the feature map respectively; first, the input is subjected to a 1x1 convolution for channel dimension reduction operation, and the five-dimensional tensor is separated into T four-dimensional tensors in the T dimension, wherein the last T-1 tensors are subjected to a shared 2D convolution kernel, then the last frame is subtracted from the previous frame, and splicing is performed in the T dimension, and the motion features of each frame are calculated according to formula (1), wherein the modeling of the motion features is represented as:

[0017] F m = K * F r [:, t+1, :, :, :] - F r [:, t, :, :, :] (1)

[0018] wherein K is a 3x3 convolution, first the input is subjected to a 1x1 convolution for Squeeze operation, the motion features of each frame are calculated according to the above formula, and 0 is filled into the last position, represented as: F M = [F m (1), …, F m (t-1), 0], wherein F M has a size of T x c / r x H x W, then F M is subjected to spatial average pooling, dimension reduction operation of 1x1 convolution, and Sigmod activation to obtain a mask M.

[0019] Further, the channel excitation module in step 3) contains time sequence information in the video sequence, a 1D is inserted between the Squeeze and Unsqueeze of the channel, and a convolution in the time domain is used to enhance the degree of mutual dependence of the channels in the time domain; the channel excitation module calculates a channel-based attention matrix, and the excitation spatial information of the input features is multiplied by the attention matrix; in addition, the channel excitation module inserts a 1x1 convolution layer between two FC layers to describe the time information of the channel features.

[0020] Further, the network structure of the sparse Transformer in step 3) is composed of a sparse attention module, a multilayer perceptron (MLP) and a position convolution encoding layer, and the self-attention function adopts a proportional point generation method for operation of a query Q, a key K and a value V, and the process can be represented as:

[0021]

[0022]

[0023] where dx is the key dimension of attention matrix, SAttention is sparse self-attention; it is added to the Q key value before the backbone network extracts the feature input Transformer; then by extending the attention mechanism to multiple heads, the representation ability of the model is enhanced, and its formula is as follows:

[0024] MultiHead(Q i ,K i ,V i )=Concat(H1,…,H n )W O (4)

[0025]

[0026] In the above formula, W i Q ,W i K ,W i V are parameters of linear projection, n is the number of attention heads, and after Add&Norm, it is LayerNorm(x+Linerlayer(x)), where Norm is the smooth input feature;

[0027] Position encoding is used in self-attention; then the calculated x is input into the MLP layer and the position convolution encoding layer, and its formula is as follows:

[0028] PE(Q,K,V)=Norm(x+MLP(x)) (6)

[0029]

[0030] Where, MLP is a fully connected feedforward network that converts the original feature into an arbitrary dimension, and formula (7) is the position convolution encoding layer for encoding the image to preserve the local feature detail information of the target.

[0031] The application provides a visual target tracking system based on space-time attention, which designs a spatial-temporal excitation (STE) module, a motion excitation (ME) module and a channel excitation (CE) module to model time and space information in a video sequence, then fuses three complementary features, and effectively aggregates time and space information.

[0032] The application also relates to a sparse Transformer structure, which processes data coded by three streams of space-time, motion and space, and adopts a sparse coding mode to reduce calculation complexity and provide fine features by a convolution position coding layer: CNN effectively retains local features by convolution but cannot capture global dependency, the application adopts multi-frame input to extract time sequence information by a space-time network structure, then captures long-distance dependency by self-attention of a sparse Transformer to increase stability of an output result, better processes a shielding scene, and predicts motion of a target. Linear calculation of the Transformer ignores local feature details, which reduces distinguishability between a target background and a foreground. Therefore, compared with a classic Transformer structure, the application proposes a network structure capable of combining local features of CNN and global features of the sparse Transformer to model mixed space-time information. The application extracts space-time features as input of a Transformer coding structure, proposes an encoding layer and a decoding layer of the sparse Transformer, encodes an image by the encoding layer of the Transformer. In addition, linear operation in the encoding process damages position information in each patch, making it more difficult to extract local information in the patch, the application designs a multi-layer perceptron (MLP) and a position convolution coding layer in the Transformer to flexibly process a position coding layer with variable resolution, and then enhances space-time feature representation capability by a Transformer encoder and a decoder. The sparse Transformer structure captures long-distance dependency by a Transformer attention mechanism; finally, the position convolution coding layer is used in the sparse Transformer encoder and the decoder to provide accurate position coding for feature fusion, and a space-time feature fusion network based on the Transformer enhancement is constructed to learn efficient space-time information.

[0033] The application further provides a visual target tracking algorithm based on space-time attention. The space-time information in a video sequence is modeled through three modules of a space-time excitation module, a motion excitation module and a channel excitation module, then three kinds of mixed complementary features are fused to effectively aggregate time, space and motion information. An efficient space-time information is learned through constructing a space-time feature fusion network based on Transformer enhancement. It is proved that the space-time network enhanced by the constructed context information can effectively model the space-time information, so that the tracker has good tracking performance.

[0034] Therefore, compared with the prior art, the application has the following advantages:

[0035] 1) The application can achieve real-time tracking effect, that is, has certain economic benefits.

[0036] 2) In the complex and changeable tracking environment of illumination change, scale change, occlusion, background blur and deformation, the application still has obvious advantages in maintaining high precision and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The figure is a response map result analysis diagram in the background art;

[0038] Figure 2 The figure is a system block diagram of the application;

[0039] Figure 3 In the method of the application, the network structure diagram of the space-time excitation module, the motion excitation module and the channel excitation module;

[0040] Figure 4 The figure is a sparse Transformer network structure diagram in the method of the application;

[0041] Figure 5 The figure is an experimental result diagram of the method of the application;

[0042] Figure 6 The figure is a heat map visualization diagram of the method of the application;

[0043] Figure 7 The figure is a radar chart of the method of the application under different attributes;

[0044] Figure 8 The figure is a comparison result diagram of the method of the application in VOT experiment. DETAILED DESCRIPTION

[0045] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0046] Reference Figure 2The structure of the specific embodiment of the visual target tracking system based on space-time attention provided by the application mainly comprises four parts of an image acquisition module, a feature extraction module, a space-time feature enhancement module and a classification regression positioning module. The image acquisition module is used for acquiring multiple frames of images in a video sequence as training images and test images respectively. The feature extraction module is used for inputting the training images and the test images into a visual tracking network for feature extraction to obtain feature representations of the video sequence. The space-time feature enhancement module comprises a space-time excitation module, a motion excitation module and a channel excitation module. The space-time information in the video sequence is modeled by the three modules simultaneously, then the three kinds of mixed complementary features are fused to effectively aggregate the time and space information. The integrated feature information is input into a space-time feature fusion network enhanced by a sparse Transformer to model the space-time information. The classification regression positioning module is used for target classification operation on the integrated feature map. Finally, the bounding box is obtained by the offset predicted in the classification map and the coordinates of the best prediction point.

[0047] Among them, the space-time excitation module adopts single-channel 3D convolution to represent space-time information; compared with the traditional 3D convolution operation, the calculation efficiency of STE is higher, and each channel of the input feature can perceive the space-time information from a fine feature excitation.

[0048] The motion excitation module mainly describes the movement of the action between each two adjacent frames. The motion features of adjacent frames are calculated mainly according to the frame difference method by subtracting the latter frame from the former frame and splicing in the dimension. The time difference between adjacent frames is calculated, and then the time difference is used to excite the motion-sensitive channel.

[0049] The channel excitation module adaptively recalibrates the channel feature response by explicitly modeling the interdependence between channels in the time domain, greatly utilizing the dependence between channels. The video sequence in the channel excitation module contains time sequence information. A 1D convolution in the time domain is inserted between the Squeeze and Unsqueeze of the feature channel to enhance the degree of interdependence of the channels in the time domain. The attention matrix is multiplied by the input feature to excite the spatial information of the feature.

[0050] When the target is in a complex environment, in order to capture the space-time, motion and spatial information of the target, the space-time excitation module, the motion excitation module and the channel excitation module are used to construct mixed features with three kinds of complementary information. The fused feature mapping encodes the target time, motion and appearance features. These features are input into the Transformer encoder and decoder to enhance the fused features.

[0051] The classification regression positioning module includes a model predictor f and a conv calculation, the model predictor f is initialized by a target region feature first, the model predictor f is optimized in an iterative manner by combining sparse Transformer encoder aggregation features, then f and a sparse Transformer decoder are used for conv calculation to output weights, and the weights are used for target classification positioning operation of a feature map.

[0052] The application uses a multi-frame video sequence as input, extracts deep features by using a ResNet-50 backbone network, inputs the feature map into a Transformer encoder and a decoder, inputs the encoding result into an optimizer to form a model predictor f, the model predictor f outputs the weights of a convolution layer, the weights are used in target classification operation of a feature map extracted in a test frame, and finally a bounding box is obtained by the offset predicted in a classification map and the coordinates of an optimal prediction point.

[0053] The visual target tracking method based on spatiotemporal attention provided by the application includes the following steps:

[0054] 1) Obtain multiple frames of images in a video sequence as training images and test images respectively through image acquisition;

[0055] 2) Input the training images and test images into a visual tracking network through a feature extraction module for feature extraction to obtain feature representation of the video sequence;

[0056] 3) Model the spatiotemporal information in the video sequence through a spatiotemporal excitation module, a motion excitation module and a channel excitation module, then fuse the three kinds of hybrid complementary features, input the integrated feature information into a sparsely enhanced Transformer enhanced spatiotemporal feature fusion network to model the spatiotemporal information;

[0057] 4) Perform target classification operation on the integrated feature map through a classification regression positioning module, and finally obtain a bounding box by the offset predicted in a classification map and the coordinates of an optimal prediction point.

[0058] Referring to Figure 3 The spatiotemporal excitation module mainly uses single-channel 3D convolution to represent spatiotemporal information, the structure splits the interaction of time and space, interacts only in space, calculates Attention alone, and then calculates Attention again in time, which enables a spatiotemporal Mask M (N x T x 1 x H x W) to be obtained with very small calculation amount. The corresponding features excited by spatiotemporal information are obtained by point multiplication of the attention map and the input features. Compared with traditional 3D convolution operation, the calculation efficiency of STE is higher, because the features F input into the 3D convolution are cross-channel averaged. Each channel of the input X can perceive the importance of spatiotemporal information from a fine feature excitation M.

[0059] The motion excitation module ME mainly describes the movement of the action between each adjacent two frames, wherein the input is a 5D tensor (N, T, C, H, W). N corresponds to a batch, T is the number of video frames, C is the number of channels, H and W correspond to the height and width of the feature map respectively. First, the input is subjected to a 1*1 convolution for channel dimension reduction operation, and the five-dimensional tensor is separated into T four-dimensional tensors in the T dimension, wherein the last T-1 tensors are subjected to a shared 2D convolution kernel, then the last frame is subtracted from the previous frame, and splicing is performed in the T dimension. The motion features of each frame are calculated according to formula (1), wherein the modeling of the motion features is represented as

[0060] F m =K*F r [:,t+1,:,:,:]-F r [:,t+1,:,:,:] (1)

[0061] wherein K is a 3*3 convolution, the input is first subjected to a 1*1 convolution for Squeeze operation, the motion features of each frame are calculated according to the above formula, and 0 is filled into the last position, represented as F M =[F m (1),…,F m (t-1),0] wherein F M is T*c / r*H*W, then F M is subjected to spatial average pooling, 1*1 convolution dimension reduction operation, and Sigmod activation to obtain the mask M.

[0062] The channel excitation module network structure The video sequence contains time sequence information, so a 1D convolution in the time domain is inserted between the channel Squeeze and Unsqueeze to enhance the mutual dependence of the channels in the time domain. A channel-based attention matrix is calculated. The input feature space information is excited by point multiplication of the attention matrix. In addition, the CE module inserts a 1*1 convolution layer between the two FC layers to describe the time information of the channel features.

[0063] Referring to Figure 4 , the sparse Transformer network structure of the present application comprises the following contents:

[0064] The sparse Transformer network structure is shown in Figure 4 This part is composed of a sparse attention module, a multilayer perceptron (MLP) and a position convolution encoding layer. The self-attention function adopts the proportional point generation method for operation of the query Q, the key K and the value V, and the process can be represented as:

[0065]

[0066]

[0067] where dx is the key dimension of attention matrix, SAttention is sparse self-attention. It is added to the Q key value before the backbone network extracts the feature input into the Transformer. Then by extending the attention mechanism to multiple heads, the representation ability of the model is enhanced, and its formula is as follows:

[0068] MultiHead(Q i ,K i ,V i )=Concat(H1,…,H n )W O (4)

[0069] H i =SAttention(QW i Q ,KW i K ,VW i V ) (5)

[0070] In the above formula, W i Q ,W i K ,W i V ∈ linear projection parameters, n is the number of attention heads, and after Add&Norm, it is LayerNorm(x+Linerlayer(x)), where Norm is the smooth input feature;

[0071] The Patch Flatten operation will damage the position information within the Patch, and for this purpose, we use position encoding in self-attention. Then the calculated x is input into the MLP layer and the position convolution encoding layer, and its formula is as follows:

[0072] PE(Q,K,V)=Norm(x+MLP(x)) (6)

[0073]

[0074] where MLP is a fully connected feedforward network that converts the original feature into an arbitrary dimension, and formula 7 is the position convolution encoding layer that encodes the image to preserve the local feature detail information of the target.

[0075] To verify the effectiveness of the method, the algorithm is implemented on an Inteli5-8400 CPU@2.80GHz processor using Python programming under the Ubuntu operating system, and is accelerated by a GPU (NVIDIA GTX 1080Ti).

[0076] The training data is from the split of TrackingNet, COCO, LaSOT and GOT-10k datasets. The experimental details are as follows: ResNet-50 pre-trained on the ImageNet dataset is selected to initialize the model, and the neural network is initialized. All experiments are trained for 50 epochs, and the ADAM optimizer is used to accelerate network convergence, and the initial learning rate of the backbone network is 10 -5 , and the learning rate is reduced by 10 -1 .

[0077] Referring to Figure 5 Due to the complexity of underwater targets, the Turtle video sequence is selected to observe the experimental effect, the target foreground and background in the video sequence are less distinguishable, and the multiple appearances of the target in the movement process are interfered by a large number of similar interference objects, and compared with other algorithms, the target can be effectively tracked by the present application. Figure 5 The second graph in the figure is the Skiing video sequence, due to the appearance of occlusion and target disappearance, the present application can better track the target compared with the background technology, and when the target disappears and reappears, the method of the present application can still track the target.

[0078] Referring to Figure 6 To observe the effect, the present application visualizes the activation map of the training model backbone network, compares the heat map visualization of the background technology, the present application containing the convolution position encoding layer and the algorithm not containing the encoding layer, as shown in Figure 6 , wherein Tr is the heat map visualization result of the standard Transformer, and PE is the heat map visualization result after introducing the convolution position encoding layer. As can be seen from the figure, in the case of severe deformation of the target appearance, the background technology has higher activation value around the interference object, indicating that these channels pay more attention to the interference object rather than the target itself, while the method of the present application has higher activation value in the target area, and the time sequence information pays more attention to the part that moves, and the spatial information pays more attention to the fine spatial features of the complete part. This verifies that the backbone network effectively extracts the complementary information of the target.

[0079] To verify the actual effect of the algorithm, LaSOT, VOT2019 and VOT2018-LT datasets are selected for evaluation. DiMP, PrDiMP, TrDiMP, TrTr, STMTrack and other background technologies are selected as reference methods.

[0080] Referring toFigure 7 The AUC (Area Under Curve) of 14 different attributes is tested on the LaSOT dataset, and a radar chart is drawn. These attributes are: camera motion (CM), view point change (VC), fast motion (FM), and the like. In motion videos, low resolution images and target deformation often occur. In such videos, the spatial features of single images are very limited, and the method fully and effectively aggregates spatial information in the time dimension. Compared with the background technology, the AUC is improved by 3.2% in motion blur videos and 2.4% in fast motion videos. Compared with other background technologies, the method has obvious advantages in motion videos. It shows that the algorithm in this chapter has good robustness when dealing with complex scenes.

[0081] To further evaluate the effectiveness of the algorithm in visual tracking, the algorithm is evaluated on the VOT2019 dataset and compared with other background technologies. VOT contains three indexes, EAO (Expected Average Overlap), Accuracy and Robustness, to analyze the tracking effect, and the results are shown in Table 1. The best result is bold, the second best is underlined, and the third best is underlined. See Figure 8 , and the right side is the result of the top ten algorithms. Compared with the background technology, the method improves the EAO in the VOT2019 dataset by 1.3%, which further illustrates the effectiveness of the method.

[0082] Table 1 Comparison of algorithm experimental results of VOT2019

[0083]

[0084] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

[0085] The technical content not specifically described in the summary and the above embodiments is the same as the prior art.

[0086] The present application is not limited to the above embodiments, and the content described in the summary can be implemented and has the good effects described.

Claims

1. A visual object tracking system based on spatio-temporal attention, characterized in that: The spatio-temporal attention-based visual target tracking system comprises, in sequence, an image acquisition module, a feature extraction module, a spatio-temporal feature enhancement module and a classification regression positioning module, the image acquisition module is used to acquire multiple frames of images in a video sequence as training images and test images respectively; the feature extraction module is used to input the training images and the test images into a visual tracking network for feature extraction to obtain feature representations of the video sequence; the spatio-temporal feature enhancement module comprises a spatio-temporal excitation module, a motion excitation module, a channel excitation module and a sparse Transformer module, the sparse Transformer module comprises a sparse Transformer encoder and a sparse Transformer decoder connected with each other, the feature extraction module is connected with the sparse Transformer encoder through the spatio-temporal excitation module, the motion excitation module and the channel excitation module respectively, and the sparse Transformer encoder and the sparse Transformer decoder are connected with the classification regression positioning module respectively, the spatio-temporal information in the video sequence is modeled through the three modules of the spatio-temporal excitation module, the motion excitation module and the channel excitation module at the same time, then the three kinds of mixed complementary features are fused, the integrated feature information is input into the spatio-temporal feature fusion network enhanced by the sparse Transformer module to model the spatio-temporal information; the classification regression positioning module comprises a model predictor f and a conv calculation, the model predictor f performs iterative calculation and optimization on the sparse Transformer decoder aggregated features, then performs conv calculation with the decoder output to output weights, and the weights are used for target classification operation of the feature map; finally, the bounding box is obtained through the offset predicted in the classification map and the coordinates of the best prediction point; the motion excitation module describes the movement of actions between each two adjacent frames, wherein the input is a 5D tensor (N, T, C, H, W); N corresponds to batches, T is the number of video frames, C is the number of channels, H and W correspond to the height and width of the feature map respectively; first, the input is subjected to a 1*1 convolution for channel dimension reduction operation, the five-dimensional tensor is separated into T four-dimensional tensors in the T dimension, wherein the last T-1 tensors are subjected to a shared 2D convolution kernel, then the last frame is subtracted from the first frame, splicing is performed in the T dimension, and the motion features of each frame are calculated according to formula (1), wherein the modeling of the motion features is represented as: F m = K * F r [:, t + 1, :, :, : ] - F r [:, t + 1, :, :, : ] (1) where K is a 3x3 convolution, first input through 1x1 convolution for Squeeze operation, according to the above formula to calculate the motion characteristics of each frame, fill 0 to the last position, expressed as: F M =[F m (1),…,F m (t-1),0] where F M size is TxcrxHxW, then F M is passed through spatial average pooling, 1x1 convolution dimension reduction operation, and Sigmod activation to obtain the mask M. 2.The spatio-temporal attention based visual object tracking system of claim 1, wherein: The spatio-temporal excitation module adopts a single-channel 3D convolution to represent spatio-temporal information; the motion excitation module calculates the time difference between adjacent frames, and then uses the time difference to excite the motion-sensitive channel; the channel excitation module adaptively recalibrates the channel feature response by explicitly modeling the interdependence between channels in terms of time.

3. A method for implementing a spatio-temporal attention based visual object tracking system, characterized in that: The method comprises the following steps: 1) acquiring multiple frames of images in a video sequence as training images and test images through an image acquisition module; 2) inputting the training images and the test images into a visual tracking network for feature extraction through a feature extraction module to obtain feature representations of the video sequence; 3) Through the three modules of space-time excitation module, motion excitation module and channel excitation module, the space-time information in the video sequence is modeled at the same time, then the three kinds of mixed complementary features are fused, and the integrated feature information is input into the sparse Transformer enhanced space-time feature fusion network to model the space-time information; the motion excitation module describes the movement of the action between each adjacent two frames, wherein the input is a 5D tensor (N, T, C, H, W); N corresponds to batch, T is the number of video frames, C is the number of channels, H and W correspond to the height and width of the feature map respectively; first, the input is subjected to a 1*1 convolution for channel dimension reduction operation, and the five-dimensional tensor is separated into T four-dimensional tensors in the T dimension, wherein the last T-1 tensors are subjected to a shared 2D convolution kernel, then the last frame is subtracted from the previous frame, and splicing is performed in the T dimension, and the motion features of each frame are calculated according to formula (1), wherein the modeling of the motion features is represented as: F m = K * F r [:, t + 1, :,:,:] - F r [:, t + 1, :,:,:] (1) where K is a 3x3 convolution, first input through 1x1 convolution for Squeeze operation, according to the above formula to calculate the motion characteristics of each frame, fill 0 to the last position, expressed as: F M = [F m (1),…,F m (t-1),0], wherein F M size is TxcrxHxW, then F M is subjected to spatial average pooling, 1x1 convolution dimension reduction operation, and Sigmod activation to obtain the mask M. 4) The target classification operation of the integrated feature map is performed through the classification regression positioning module, and finally the bounding box is obtained through the offset predicted in the classification map and the coordinates of the best prediction point.

4. The method of implementing a spatio-temporal attention based visual object tracking system according to claim 3, characterized in that: In the space-time excitation module in step 3), a single-channel 3D convolution is used to represent the space-time information, the interaction between time and space is split, the interaction is only calculated in space, and then the Attention is calculated again in time, which enables a very small amount of calculation to obtain a space-time MaskM (N* T* 1* H* W), and the corresponding features excited by the space-time information are obtained by point multiplication of the attention map and the input features.

5. The method of implementing a spatio-temporal attention based visual object tracking system according to claim 4, characterized in that: In the channel excitation module in step 3), the video sequence contains time sequence information, and a 1D is inserted between the Squeeze and Unsqueeze of the channel to enhance the degree of mutual dependence of the channel in the time domain through the convolution in the time domain; The channel excitation module calculates a channel-based attention matrix, and excites the space information of the input features through point multiplication of the attention matrix and the input features; In addition, the channel excitation module inserts a 1*1 convolution layer between the two FC layers to describe the time information of the channel features.

6. The method of implementing a spatio-temporal attention based visual object tracking system according to claim 5, characterized in that: The network structure of the sparse Transformer in step 3) is composed of a sparse attention module, a multilayer perceptron (MLP) and a position convolution encoding layer, The self-attention function adopts the proportional point generation method to operate on the query Q, key K and value V, and the process is represented as: Wherein dx is the key dimension of the attention matrix, and SAttention is the sparse self-attention; before the feature extracted by the backbone network is input into the Transformer, it is added to the Q key value; then the attention mechanism is expanded into multiple heads, which enhances the representation ability of the model, and the formula is represented as follows: MultiHead(Q i ,K i ,V i )=Concat(H1,…,H n )W O (4) H i = SAttention(QW i Q , KW i K , VW i V ) (5) W in the above formula i Q W i K W i V W is the parameter of linear projection, n is the number of attention heads, and after Add&Norm, it is LayerNorm(x+Linerlayer(x)), where Norm is the smoothing input feature Position encoding is used in the self-attention; then the calculated x is input into the MLP layer and the position convolution encoding layer, and the formula is represented as follows: PE(Q,K,V)=Norm(x+MLP(x)) (6) Wherein, the MLP is a fully connected feedforward network, which converts the original features into arbitrary dimensions, and formula (7) is a position convolutional encoding layer which encodes the image to retain the local feature detail information of the target.

Images