A spatio-temporal attention mechanism-based twin network tracking system and method

Abstract

The application discloses a kind of twin network tracking system and method based on space-time attention mechanism in the technical field of target tracking of computer vision, comprising: obtaining template image data and search image data in image data;Template image data and search image data are respectively input into template branch and search branch to extract features, and template feature data and search feature data are obtained;Search feature data is input into space-time attention mechanism module, and space-time attention information feature is obtained;Template feature data and space-time attention information feature are cross-correlated to obtain response graph;Response graph is input into classification regression network to carry out pixel-by-pixel classification and regression, and each pixel point is classified, the distance of pixel point to prediction box is calculated, and classification result and boundary box are obtained.The application can more accurately track target, effectively solve the problem that model is difficult to establish association for space-time context information, target classification and boundary box regression are relatively complex, etc.

Description

Technical Field

[0001] This invention relates to a twin network tracking system and method based on a spatiotemporal attention mechanism, belonging to the field of target tracking technology in computer vision. Background Technology

[0002] Target tracking is widely used in video analytics, intelligent transportation, virtual reality, and other fields. Its core task is to automatically estimate the target's position in subsequent frames after the target's initial position in the first frame of a video. However, the tracking process is easily affected by a series of complex factors such as occlusion, blurring, deformation, and changes in lighting. Therefore, designing a tracking method that can operate stably and efficiently in complex real-world scenarios is of great research and practical value.

[0003] Currently, existing tracking algorithms can be broadly categorized into two main types. One type applies the concept of correlation filtering to the tracking field, proposing a minimum squared error filter target tracking algorithm. This transforms the solution for the tracker template from complex time-domain operations to Fourier-domain dot product calculations, significantly reducing computational complexity and dramatically improving tracker speed. While this algorithm is fast, its accuracy is not ideal. The other type, aiming for a balance between accuracy and speed, utilizes deep learning-based Siamese network tracking algorithms. Siamese networks primarily consist of a template branch and a search branch that share parameters during feature extraction. A fully convolutional Siamese network tracking algorithm, employing similarity learning, transforms the tracking problem into a similarity matching problem by calculating the similarity value between the target template and the deep features of the search region. This algorithm can be trained end-to-end offline using large amounts of data, achieving high accuracy without requiring online adjustments. By introducing a region proposal network from the detection domain into the fully convolutional Siamese network target tracking algorithm, and using methods to distinguish foreground and background and regress bounding boxes, flexible handling of target scales is achieved, while obtaining more accurate bounding boxes. Experimental results show that the tracking performance of this combined model is significantly better than that of the correlation filtering algorithm and the fully convolutional twin network algorithm.

[0004] However, most target tracking algorithms based on Siamese networks have two major shortcomings:

[0005] (1) The tracking algorithm based on Siamese network does not pay attention to the correlation between and within video sequences, resulting in the target not being able to generate corresponding correlation in time and space.

[0006] (2) The classification and regression of the target by the tracking algorithm is relatively coarse and complex, making it difficult to form a more accurate classification result and bounding box for the final result. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a twin network tracking system and method based on spatiotemporal attention mechanism, which can track targets more accurately and effectively solve problems such as the difficulty of establishing correlation between spatiotemporal context information and the complexity of target classification and bounding box regression.

[0008] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0009] In a first aspect, the present invention provides a twin network tracking method based on a spatiotemporal attention mechanism, comprising:

[0010] Retrieve template image data and search image data from image data;

[0011] The template image data and the search image data are respectively input into the template branch and the search branch for feature extraction to obtain the template feature data and the search feature data.

[0012] The search feature data is input into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features;

[0013] The response map is obtained by cross-correlating the template feature data with the spatiotemporal attention information features.

[0014] The response map is input into the classification and regression network for pixel-by-pixel classification and regression. Each pixel is classified, and the distance from the pixel to the predicted box is calculated to obtain the classification result and the bounding box.

[0015] Furthermore, acquiring template image data and search image data from the image data includes: cropping an image of a specified pixel size from the first frame of the dataset or camera capture image centered on the target as template image data; and cropping an image of a set size from the target position in the (i+1)th frame during tracking as search image data.

[0016] Furthermore, the feature extraction formula is as follows:

[0017]

[0018] Where I is the input image matrix, I((x a ,y a ),(x b ,y b )) is a submatrix of I, (x a ,y a ) and (x b ,y b ) represents the coordinates of the top left and bottom right corners of the submatrix in a rectangular coordinate system, k is the step size for extracting feature data, and f(·) is the feature extraction function.

[0019] Furthermore, the spatiotemporal attention information features are:

[0020]

[0021] Among them, o i For spatiotemporal attention information features, Spatial attention information features, For time attention information features, z i For input features;

[0022] The spatial attention information features are:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in, For spatial characteristics, C(Z) S ) is the normalization factor, and f is the Gaussian function. and Let i and j be the features at positions i and j, respectively, and g be a univariate function. These are learnable parameters. and All are spatial feature weights.

[0031] Furthermore, the response graph is as follows:

[0032]

[0033] in, For response diagrams, is the convolutional feature extraction function, * is the convolutional cross-correlation operation, TS(·) is the actual attention module, b1 is the bias value at each position of the response map, x is the search feature data, and z is the template feature data.

[0034] Furthermore, the response map is input into a classification and regression network for pixel-by-pixel classification and regression. Each pixel is classified, and the distance from the pixel to the predicted bounding box is calculated to obtain the classification result and bounding box, including:

[0035] Upgrade the response graph to a higher dimension

[0036] The classification branch in the classification regression network is for The classification map is obtained by predicting the category of each pixel. Among them, classification diagram A cls Each (i,j,:) contains a 2D vector representing the confidence of the foreground and background of the search region;

[0037] The regression branch calculation in the classification regression network The regression map is obtained by measuring the distance from each pixel to the bounding box. Among them, regression graph A reg Each point (i,j,:) contains a 4D vector m (i,j) =(l * ,t * ,r * ,b * ), representing the distance from the corresponding point in the input search region to the four sides of the predicted bounding box, is calculated using the following formula:

[0038]

[0039]

[0040] in, l is the distance between the corresponding point in the search region and the upper left corner of the predicted bounding box. * x represents the distance between the corresponding point in the search region and the upper-left corner of the predicted bounding box, where x is the x-coordinate of the corresponding point in the search region and x0 is the x-coordinate of the upper-left corner of the predicted bounding box. t represents the distance between the corresponding point in the search region and the top-left corner of the predicted bounding box. * y is the distance between the corresponding point in the search region and the top-left corner of the predicted bounding box, where y is the ordinate of the corresponding point in the search region and y0 is the ordinate of the top-left corner of the predicted bounding box. r is the distance between the corresponding point in the search region and the lower right corner of the predicted bounding box. * x1 is the x-coordinate of the lower right corner of the predicted bounding box, where x is the distance between the corresponding point in the search area and the lower right corner of the predicted bounding box. b is the distance between the corresponding point in the search area and the lower right corner of the predicted bounding box. * y1 is the distance between the corresponding point in the search area and the ordinate of the lower right corner of the predicted bounding box;

[0041] Centrality branch calculation in the classification regression network The centrality map is obtained by measuring the distance from each pixel to the center of the target. Centrality diagram A cenEach point (i,j,:) contains a 1D vector C(i,j) representing the distance between the corresponding point in the search region and the center of the target, as shown in the formula:

[0042]

[0043]

[0044] Where, Γ(m) (i，j) ) is an indicator function.

[0045] Furthermore, the classification loss function L of the classification branch cls Using the cross-entropy loss function, the regression loss function L of the regression branch reg Using the IOU loss function, the centrality loss function L for the centrality branch. cen for:

[0046]

[0047] The overall loss function L is:

[0048] L = L cls +λ1L reg +λ2L cen

[0049] Here, λ1 and λ2 are both hyperparameters.

[0050] Secondly, the present invention provides a twin network tracking system based on a spatiotemporal attention mechanism, comprising:

[0051] Data acquisition module: used to acquire template image data and search image data from the image data;

[0052] Feature extraction module: Used to input template image data and search image data into the template branch and search branch respectively for feature extraction, to obtain template feature data and search feature data;

[0053] Spatiotemporal feature processing module: used to input search feature data into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features;

[0054] Cross-correlation module: used to cross-correlate template feature data with spatiotemporal attention information features to obtain a response map;

[0055] The classification and regression module is used to input the response map into the classification and regression network for pixel-by-pixel classification and regression. It classifies each pixel, calculates the distance from the pixel to the predicted box, and obtains the classification result and bounding box.

[0056] Thirdly, the present invention provides a twin network tracking device based on a spatiotemporal attention mechanism, including a processor and a storage medium;

[0057] The storage medium is used to store instructions;

[0058] The processor is configured to operate according to the instructions to perform the steps of the method according to any of the foregoing.

[0059] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0060] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0061] This invention provides a Siamese network tracking algorithm based on a spatiotemporal attention module. Building upon the SiamFC algorithm, it introduces a spatial attention module and a temporal attention module based on nonlocal operations to model the temporal and spatial context information surrounding the target. This allows for full utilization of the interdependencies between different frames and channels, improving the network's feature extraction capabilities. Finally, the pixels in the response map are classified into foreground and background categories, and each pixel is regressed and treated as a relative bounding box. Ideally, a centrality branch is used to suppress the generation of low-quality bounding boxes, reducing computational cost and the complexity of location prediction. Attached Figure Description

[0062] Figure 1 This is a flowchart of the twin network target tracking method based on spatiotemporal attention mechanism and classification regression provided in Embodiment 1 of the present invention. Detailed Implementation

[0063] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0064] Example 1:

[0065] A twin network tracking method based on a spatiotemporal attention mechanism is described below with reference to specific embodiments. Figure 1 As shown, it includes the following steps:

[0066] Step S1: The Siamese network is mainly divided into a template branch and a search branch. Its function is to extract deep features from the image data. We crop a 127×127×3 pixel image centered on the target as the template image from the first frame of the dataset or the image captured by the camera. During the tracking process, we crop a 255×255×3 image centered on the target position in the i+1th frame as the search image. Then, we input the template image data and the search image data into the template branch and the search branch respectively for feature extraction, and obtain the template feature data w1 and the search feature data x respectively.

[0067] Step S2: Input the search feature data x into the spatiotemporal attention mechanism module to learn the temporal and spatial correlation of the feature data within T frames, and obtain the spatiotemporal attention information features.

[0068] Step S3: Cross-correlate the template feature data w1 with the spatiotemporal attention information features to obtain feature data c.

[0069] Step S4: The feature data is used as input to the classification and regression network. Pixel-by-pixel classification and regression are performed on the feature data to distinguish whether each pixel belongs to the foreground or the background. The distance from the pixel to the predicted box is calculated and the centrality branch is combined to suppress the generation of low-quality bounding boxes, reducing the amount of computation and simplifying the complexity of location prediction.

[0070] The specific steps of S1 are as follows: input the template image data and the search image data into the template branch and the search branch for feature extraction, respectively, to obtain the template feature data and the search feature data. The two branches share parameters in the process of extracting feature data, and the feature extraction process is shown in formula (1):

[0071]

[0072] Where I is the input image matrix, I((x a ,y a ),(x b ,y b )) is a submatrix of I, (x a ,y a ) and (x b ,y b ) represents the coordinates of the top left and bottom right corners of the submatrix in a rectangular coordinate system, k is the step size for extracting feature data, and f(·) is the feature extraction function.

[0073] The specific steps of S2 are as follows: The search feature data is used as input to the spatiotemporal attention module. The dimensions of the features are W×H×C×T, where W, H, C, and T represent length, width, channels, and the number of frames, respectively. The input features of the spatial attention module are represented as... Right now N = W × H; embed the input features into a Gaussian function (θ and θ). The correlation between two positions is calculated in the formula, and a weighted average is applied to each position:

[0074]

[0075] in, and These are the features at position i and position j, respectively, and the normalization factor. The function g is a univariate function, that is... These are learnable parameters; the function f is a Gaussian function:

[0076]

[0077] in, and For spatial feature weights, in order to reduce computational cost, in the function Add a 2×2 max pooling layer after g.

[0078] By using the function ω to ensure that the dimension of the weighted result is consistent with the input, we obtain the spatial attention information W×H×T×C:

[0079]

[0080] in, For spatial feature weights, As spatial features, the input features of the temporal attention module are represented as follows: Right now Attention information in the time dimension is obtained through formulas (1), (2), and (3). By fusing spatial attention information with input features, we obtain spatiotemporal attention information features:

[0081]

[0082] Among them, o i For spatiotemporal attention information features, z i For input features.

[0083] The specific steps of S3 are as follows: perform convolutional operations on the module features and spatiotemporal attention information features to obtain the response map.

[0084]

[0085] in is the convolutional feature extraction function, * is the convolutional cross-correlation operation, TS(·) is the actual attention module, b1 is the bias value at each position of the response map, x is the search feature data, and z is the template feature data.

[0086] The specific steps of S4 are: the response graph generated by the cross-correlation of the twin network. Upgrade to a higher dimension Classification branches for The classification map is obtained by predicting the category of each pixel. The regression branch calculates the distance from each pixel to the bounding box to obtain the regression map. The centrality branch calculates the distance from each pixel to the center of the target to obtain the centrality map. Each pixel (I, j) on the response map can be mapped to the top-left and bottom-right corners of the ground truth bounding boxes (x, y), (x0, y0), and (x1, y1) in the search region of the input branch. For the classification map A... cls Each (i,j,:) contains a 2D vector representing the confidence levels of the foreground and background in the search region. For regression plot A... reg Each point (i,j,:) contains a 4D vector m (i,j) =(l * ,t * ,r * ,b * ), representing the distance from the corresponding point in the input search region to the four sides of the predicted bounding box, is defined as follows:

[0087]

[0088]

[0089] in, l is the distance between the corresponding point in the search region and the upper left corner of the predicted bounding box. * x represents the distance between the corresponding point in the search region and the upper-left corner of the predicted bounding box, where x is the x-coordinate of the corresponding point in the search region and x0 is the x-coordinate of the upper-left corner of the predicted bounding box. t represents the distance between the corresponding point in the search region and the top-left corner of the predicted bounding box. * y is the distance between the corresponding point in the search region and the top-left corner of the predicted bounding box, where y is the ordinate of the corresponding point in the search region and y0 is the ordinate of the top-left corner of the predicted bounding box. r is the distance between the corresponding point in the search region and the lower right corner of the predicted bounding box. * x1 is the x-coordinate of the lower right corner of the predicted bounding box, where x is the distance between the corresponding point in the search area and the lower right corner of the predicted bounding box. b is the distance between the corresponding point in the search area and the lower right corner of the predicted bounding box. *y1 is the distance between the corresponding point in the search area and the ordinate of the lower right corner of the predicted bounding box;

[0090] The indicator function is defined as follows:

[0091]

[0092] Since pixels far from the target location center tend to produce low-quality predicted bounding boxes, affecting the algorithm's tracking performance, a centrality branch is added in parallel with the classification branch to remove outliers. The centrality branch ultimately generates a centrality feature map A. cen A cen Each point (i,j,:) contains a 1D vector C(i,j), representing the distance between the corresponding point in the search region and the center of the target.

[0093]

[0094] If the point (x,y) falls within the background region, the value of C(i,j) is 0.

[0095] The loss functions in this paper include classification loss function, regression loss function, and centrality loss function. Classification loss function L cls Using the cross-entropy loss function and the regression loss function L reg The IOU loss function is used.

[0096] Centrality loss function L cen The definition is as follows:

[0097]

[0098] The overall loss function L is shown in the following equation:

[0099] L = L cls +λ1L reg +λ2L cen (11)

[0100] Where λ1 and λ2 are hyperparameters.

[0101] This invention uses a spatiotemporal attention mechanism to effectively enhance the network model's connection to the target in the temporal and spatial context, establishes a spatiotemporal context connection for the target, and performs classification and regression learning to classify and predict each pixel to distinguish between foreground and background. It uses centrality branch to suppress the generation of low-quality bounding boxes. Combined with classification and regression, it can effectively simplify the complexity of location prediction and improve positioning accuracy.

[0102] Example 2:

[0103] A Siamese network tracking system based on a spatiotemporal attention mechanism, which can implement the Siamese network tracking method based on a spatiotemporal attention mechanism described in Embodiment 1, includes:

[0104] Data acquisition module: used to acquire template image data and search image data from the image data;

[0105] Feature extraction module: Used to input template image data and search image data into the template branch and search branch respectively for feature extraction, to obtain template feature data and search feature data;

[0106] Spatiotemporal feature processing module: used to input search feature data into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features;

[0107] Cross-correlation module: used to cross-correlate template feature data with spatiotemporal attention information features to obtain a response map;

[0108] The classification and regression module is used to input the response map into the classification and regression network for pixel-by-pixel classification and regression. It classifies each pixel, calculates the distance from the pixel to the predicted box, and obtains the classification result and bounding box.

[0109] Example 3:

[0110] This invention also provides a twin network tracking device based on a spatiotemporal attention mechanism, which can implement the twin network tracking method based on a spatiotemporal attention mechanism described in Embodiment 1, including a processor and a storage medium;

[0111] The storage medium is used to store instructions;

[0112] The processor is configured to operate according to the instructions to perform the steps of the following method:

[0113] Retrieve template image data and search image data from image data;

[0114] The template image data and the search image data are respectively input into the template branch and the search branch for feature extraction to obtain the template feature data and the search feature data.

[0115] The search feature data is input into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features;

[0116] The response map is obtained by cross-correlating the template feature data with the spatiotemporal attention information features.

[0117] The response map is input into the classification and regression network for pixel-by-pixel classification and regression. Each pixel is classified, and the distance from the pixel to the predicted box is calculated to obtain the classification result and the bounding box.

[0118] Example 4:

[0119] This invention also provides a computer-readable storage medium that implements the Siamese network tracking method based on a spatiotemporal attention mechanism as described in Embodiment 1. The medium stores a computer program that, when executed by a processor, performs the steps of the following method:

[0120] Retrieve template image data and search image data from image data;

[0121] The template image data and the search image data are respectively input into the template branch and the search branch for feature extraction to obtain the template feature data and the search feature data.

[0122] The search feature data is input into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features;

[0123] The response map is obtained by cross-correlating the template feature data with the spatiotemporal attention information features.

[0124] The response map is input into the classification and regression network for pixel-by-pixel classification and regression. Each pixel is classified, and the distance from the pixel to the predicted box is calculated to obtain the classification result and the bounding box.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A Siamese network tracking method based on a spatiotemporal attention mechanism, characterized in that, include: Acquire template image data and search image data from image data; including: cropping an image of a specified pixel size from the first frame of the dataset or camera captured by the target as template image data, and cropping an image of a set size from the target position in the (i+1)th frame during the tracking process as search image data. The template image data and search image data are input into the template branch and search branch respectively for feature extraction, resulting in template feature data and search feature data; wherein, the feature extraction formula is: ， in, It is the input image matrix. for submatrix, and Let be the coordinates of the top-left and bottom-right corners of the submatrix in a Cartesian coordinate system, and k be the step size for extracting feature data. For feature extraction functions; The search feature data is input into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features; wherein, the spatiotemporal attention information features are: ， in, For spatiotemporal attention information features, Spatial attention information features, For time attention information features, For input features; The spatial attention information features are: ， ， ， ， ， ， ， in, For spatial features, C ( ) is the normalization factor. It is a Gaussian function. and They are the locations and location The characteristic of g is that g is a univariate function. These are learnable parameters. , and All are spatial feature weights; The response map is obtained by cross-correlating the template feature data with the spatiotemporal attention information features; wherein, the response map is: ， in, For response diagrams, For convolutional feature extraction functions, For convolution cross-correlation operations, For actual attention modules, For each position in the response graph, x represents the search feature data, and z represents the template feature data; The response map is input into the classification and regression network for pixel-by-pixel classification and regression. Each pixel is classified, and the distance from the pixel to the predicted box is calculated to obtain the classification result and the bounding box.

2. The Siamese network tracking method based on spatiotemporal attention mechanism according to claim 1, characterized in that, The response map is input into a classification and regression network for pixel-by-pixel classification and regression. Each pixel is classified, and the distance from the pixel to the predicted bounding box is calculated to obtain the classification result and bounding box, including: Upgrade the response graph to a higher dimension ; The classification branch in the classification regression network is for The classification map is obtained by predicting the category of each pixel. Among them, the classification diagram Each It contains a 2D vector representing the confidence levels of the foreground and background of the search area; The regression branch calculation in the classification regression network The regression map is obtained by measuring the distance from each pixel to the bounding box. Among them, regression plot every point Contains a 4D vector , representing the distance from the corresponding point in the input search region to the four sides of the predicted bounding box, is calculated using the following formula: ， in, The distance between the corresponding point in the search area and the upper left corner of the predicted bounding box. x represents the x-coordinate distance between the corresponding point in the search region and the top-left corner of the predicted bounding box, where x is the x-coordinate of the corresponding point in the search region. To predict the x-coordinate of the top-left corner of the bounding box, The distance between the corresponding point in the search area and the top-left corner of the predicted bounding box is represented by the y-coordinate. y is the ordinate distance between the corresponding point in the search region and the top-left corner of the predicted bounding box, where y is the ordinate of the corresponding point in the search region. To predict the y-coordinate of the top-left corner of the bounding box, The distance between the corresponding point in the search area and the lower right corner of the predicted bounding box. The distance between the corresponding point in the search area and the lower right corner of the predicted bounding box. To predict the x-coordinate of the bottom right corner of the bounding box, The distance between the corresponding point in the search area and the lower right corner of the predicted bounding box is represented by the y-coordinate. The distance between the corresponding point in the search area and the lower right corner of the predicted bounding box is represented by the y-coordinate. To predict the y-coordinate of the bottom right corner of the bounding box; Centrality branch calculation in the classification regression network The centrality map is obtained by measuring the distance from each pixel to the center of the target. Centrality map every point Contains a 1D vector representing the distance between the corresponding point in the search area and the center of the target. The formula is: ， ， in, This is an indicator function.

3. The Siamese network tracking method based on spatiotemporal attention mechanism according to claim 2, characterized in that, The classification loss function of the classification branch Using the cross-entropy loss function, the regression loss function of the regression branch Using the IOU loss function, the centrality loss function of the centrality branch. for: ， The overall loss function L is: ， in, and All of these are hyperparameters.

4. A twin network tracking system based on a spatiotemporal attention mechanism, based on the twin network tracking method based on the spatiotemporal attention mechanism according to any one of claims 1-3, characterized in that, include: Data acquisition module: used to acquire template image data and search image data from the image data; Feature extraction module: Used to input template image data and search image data into the template branch and search branch respectively for feature extraction, to obtain template feature data and search feature data; Spatiotemporal feature processing module: used to input search feature data into the spatiotemporal attention mechanism module to obtain spatiotemporal attention information features; Cross-correlation module: used to cross-correlate template feature data with spatiotemporal attention information features to obtain a response map; The classification and regression module is used to input the response map into the classification and regression network for pixel-by-pixel classification and regression. It classifies each pixel, calculates the distance from the pixel to the predicted box, and obtains the classification result and bounding box.

5. A twin network tracking device based on a spatiotemporal attention mechanism, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 3.

Images