Target tracking method based on spatio-temporal feature fusion

Abstract

The application discloses a target tracking method based on space-time feature fusion, comprising the following steps: obtaining an initial frame picture and a current frame picture as a template picture and a search picture respectively, inputting the template picture and the search picture into a target tracking model, using a feature extraction network to extract features of the template picture and the search picture respectively, and obtaining a template depth feature map and a search depth feature map; using a memory network to multiply the template depth feature map with a stacked first memory frame and multiply the search depth feature map with a stacked second memory frame in a pixel level manner, obtaining a template attention map and a search attention map; and respectively performing feature information interaction in a channel dimension, obtaining a template feature and a search region feature; using a feature saliency network to process the template feature and the search region feature respectively, obtaining a template saliency feature map and a search saliency feature map; and using a Head network to determine a target position based on the template saliency feature map and the search saliency feature map.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a target tracking method based on spatiotemporal feature fusion. Background Technology

[0002] Visual tracking has received widespread attention over the past few decades. The goal of tracking is to locate a target marked in the initial frame throughout a video sequence. However, with the increasing demand for practical applications such as intelligent video surveillance, autonomous driving, video human-computer interaction, and drone tracking, current visual trackers still struggle to achieve satisfactory results when faced with technical challenges such as appearance changes, interference, and occlusion.

[0003] In the early stages, as research focus shifted to Siamese networks, the tracking task could be described as a simple template matching process. These Siamese network methods could address some common problems by utilizing cross-correlation operations. However, these popular Siamese trackers have two drawbacks: (1) they process all spatial features in the same way, ignoring more salient target content; and (2) unlike the human eye, which can capture dynamic information, the tracking process is statically implemented. Therefore, changes in target appearance and environmental context are largely ignored in these methods.

[0004] Therefore, in recent years, efforts have been made to embed memory networks into Siamese networks when observing targets. Among these methods, memory networks that incorporate temporal information have been proposed. One researcher has developed a fast learning model that can effectively utilize appearance changes and background suppression to learn objects online in the first few frames; another has introduced a score prediction head scheme to determine whether to update the template; and yet another has proposed a memory network to store prior information about the target and avoid the template update process, thus achieving template-free tracking.

[0005] Despite the breakthroughs achieved by the aforementioned methods, it remains difficult to fully utilize salient features, which manifests in three aspects: First, traditional memory networks are limited to the spatial dimension, neglecting useful channel information. Second, memory networks only consider template update methods around the search region, or attention-like mechanisms, and few methods unify them, thus resulting in insufficient utilization of memory frames. Furthermore, it is difficult to integrate features from both spatial and temporal dimensions.

[0006] In related technologies, patent application CN111915648A proposes a long-term target motion tracking method based on common sense and memory networks. It uses a single-layer memory network, obtains filters by storing templates or memory features, and then convolves the filters with the current features. A threshold is used to determine whether the convolution result meets the conditions, thus completing the construction of the memory network. Furthermore, this scheme first obtains the tracking result, then compares it with the filters in the memory network. If the result meets the conditions, the tracking result is directly used as the output; otherwise, a semantic target-aware feature image set is used to re-localize the target in the current frame image before proceeding with subsequent operations. Overall, the algorithm is quite complex.

[0007] The patent application document with publication number CN113052875A proposes a target tracking algorithm based on state-aware template update. The algorithm updates the template based on three feature maps: the initial target template, the target template used in the current frame, and the template extracted from the target being tracked in the current frame. It uses convolution and activation to directly update the template on the three concatenated feature maps, which is a meta-learning idea. Summary of the Invention

[0008] The technical problem to be solved by this invention is how to make full use of salient features to improve target tracking performance.

[0009] The present invention solves the above-mentioned technical problems through the following technical means: A target tracking method based on spatiotemporal feature fusion is proposed, the method comprising: The initial frame image and the current frame image are used as the template image and the search image, respectively. The template image and the search image are used as inputs to the target tracking model, which includes a feature extraction network, a memory network, a feature saliency network and a head network. The template image and the search image are used as inputs to the feature extraction network, and the output of the feature extraction network is fed into the head network via the memory network and the feature saliency network. A feature extraction network is used to extract features from the template image and the search image respectively, resulting in a template depth feature map and a search depth feature map. The template depth feature map is multiplied with the stacked first memory frame and the search depth feature map is multiplied with the stacked second memory frame in a pixel-level manner using a memory network to obtain the template attention map and the search attention map. The template attention map and the second memory frame are then interacted with each other in the channel dimension, and the search attention map and the first memory frame are also interacted with each other in the channel dimension to obtain the template features and the search region features. The template features and the search region features are processed using a feature saliency network to obtain a template saliency feature map and a search saliency feature map, respectively. The target location is determined using the Head network based on the template salient feature map and the search salient feature map.

[0010] Furthermore, the backbone network of the feature extraction network adopts a ResNet-50 network structure. The feature extraction network is used to extract features from the template image and the search image respectively, obtaining a template depth feature map and a search depth feature map, including: The template image and the search image are mapped to the feature space using the forward propagation of the backbone network; The corresponding feature maps are extracted using the last three layers of the backbone network and downsampled to obtain the template depth feature map and the search depth feature map.

[0011] Furthermore, the memory network includes a first non-local module, a second non-local module, and a cross-attention module. The input of the first non-local module is the first memory frame and the template depth feature map, and the input of the second non-local module is the second memory frame and the search depth feature map. The first non-local module performs a reshape operation on the first memory frame and multiplies it with the template depth feature map in a pixel-level manner to obtain a template attention map, and multiplies the reshaped first memory frame with the original first memory frame to obtain a first reconstructed memory feature. The second non-local module performs a reshape operation on the second memory frame and multiplies it with the search depth feature map at the pixel level to obtain a search attention map, and multiplies the reshaped second memory frame with the original second memory frame to obtain a second reconstructed memory feature; The cross-attention module multiplies the template attention map with the second reconstructed memory feature and the search attention map with the first reconstructed memory feature to obtain the template feature information interaction map and the search feature information interaction map. The template feature is obtained by concatenating the template feature information interaction map with the template depth feature map; The search region features are obtained by concatenating the search feature information interaction map with the search depth feature map.

[0012] Furthermore, the calculation expressions for the template attention map and the search attention map are respectively:

[0013]

[0014] In the formula, and These are the template attention map and the search attention map, respectively. It is a template depth feature map The features obtained after one convolution operation It is a search for deep feature maps The features obtained after one convolution operation This is the first memory frame. This is the second memory frame. The residual structure is composed of learnable hyperparameters. and These are the first and second memory frames after the Reshape operation, respectively. This represents the dot product.

[0015] Furthermore, the calculation expression for the template feature information interaction graph and the search feature information interaction graph is as follows:

[0016]

[0017] In the formula, and These are respectively the template feature information interaction graph or the search feature information interaction graph. and These are the template attention map and the search attention map, respectively. and This represents the output inferred through the cross-attention module. and These are the first reconstructed memory features and the second reconstructed memory features, respectively. This represents the dot product.

[0018] Furthermore, the calculation expressions for the template features and the search region features are as follows:

[0019]

[0020] In the formula, and This refers to the interaction graph between the template feature information and the search feature information. For template depth feature map, To search for deep feature maps, For template features, Features of the search area.

[0021] Furthermore, the feature saliency network includes a first attention module, a second attention module, a first spatial attention convolutional block, and a second spatial attention convolutional block. Both the first attention module and the second attention module include parallel channel self-attention sub-modules and spatial self-attention sub-modules. The template features and the search region features are used as inputs to the first attention module and the second attention module, respectively. The outputs of the two parallel sub-modules in the first attention module are fused together and used as inputs to the first spatial attention convolutional block. The result of fusing the output of the first spatial attention convolutional block with the outputs of the two parallel sub-modules is then fused to obtain the template saliency feature map. The outputs of the two parallel sub-modules in the second attention module are fused together and used as the input of the second spatial attention convolutional block. The result of fusing the output of the second spatial attention convolutional block with the outputs of the two parallel sub-modules is then fused to obtain the search salient feature map.

[0022] Furthermore, the convolutional and fully connected layers in the first and second spatial attention convolutional blocks are replaced with GELU functions.

[0023] Furthermore, the step of using the Head network to determine the target location based on the template salient feature map and the search salient feature map includes: The template salient feature map and the search salient feature map are input into the Head network to obtain the score map and regression map. The target location is determined based on the regression graph.

[0024] Furthermore, before taking the template image and the search image as inputs to the target tracking model, the method further includes: Select the first video sequence and sample positive sample pairs, then randomly select a frame as the search frame. m One frame is used as a memory frame; Select the second video sequence and perform negative sample pair sampling, randomly select a frame as the template frame, select the third video sequence and perform negative sample pair sampling, and randomly select a frame as the search frame. Randomly select several different third video sequences, and randomly select one frame from each of these third video sequences as a memory frame; The target tracking model is trained using the positive and negative sample pairs obtained from sampling, and the selected memory frames are divided into the first memory frame and the second memory frame according to a set ratio.

[0025] There is a certain probability of sampling positive sample pairs. In the video sequence, two frames are randomly selected as the search frame and the template frame, respectively. At the same time, three frames are randomly selected from the same video sequence as memory frames. There is a certain probability of sampling negative sample pairs. In the video sequence, a frame is randomly selected as the template frame, then a video sequence is randomly selected, and a frame is selected as the search frame. This process is repeated three times, with three different video sequences being randomly selected, and a frame from each video sequence is selected as the memory frame. The target tracking model is trained using the positive and negative sample pairs obtained from sampling.

[0026] The advantages of this invention are: (1) This invention first uses a feature extraction network to extract the template depth feature map corresponding to the template image and the search depth feature map corresponding to the search image. Then, the template depth feature map and the search depth feature map are fed into a memory network. The template depth feature map is multiplied with the first stacked memory frame and the search depth feature map is multiplied with the second stacked memory frame in a pixel-level manner to obtain the template attention map and the search attention map. Since the object itself contains rich semantic information, the frame image is multiplied with the stacked memory frames in a pixel-level manner so that the obtained spatial feature map contains the information in the memory frame and emphasizes the spatial position of the target. Feature information interaction is performed in the channel dimension to learn the internal representation of the object in the channel dimension. Channels that are more similar to the template image and channels that are more similar to the search image can obtain higher responses, thereby solving the problem of mismatch between the memory frame and the current frame. Compared with existing related tracking algorithms, the target tracking method proposed in this invention has achieved competitive accuracy and success rate results.

[0027] (2) In the feature saliency network part, it is built on the basis of CBAM and DANet. These two modules are highly efficient and have short inference time. The proposed attention module solves the incoordination problem between non-local modules and convolution. Moreover, through parallel channel self-attention sub-modules and spatial self-attention sub-modules, the extracted features are more complete and the characteristics are more prominent, and feature fusion can be achieved simply and quickly.

[0028] (3) By replacing the convolutional and fully connected layers in the first and second spatial attention convolutional blocks with GELU, the idea of ​​random regularization in activation is introduced, making the understanding more intuitive and having stronger nonlinearity and generalization ability.

[0029] (4) During the model training phase, the sampling strategy set can provide higher quality memory frames, thereby achieving better tracking results.

[0030] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the target tracking method based on spatiotemporal feature fusion proposed in an embodiment of the present invention; Figure 2 This is a network structure diagram of the target tracking model proposed in this embodiment of the invention; Figure 3 This is a structural diagram of the memory network in an embodiment of the present invention; Figure 4 These are structural diagrams of the channel attention convolutional block and the spatial attention convolutional block in the feature saliency network in this embodiment of the invention. (a) is the structural diagram of the CDM1 submodule, and (b) is the structural diagram of the CDM2 submodule. Figure 5 This is a conceptual diagram of the sampling strategy in an embodiment of the present invention; Figure 6 The target tracking algorithm proposed in this embodiment of the invention is compared and evaluated with nine other high-performance mainstream algorithms on the OTB100 benchmark dataset. Among them, (a) is the success rate map and (b) is the accuracy map. Detailed Implementation To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] like Figures 1 to 2 As shown in the figure, this invention proposes a target tracking method based on spatiotemporal feature fusion, the method comprising the following steps: S10. Obtain the initial frame image and the current frame image as the template image and the search image, respectively; It should be noted that the frame images obtained in this embodiment may be frame images from intelligent video surveillance video sequences, frame images from video sequences captured during autonomous driving, frame images from video sequences captured during human-computer interaction, frame images from environmental video sequences captured during drone tracking, etc.

[0033] It should be noted that the target tracking task refers to locating the target marked in the initial frame in the entire video sequence. Therefore, in this embodiment, the initial frame image in the video sequence is used as the template image, and the remaining frame images are used as search images in turn to perform target tracking.

[0034] S20. The template image and the search image are respectively used as inputs to the target tracking model, which includes a feature extraction network, a memory network, a feature saliency network and a head network. The template image and the search image are used as inputs to the feature extraction network. The output of the feature extraction network is connected to the input of the head network through the memory network and the feature saliency network. S30. Use a feature extraction network to extract features from the template image and the search image respectively, to obtain the template depth feature map and the search depth feature map; S40. Using a memory network, the template depth feature map is multiplied with the stacked first memory frame in a pixel-level manner, and the search depth feature map is multiplied with the stacked second memory frame to obtain the template attention map and the search attention map; and the template attention map and the second memory frame are respectively interacted in the channel dimension to obtain the template features and the search region features. S50. The template features and the search region features are processed using a feature saliency network to obtain a template saliency feature map and a search saliency feature map, respectively. S60. Using the Head network, determine the target location based on the template salient feature map and the search salient feature map.

[0035] In this embodiment, after extracting the template depth feature map and the search depth feature map, the template depth feature map and the search depth feature map are fed into the memory network. The template depth feature map is multiplied with the stacked first memory frame in a pixel-level manner, and the search depth feature map is multiplied with the stacked second memory frame to obtain the template attention map and the search attention map. Since the self-attention map itself contains rich semantic information and the memory frame contains rich temporal information, the frame image is multiplied with the stacked memory frame in a pixel-level manner so that the obtained spatial feature map contains the information in the memory frame and emphasizes the spatial location of the target. After fusing the memory frames, the feature information of the obtained features is interacted between the two branches (template and search) in the channel dimension in the memory dimension. The internal representation of the object in the channel dimension is learned, so that the prominent channels in the template features strengthen the features of the search branch, and the prominent channels in the search features strengthen the features of the template branch, thereby achieving competitive accuracy and success rate results.

[0036] In one embodiment, the backbone network of the feature extraction network adopts a ResNet-50 network structure. The feature extraction network is used to extract features from the template image and the search image respectively, obtaining a template depth feature map and a search depth feature map, including: The template image and the search image are mapped to the feature space using the forward propagation of the backbone network; The corresponding feature maps are extracted using the last three layers of the backbone network and downsampled to obtain the template depth feature map and the search depth feature map.

[0037] Specifically, the inputs to the feature extraction network are a 3×127×127 template image and a 3×255×255 search image. Using ResNet-50 as the backbone network, corresponding feature maps are extracted from the last three layers of the backbone network to achieve multi-level prediction. The three feature spaces extracted by the backbone network have sizes of 512×15×15 and 512×31×31, respectively. Downsampling of these three feature maps simplifies the feature processing, thus facilitating computation. Finally, lists containing three feature maps are obtained: a template depth feature map and a search depth feature map. The feature map sizes in these two lists are 256×7×7 and 256×31×31, respectively.

[0038] In one embodiment, such as Figure 3 As shown, the memory network includes a first non-local module, a second non-local module, and a cross-attention module. The input of the first non-local module is the first memory frame and the template depth feature map, and the input of the second non-local module is the second memory frame and the search depth feature map. The first non-local module performs a reshape operation on the first memory frame and multiplies it with the template depth feature map in a pixel-level manner to obtain a template attention map, and multiplies the reshaped first memory frame with the original first memory frame to obtain a first reconstructed memory feature. The second non-local module performs a reshape operation on the second memory frame and multiplies it with the search depth feature map at the pixel level to obtain a search attention map, and multiplies the reshaped second memory frame with the original second memory frame to obtain a second reconstructed memory feature; The cross-attention module multiplies the template attention map with the second reconstructed memory feature and the search attention map with the first reconstructed memory feature to obtain the template feature information interaction map and the search feature information interaction map. The template feature is obtained by concatenating the template feature information interaction map with the template depth feature map; The search region features are obtained by concatenating the search feature information interaction map with the search depth feature map.

[0039] It should be noted that the first memory frame Second memory frame The memory frames are stored in the template branch memory and the search branch memory, respectively. The memory network extracts the memory frame information from the memory and reconstructs it using the Reshape operation. Then, the Non-local module multiplies the reconstructed memory frame with the template depth feature map or the search depth feature map in a pixel-level manner to obtain a pixel-level attention map. The updated template features and refined search region features are obtained by tensor multiplication of the original features and the attention map, and these features maintain the original depth feature map size.

[0040] This embodiment constructs a memory network using a memory bank approach, obtaining rich memory feature representations of the target using stacked memory frames, and passing these memory frames to the current tracking process in a non-local manner. After fusing the memory frames, the memory network performs feature information interaction between the two branches (template and search) in the memory dimension. By using cross attention, prominent channels in the template features strengthen the search features, and prominent channels in the search features strengthen the features of the template branch.

[0041] This embodiment, by setting up memory frames, firstly highlights the feature representation of objects in a non-local manner, thereby improving the target object's responsiveness in both channel and spatial ranges. Secondly, by introducing memory frames, historical information is utilized during tracking, thus incorporating the prior knowledge from the memory frames into the target template. The target's appearance and state information from the memory frames is stored in a memory bank, and this robust target template is used for tracking to obtain accurate tracking results. The non-local approach yields attention maps in both channel and spatial ranges, resulting in a more responsive template in the channel space compared to the salient representation of the target obtained through meta-learning.

[0042] In one embodiment, after the first memory frame is reconstructed, it is multiplied with the template depth feature map at the pixel level to obtain a template attention map. Specifically, the reconstructed first memory frame is multiplied with the template depth feature map, and the multiplication result is multiplied with the original first memory frame to obtain the template attention map. .

[0043] After reconstruction, the second memory frame is multiplied with the search depth feature map at the pixel level to obtain the search attention map. Specifically, the reconstructed second memory frame is multiplied with the search depth feature map, and the multiplication result is multiplied with the original second memory frame to obtain the search attention map. .

[0044] In one embodiment, the calculation expressions for the template attention map and the search attention map are as follows:

[0045]

[0046] In the formula, and These are the template attention map and the search attention map, respectively. It is a template depth feature map The features obtained after one convolution operation It is a search for deep feature maps The features obtained after one convolution operation, This is the first memory frame. This is the second memory frame. The residual structure is composed of learnable hyperparameters. and These are the first and second memory frames after the Reshape operation, respectively. This represents the dot product.

[0047] It should be noted that, and They are respectively K and V , K and V It is the input to the Transformer, and it is found in the methods of the Transformer class. K and Q Perform a dot product to obtain an attention map, then combine it with... V Perform a dot product to obtain the features after attention enhancement.

[0048] Furthermore, the application of the cross-attention module can extend the non-block module in the channel dimension, satisfying the collaboration between the two branches. Through the cross-attention module, the template feature information interaction graph and the search feature information interaction graph are obtained, and the calculation expression is:

[0049]

[0050] In the formula, and These are respectively the template feature information interaction graph or the search feature information interaction graph. and These are the template attention map and the search attention map, respectively. and This represents the output inferred through the cross-attention module. and These are the first reconstructed memory features and the second reconstructed memory features, respectively. This represents the dot product.

[0051] In this embodiment, a cross-attention module is added between the memory features of the template branch and the search feature map. This cross-attention module calculates the cross-attention features of the template feature map and the cross-attention features of the search feature map, and then performs information fusion of the template features and the search features, so that the search features can perceive the information of the template feature map in advance, thereby increasing the accuracy.

[0052] In one embodiment, the calculation expressions for the template features and the search region features are as follows:

[0053]

[0054] In the formula, and This refers to the interaction graph between the template feature information and the search feature information. For template depth feature map, To search for deep feature maps, For template features, Features of the search area.

[0055] In one embodiment, such as Figure 2 and Figure 4 As shown, the feature saliency network includes a first attention module, a second attention module, a first spatial attention convolutional block, and a second spatial attention convolutional block. Both the first attention module and the second attention module include parallel channel self-attention submodules CDM1 and spatial self-attention submodules CDM2. The template features and the search region features are used as inputs to the first attention module and the second attention module, respectively. The outputs of the two parallel submodules in the first attention module are fused together and used as inputs to the first spatial attention convolutional block. The output of the first spatial attention convolutional block is fused with the outputs of the two parallel submodules, and then fused together to obtain the template saliency feature map. The outputs of the two parallel sub-modules in the second attention module are fused together and used as the input of the second spatial attention convolutional block. The result of fusing the output of the second spatial attention convolutional block with the outputs of the two parallel sub-modules is then fused to obtain the search salient feature map.

[0056] It should be noted that the feature saliency network part of the target tracking model is built upon the methods of CBAM and DANet, both of which are highly efficient and have short inference times. Inspired by them, this embodiment proposes an attention module to address the inconsistency between non-local and convolutional methods. This attention module consists of two parallel sub-modules: channel self-attention (CDM1) and spatial self-attention (CDM2). Feature fusion is performed using a simple feature fusion method, which accelerates the fusion process. By parallelizing the two attention sub-modules in the feature saliency network, this embodiment can infer salient feature maps.

[0057] Furthermore, in submodules CDM1 and CDM2, K and Q Obtained from a single convolutional layer. K and Q It is a projection of the original feature map, which increases the representational power in different feature spaces. K and Q It is responsible for generating attention maps in the channel dimension.

[0058] Spatial attention convolutional blocks in CBAM are used to generate V This process can be referred to in the diagram as V - Create submodules, V A feature map representing the information being queried.

[0059] K and V are the inputs to the Transformer. In the Transformer class methods, K and Q are multiplied to obtain the attention map, which is then multiplied with V to obtain the attention-enhanced features.

[0060] Furthermore, such as Figure 4 As shown, the network structure of the channel self-attention submodule CDM1 is a submodule based on the channel attention map. First, the network structure of the channel self-attention submodule CDM1 is... K and Q Perform a dot product to obtain the channel attention map. Then, the original feature map is processed... V - Create a submodule (a) to achieve space enhancement. V .Will K, Q, V The input to the Transformer yields a feature map with significantly enhanced saliency.

[0061] The spatial self-attention submodule CDM2 has a network structure based on spatial attention graphs. First, the network structure of the CDM2 submodule is... K transpose and Q The transpose of the original feature map is multiplied by a dot product to obtain the spatial attention map. Then, the original feature map is processed... V - Create submodule (b) to obtain channel enhancement V .Will K, Q, V The input to the Transformer yields a feature map with significantly enhanced saliency.

[0062] The channel self-attention submodule CDM1 and spatial self-attention submodule CDM2 designed in this embodiment are not obtained by convolution on the original feature map during the generation of V, but rather by creating a V submodule. This V creation submodule supplements the self-attention module. That is, if the self-attention module generates a channel attention map, this submodule enhances the spatial features of the V feature map to obtain a spatial representation of the salient target. If the self-attention module generates a spatial attention map, this submodule enhances the channel features of the V feature map to learn the feature responses between different channels.

[0063] Furthermore, such as Figure 4 As shown, the convolutional and fully connected layers in the first and second spatial attention convolutional blocks are replaced with GELU functions.

[0064] The first spatial attention convolutional block focuses on the channel features of the target when generating the attention map, enhancing the response of the features in the channel; the second spatial attention convolutional block focuses on the spatial features of the target when generating the attention map, only focusing on the spatial position of the target.

[0065] Specifically, the structure of the first spatial attention convolutional block is as follows: First, max pooling and average pooling operations are applied to the spatial dimension of the input feature map. Then, the resulting max-pooled and average-pooled feature maps are concatenated, and after a convolutional layer for dimensionality reduction, the final output is obtained. Its mathematical expression can be described as follows:

[0066]

[0067] Where G is the GELU function, Conv represents the convolution operation, C represents the concatenation operation, Me is the average value, Ma is the maximum value, IQ is the intermediate variable, input is the input feature map, and output is the output feature map.

[0068] The structure of the second-space attention convolutional block is as follows: First, dimensionality reduction is used to reduce the number of channels in the feature map, thereby reducing the subsequent computational cost. Second, global max pooling and global average pooling operations are applied in the channel dimension. After adding the resulting feature vectors, the channel attention vectors are obtained. Then, the vectors are multiplied with the original feature map to obtain the channel-enhanced feature map, the mathematical expression of which is shown below:

[0069]

[0070] in, and These are global max pooling and global average pooling, with Soft() representing the softmax operation.

[0071] Considering that the template branch and the search branch may not focus on the same details, this embodiment uses the same architecture with different parameters to form the final module. Furthermore, the convolutional and fully connected layers in the generative submodule V are replaced with GELU. The GELU method introduces the idea of ​​randomized regularization in activation, which makes it more intuitive to understand and has stronger nonlinearity and generalization capabilities.

[0072] In one embodiment, step S60: determining the target location using the Head network based on the template salient feature map and the search salient feature map, specifically includes the following steps: The template salient feature map and the search salient feature map are input into the Head network to obtain the score map and regression map. The target location is determined based on the regression graph.

[0073] In one embodiment, before performing target tracking using a target tracking model, the method further includes the following sampling step: Specifically, the sampling strategy is as follows: Figure 5 As shown, the memory bank is a control module designed to store memory frames. In this embodiment, a template branch memory bank and a search branch memory bank are set up, and an unspecified number of memory frames are stored in the memory bank.

[0074] By sampling positive sample pairs in a video sequence under normal conditions, these frames are selected as the current frame. From the same video sequence, three frames are randomly selected as search memory frames and added to the memory bank. During training, the feature maps obtained from these three frames are cropped to obtain three template memory frames. The sampling method helps to crop the memory frames according to the size defined by the template and search branch. By setting a threshold, negative sample pairs are sampled with probability. Six memory frames are randomly extracted from a video sequence different from the video sequence from which the current frame is picked (half forming memory frames in the template branch, and the other half forming memory frames in the search branch). During sampling, the memory bank retains memory frames in both the template and search branches.

[0075] Specifically, in this embodiment, the sampling probability of positive samples is set to 0.75, and the sampling probability of negative samples is set to 0.25. That is, the input of the network has a probability of 0.75 as a positive sample pair, and the input of the network has a probability of 0.25 as a negative sample pair.

[0076] It should be understood that other numbers of memory frames could be selected in this embodiment, but 3 frames is a relatively better choice after actual comparison. In this method, memory frames need to be fused during training, so the more frames in a video sequence used as memory frames, the better, because these frames can simulate the changes in the appearance of objects at different times; however, in actual training, selecting more frames often means greater memory consumption and training time. Therefore, selecting 3 frames as the number of memory frames in this method is more appropriate, on the one hand extracting as many memory frames as possible, and on the other hand enabling the model to be trained effectively (shortening training time).

[0077] It should be noted that this embodiment, by setting a novel sampling strategy and using conditions to filter memory frames, can provide higher-quality memory frames. Memory frames that meet the conditions are sent to the memory network for recording and extracting historical information, thereby achieving better tracking results. Compared to existing tracking algorithms, this invention achieves better tracking performance, as verified by the following experiments: The dataset used in this embodiment is OTB100 [Yi Wu, Jongwoo Lim, and Ming-HsuanYang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1834–1848, 2015. 2]. It consists of 100 video frames and was proposed in 2015. Different datasets are also labeled with different attributes, a total of 11 different attributes, which may represent common difficulties in the field of object tracking. For example, illumination variation (IV), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-plane rotation (OPR), out-of-view (OV), background similarity (BC), and low resolution (LR).

[0078] The quality of the tracking algorithm depends on the accuracy map and success rate map of the OTB100 dataset. The accuracy map refers to the percentage of video frames in which the center point of the target location estimated by the tracking algorithm is less than a given threshold distance from the manually marked target center point. Since the accuracy map cannot reflect changes in the size and proportion of the target object, a success rate map was proposed. The success rate map calculates the percentage of frames with a given overlap rate threshold out of the total number of frames. The formula for calculating the overlap rate is as follows:

[0079] in, It's the overlap rate. It is the bounding box region obtained by the tracking algorithm. It is the bounding box region of the true value. It's an intersection operation. It is a union operation.

[0080] like Figure 6 As shown, the tracking algorithm proposed in this embodiment of the invention was tested on the OTB100 dataset, and the results were compared with those of nine mainstream algorithms in recent years. The algorithm in this embodiment of the invention achieved good results.

[0081] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0082] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0083] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A target tracking method based on spatiotemporal feature fusion, characterized in that, The method includes: The initial frame image and the current frame image are used as the template image and the search image, respectively. The template image and the search image are used as inputs to the target tracking model, which includes a feature extraction network, a memory network, a feature saliency network and a head network. The template image and the search image are used as inputs to the feature extraction network, and the output of the feature extraction network is fed into the head network via the memory network and the feature saliency network. A feature extraction network is used to extract features from the template image and the search image respectively, resulting in a template depth feature map and a search depth feature map. The template depth feature map is multiplied with the stacked first memory frame and the search depth feature map is multiplied with the stacked second memory frame in a pixel-level manner using a memory network to obtain the template attention map and the search attention map. The template attention map and the second memory frame are then interacted with each other in the channel dimension, and the search attention map and the first memory frame are also interacted with each other in the channel dimension to obtain the template features and the search region features. The template features and the search region features are processed using a feature saliency network to obtain a template saliency feature map and a search saliency feature map, respectively. The target location is determined using a Head network based on the template salient feature map and the search salient feature map; The backbone network of the feature extraction network adopts a ResNet-50 network structure. The memory network includes a first non-local module, a second non-local module, and a cross-attention module. The input of the first non-local module is the first memory frame and the template depth feature map, and the input of the second non-local module is the second memory frame and the search depth feature map. The outputs of both the first and second non-local modules are connected to the cross-attention module. The first non-local module performs a reshape operation on the first memory frame and multiplies it with the template depth feature map at the pixel level to obtain a template attention map. It also multiplies the reshaped first memory frame with the original first memory frame to obtain a first reconstructed memory feature. The second non-local module performs a reshape operation on the second memory frame and multiplies it with the search depth feature map at the pixel level to obtain a search attention map. It also multiplies the reshaped second memory frame with the original second memory frame to obtain a second reconstructed memory feature. The cross-attention module multiplies the template attention map with the second reconstructed memory feature and the search attention map with the first reconstructed memory feature to obtain a template feature information interaction map and a search feature information interaction map. The template feature is then concatenated with the template depth feature map to obtain the template feature. Finally, the search feature information interaction map is concatenated with the search depth feature map to obtain the search region feature. The feature saliency network includes a first attention module, a second attention module, a first spatial attention convolutional block, and a second spatial attention convolutional block. Both the first and second attention modules include parallel channel self-attention sub-modules and spatial self-attention sub-modules. Template features and search region features are used as inputs to the first and second attention modules, respectively. The output of the first attention module is connected to the first spatial attention convolutional block, and the output of the second attention module is connected to the second spatial attention convolutional block.

2. The target tracking method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The step of using a feature extraction network to extract features from the template image and the search image respectively, to obtain a template depth feature map and a search depth feature map, includes: The template image and the search image are mapped to the feature space using the forward propagation of the backbone network; The corresponding feature maps are extracted using the last three layers of the backbone network and downsampled to obtain the template depth feature map and the search depth feature map.

3. The target tracking method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The calculation expressions for the template attention map and the search attention map are as follows: In the formula, and These are the template attention map and the search attention map, respectively. It is a template depth feature map The features obtained after one convolution operation It is a search for deep feature maps The features obtained after one convolution operation This is the first memory frame. This is the second memory frame. The residual structure is composed of learnable hyperparameters. and These are the first and second memory frames after the Reshape operation, respectively. This represents the dot product.

4. The target tracking method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The calculation formula for the template feature information interaction graph and the search feature information interaction graph is as follows: In the formula, and These are respectively the template feature information interaction graph or the search feature information interaction graph. and These are the template attention map and the search attention map, respectively. and This represents the output inferred through the cross-attention module. and These are the first reconstructed memory features and the second reconstructed memory features, respectively. This represents the dot product.

5. The target tracking method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The calculation expressions for the template features and the search region features are as follows: In the formula, and This refers to the interaction graph between the template feature information and the search feature information. For template depth feature map, To search for deep feature maps, For template features, Features of the search area.

6. The target tracking method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The outputs of the two parallel sub-modules in the first attention module are fused together and used as the input of the first spatial attention convolutional block. The output of the first spatial attention convolutional block is then fused with the outputs of the two parallel sub-modules to obtain the template salient feature map. The outputs of the two parallel sub-modules in the second attention module are fused together and used as the input of the second spatial attention convolutional block. The result of fusing the output of the second spatial attention convolutional block with the outputs of the two parallel sub-modules is then fused to obtain the search salient feature map.

7. The target tracking method based on spatiotemporal feature fusion as described in claim 6, characterized in that, The convolutional and fully connected layers in the first and second spatial attention convolutional blocks are replaced with GELU functions.

8. The target tracking method based on spatiotemporal feature fusion as described in claim 1, characterized in that, The step of using a Head network to determine the target location based on the template salient feature map and the search salient feature map includes: The template salient feature map and the search salient feature map are input into the Head network to obtain the score map and the regression map; The target location is determined based on the regression graph.

9. The target tracking method based on spatiotemporal feature fusion as described in any one of claims 1-8, characterized in that, Before taking the template image and the search image as inputs to the target tracking model, respectively, the method further includes: Select the first video sequence and sample positive sample pairs, then randomly select a frame as the search frame. m One frame is used as a memory frame; Select the second video sequence and perform negative sample pair sampling, randomly select a frame as the template frame, select the third video sequence and perform negative sample pair sampling, and randomly select a frame as the search frame. Randomly select several different third video sequences, and randomly select one frame from each of these third video sequences as a memory frame; The target tracking model is trained using the positive and negative sample pairs obtained from sampling, and the selected memory frames are divided into the first memory frame and the second memory frame according to a set ratio.

Images