A target tracking method based on gated attention mechanism and spatiotemporal memory network

Abstract

The present invention relates to the field of target tracking technology in computer vision. Specifically, it discloses a target tracking method based on a gated attention mechanism and a spatiotemporal memory network. The method primarily comprises a feature extraction module, a gated attention mechanism feature fusion module, a spatiotemporal memory feature reading module, and a dual-branch prediction module. The method inputs a memory frame image and a current query frame image into a backbone network to extract features; employs a gated attention mechanism to fuse and enhance the memory frame image features and the query frame image features; utilizes a dual-branch detection module to respectively predict the target location box and the segmentation mask results; and dynamically selects the memory frame template based on the confidence score and temporal distance constraints. The method enhances the fusion of memory frame image features and query frame image features through a gated attention mechanism, fully utilizing spatiotemporal memory information, effectively reducing interference from situations such as long-term target occlusion and high non-target similarity, and improving the accuracy and robustness of the tracking model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a target tracking method based on a gated attention mechanism and a spatiotemporal memory network. Background Art

[0002] Vision object tracking is a key research area in computer vision. Its task is to mark the position and state of the target in the initial frame or a specific frame of a video sequence, so that the tracker can automatically track the target in all subsequent video sequences. Vision tracking technology has a wide range of applications in autonomous driving, video surveillance, motion analysis, medical diagnosis, and many other areas.

[0003] Research methods for visual object tracking primarily include early classic algorithms such as optical flow and correlation filter-based algorithms, as well as current mainstream deep learning methods. Deep learning methods are further categorized into two main frameworks: Siamese network-based methods transform the tracking task into a similarity matching problem between images; and Transformer-based methods introduce a self-attention mechanism to dynamically weight the target region of interest.

[0004] Most tracking methods based on Siamese networks use the initial or previous frame as a template image, which has limited adaptability to complex situations such as target shape, changing lighting conditions, and occlusion by interfering objects. Continuously increasing the number of templates consumes significant computational resources, impacting real-time tracking. The Transformer architecture can capture long-range dependencies and global contextual information, making it more adaptable to changes in targets and scenes. However, its training complexity is high and it is susceptible to input data noise. Spatiotemporal memory refers to information stored by the model about the timing and position of the target. Incorporating this information into the network can improve target tracking performance. Existing tracking methods based on spatiotemporal memory networks use a memory mechanism to store multi-frame historical information about the target, perform global feature matching, and dynamically update the template. This improves the tracker's adaptability to targets to a certain extent, but it still underutilizes the spatiotemporal memory information between memory frames. Under interference conditions such as long-term and large-scale occlusion of the target or high non-target similarity, it is prone to tracking loss, template contamination, resulting in target frame switching, and inaccuracy. Summary of the Invention

[0005] To address the shortcomings of existing tracking technologies, the present invention provides a target tracking method based on a gated attention mechanism and a spatiotemporal memory network. This method can fully utilize spatiotemporal memory information, focus on the target area of ​​interest in the memory information, and effectively improve the performance of the tracker in complex scenarios such as target shape changes, long-term occlusion, and high similarity with interfering objects.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows: a target tracking method based on a gated attention mechanism and a spatiotemporal memory network, the method steps are as follows:

[0007] S1 Feature Extraction: The memory frame image and the query frame image are respectively sent to the backbone network to extract the corresponding memory frame image features and query frame image features;

[0008] S2 feature enhancement: The adaptive gated attention mechanism is used to enhance the memory frame image features and the query frame image features. The dynamic gating unit adaptively selects a pair of memory frame image features and query frame image features from the three attention modules: spatial attention, channel attention, and channel-spatial mixed attention, and performs weighted processing.

[0009] S3 spatiotemporal memory reading: Calculate the similarity matrix between the enhanced memory frame image features and the query frame image features as weight information to weight the memory frame image features and obtain fusion features;

[0010] S4 network prediction: The fused features are fed into the classification regression network and decoder to predict the tracking and segmentation results respectively;

[0011] S5 template update: Dynamically filter memory frame templates based on confidence scores and temporal distance constraints.

[0012] MobileNet is used as the backbone network in the feature extraction step Memory frames and query frames use the backbone network structure respectively and However, the two network branches do not share weights; the memory frame template contains T memory frame images, which are input into the backbone network Then it is spliced ​​with the T mask images after mapping, and features are gradually extracted through four stages including depth-wise separable convolution modules, and the final output is the memory frame image feature F with a dimension of 3×1024×20×20 m ; A query frame image is similarly input into the backbone network Get the query frame image feature F with a dimension of 1×1024×20×20 q , and retain the intermediate features F extracted by the first and second stages q4 and F q8 , the dimensions are 1×48×40×40 and 1×128×20×20 respectively.

[0013] The feature enhancement step uses an adaptive gated attention mechanism to enhance the target area of ​​interest in the memory frame image features and the query frame image features. The feature enhancement step is specifically as follows:

[0014] S21: Generate one-hot vector by dynamic gating unit, where the memory frame gated one-hot vector is α mi =[α mi1 ,α mi2 ,α mi3 ] T , the gated one-hot vector of the query frame is α q ;

[0015] S22: Select the attention module according to the one-hot vector, and get the attention mechanism as A s =[A1,A2,A3] T ;

[0016] S23: Use the selected attention mechanism and one-hot vector to assign different weights to the information at each position in the feature, enhance the information of the target-related area in the feature, and finally obtain the memory frame image feature F' after the gated attention mechanism is strengthened. m and query frame image features F' q The calculation formulas are:

[0017] F' mi =(A s T α mi )F mi

[0018] F' m =concat(F' m1 ,F' m2 ,F' m3 )

[0019] F' q =(A s T α q )F q

[0020] Where i=1, 2, 3, and concat() is a function for connecting multiple vectors.

[0021] The specific steps for the dynamic gating unit to generate a one-hot vector are as follows:

[0022] S211: First, a global average pooling layer is used to compress the global information of the feature F with an input dimension of C×H×W into the channel descriptor to obtain a feature vector F with a length of C. k , where C is the number of channels, H and W are height and width;

[0023] S212: Then send it to the first fully connected layer, F k The feature dimension is compressed to C / r, where r is a constant and the calculation formula is:

[0024] F k′ =ω1a(F k )+b1

[0025] Where a is the global average pooling operation, ω1 is the weight matrix, and b1 is the bias vector;

[0026] S213: The output of the first fully connected layer is sent to the ReLU layer, and after nonlinear activation, it is sent to the second fully connected layer to map the number of channels of the feature vector back to the original feature dimension C. The calculation formula is:

[0027] ε k =ω2(σ(ω1a(F k )+b1))+b2

[0028] Where σ is the ReLU activation function, ω2 is the weight matrix, and b2 is the bias vector;

[0029] S214: The gate signal ε is obtained k The GumbelSoftmax function generates the one-hot vector α of the gate unit, which is calculated as follows:

[0030]

[0031] in is the random noise sampled from the Gumbel distribution, τ is the temperature parameter that affects the Gumbel distribution and controls the smoothness of the distribution of the Softmax output.

[0032] In the spatiotemporal memory reading step, the similarity between each pixel of the memory frame image features and the query frame image features obtained in the feature enhancement step is first calculated, and a similarity matrix w is obtained after softmax normalization; the similarity matrix w is used as weight information to weight the memory frame image features to obtain enhanced memory frame image feature information; the query frame image features are used to retrieve target-related information in the memory frame image features, and different importance is assigned to the memory information according to the attention mechanism, and the weighted memory frame image features and the query frame image features are spliced ​​to obtain a shared feature y with a feature dimension of 256.

[0033] In the network prediction step, the network uses a dual-branch structure, and the tracking branch and the segmentation branch share features. First, the shared feature y obtained in the spatiotemporal memory reading step is randomly discarded in the spatial dimension to reduce the risk of overfitting. Then, it passes through three dilated convolution layers with different expansion rate parameters and is connected in a residual manner to achieve multi-scale feature fusion. In the tracking branch, the classification network and the regression network respectively perform convolution operations on the fused feature y to generate the final classification response map R. cls, Centrality Response Chart R ctr and the regression offset R reg , where the center response is also calculated from the classification features; in the segmentation branch, the decoder structure contains several upsampling modules and residual modules, which combine the fused feature y and the feature F obtained in the feature extraction step with feature dimensions of 48 and 128 respectively q4 and F q8 Restore to 1 / 4 resolution and obtain the single-channel mask prediction result.

[0034] The input feature y of the classification regression network of the tracking branch in the network prediction step remains unchanged after multi-scale feature fusion, and the cls feature and reg feature are extracted respectively through the convolution layer, and then the dimensions are reduced to 1 and 4 respectively through the lightweight convolution layer; the final classification response map R cls , Centrality Response Chart R ctr and the regression offset R reg The sizes of are 1×1×H×W, 1×1×H×W, and 1×4×H×W respectively; the calculation of the centrality response and the calculation of the classification response share features, and the two are multiplied together in the inference stage to suppress the high confidence scores of pixels that deviate from the center of the target, ensuring that the model pays more attention to the pixels in the central area.

[0035] The decoder of the segmentation branch in the network prediction step first uses the residual module to perform channel compression operation on the fusion feature map y, maintaining the original resolution but halving the feature dimension, and then uses a convolution kernel of size 3×3 and a filling rate of 1 to obtain y1; y1 is combined with the 1 / 8 resolution feature map F q8 Add element by element to get the feature map y2; then use the bilinear interpolation method to upsample the feature map y2 to restore the resolution to 1 / 4, and add it to the 1 / 4 resolution feature map F after adjusting the number of channels. q4 Element-by-element addition yields y3; the feature map y3 passes through the convolutional layer and is processed by the activation function of the ReLU layer to generate a single-channel segmentation mask; the resolution of the final predicted segmentation mask is 1 / 4 of the original image, ensuring the operating efficiency of the model.

[0036] The template updating step is to dynamically select the memory frame template according to the confidence score and time distance constraint; first, the classification response map R cls , Centrality Response Chart R ctr The dimension 1×1×H×W is reshaped into 1×(H×W)×1, and the sigmoid activation function is used to map its value to between 0 and 1, and then the two are multiplied to obtain the classification confidence score S1; the decoded segmentation mask map and the query frame image feature map are passed through the backbone network again Extract feature F qmAfter that, the mask confidence score S2 in the range of 0 to 1 is obtained through the convolution layer, the fully connected layer and the activation layer; the time distance constraint S m , confidence score S C The final score S of the current query frame is calculated as follows:

[0037] S m =e -|fidx-m|

[0038] S C =S1×S2

[0039] S=S m +S C

[0040] Where fidx is the frame number of the current query frame, and m is the frame number of the memory frame stored in the template information library.

[0041] The template information library is represented by key-value pairs (fidx, S C ) in the form of storing the frame number and confidence score of each frame template; when the confidence score of the query frame is greater than the threshold, the score S of each frame in the template information library is calculated and sorted by size, and the frame template with the lowest score is replaced with the current query frame to form a new template information library.

[0042] Compared with the existing technology, the target tracking method based on the gated attention mechanism and spatiotemporal memory network provided by the present invention has the following beneficial effects:

[0043] (1) An adaptive gated attention module is proposed. By using the gated unit to filter the attention mechanism, the memory frame and query frame information are dynamically weighted, which effectively enhances the target area of ​​interest in the feature, makes full use of the spatiotemporal memory information, strengthens the fusion of the memory frame and query frame image features, and improves the template's adaptability to target changes.

[0044] (2) Compared with the design of multi-branch attention module, the present invention significantly reduces the amount of computation while maintaining the same performance.

[0045] (3) This paper proposes a segmentation branch that introduces shared features into the target tracking task, providing refined pixel-level segmentation results for the bounding box, enabling the model to learn richer feature representations. Furthermore, the confidence scores and temporal distance constraints are used to dynamically filter the memory frame templates, improving the quality of the template information and enhancing the accuracy and robustness of tracking. The tracking effect is improved in complex scenarios such as those where the target's shape changes, the target is occluded for a long time, or the target has a high similarity to interfering objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a schematic diagram of the overall network structure of the target tracking method based on the gated attention mechanism and spatiotemporal memory network of the present invention.

[0048] Figure 2 This is a flowchart of the target tracking method based on the gated attention mechanism and spatiotemporal memory network of the present invention.

[0049] Figure 3 Schematic diagram of the structure of the adaptive gated attention module of the present invention.

[0050] Figure 4 Schematic diagram of the structure of the decoder in the segmentation branch of the present invention.

[0051] Figure 5 This is a flow chart of updating the template information library of the present invention.

[0052] Figure 6 This is a comparison chart of the tracking results of the present invention and other methods. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with Examples and accompanying drawings. It should be understood that the examples are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit and scope of the present invention, variations and advantages that those skilled in the art can imagine are included in the present invention and are protected by the appended claims and their equivalents.

[0054] It should be understood that the term "a" should be understood as "at least one" or "one or more." That is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "a" should not be understood as limiting the quantity. In the present invention, except for names and terms that have been explicitly defined by the inventors, other names and terms are commonly used in the art.

[0055] In one embodiment of the present invention, a target tracking method based on a gated attention mechanism and a spatiotemporal memory network is provided. Figure 1 As shown, the overall network structure of an embodiment of the present invention includes a backbone network, an adaptive gated attention mechanism module, a spatiotemporal memory readout module, a classification regression network and a decoder.

[0056] In another embodiment of the present invention, a target tracking method based on a gated attention mechanism and a spatiotemporal memory network is provided. Figure 2 As shown, the steps of this method are as follows:

[0057] S1 Feature Extraction: The memory frame image and the query frame image are respectively sent to the backbone network to extract the corresponding memory frame image features and query frame image features;

[0058] S2 feature enhancement: The adaptive gated attention mechanism is used to enhance the memory frame image features and the query frame image features. The dynamic gating unit adaptively selects a pair of memory frame image features and query frame image features from the three attention modules: spatial attention, channel attention, and channel-spatial mixed attention, and performs weighted processing.

[0059] S3 spatiotemporal memory reading: Calculate the similarity matrix between the enhanced memory frame image features and the query frame image features as weight information to weight the memory frame image features and obtain fusion features;

[0060] S4 network prediction: The fused features are fed into the classification regression network and decoder to predict the tracking and segmentation results respectively;

[0061] S5 template update: Dynamically filter memory frame templates based on confidence scores and temporal distance constraints.

[0062] MobileNet is used as the backbone network in the feature extraction step Memory frames and query frames use similar backbone network structures and However, the two network branches do not share weights. T memory frame images are input to the backbone network. Then it is spliced ​​with the T mask images after mapping, and features are gradually extracted through four stages including depth-wise separable convolution modules, and the final output is the memory frame image feature F with a dimension of 3×1024×20×20 m A query frame image is similarly input into the backbone network. Get the query frame image feature F with a dimension of 1×1024×20×20 q , and retain the intermediate features F extracted by the first and second stages q4 and F q8 , the dimensions are 1×48×40×40 and 1×128×20×20 respectively.

[0063] The adaptive gated attention mechanism in the feature enhancement step consists of three optional attention modules and a dynamic gating unit. Specifically, Figure 3As shown in Figure 2, the three optional attention modules are spatial attention, channel attention, and channel-spatial mixed attention. A k-dimensional one-hot vector in the gated unit selects one of the multiple attention mechanisms to enhance the image features of the memory frame. The feature enhancement steps are as follows:

[0064] S21: Generate one-hot vector by dynamic gating unit, where the memory frame gated one-hot vector is α mi =[α mi1 ,α mi2 ,α mi3 ] T , the gated one-hot vector of the query frame is α q ;

[0065] S22: Select the attention module according to the one-hot vector, and get the attention mechanism as A s =[A1,A2,A3] T ;

[0066] S23: Use the selected attention mechanism and one-hot vector to assign different weights to the information at each position in the feature, enhance the information of the target-related area in the feature, and finally obtain the memory frame image feature F' after the gated attention mechanism is strengthened. m and query frame image features F' q The calculation formulas are:

[0067] F' mi =(A s T α mi )F mi

[0068] F' m =concat(F' m1 ,F' m2 ,F' m3 )

[0069] F' q =(A s T α q )F q

[0070] Where i=1, 2, 3, and concat() is a function for connecting multiple vectors.

[0071] The specific steps of generating a one-hot vector by the dynamic gating unit are as follows:

[0072] S211: First, a global average pooling layer is used to compress the global information of the feature F with an input dimension of C×H×W into the channel descriptor to obtain a feature vector F with a length of C. k , where C is the number of channels, H and W are height and width;

[0073] S212: Then send it to the first fully connected layer, F k The feature dimension is compressed to C / r, where r is a constant and the calculation formula is:

[0074] F k′ =ω1a(F k )+b1

[0075] Where a is the global average pooling operation, ω1 is the weight matrix, and b1 is the bias vector;

[0076] S213: The output of the first fully connected layer is sent to the ReLU layer, and after nonlinear activation, it is sent to the second fully connected layer to map the number of channels of the feature vector back to the original feature dimension C. The calculation formula is:

[0077] ε k =ω2(σ(ω1a(F k )+b1))+b2

[0078] Where σ is the ReLU activation function, ω2 is the weight matrix, and b2 is the bias vector;

[0079] S214: The gate signal ε is obtained k The GumbelSoftmax function generates the one-hot vector α of the gate unit, which is calculated as follows:

[0080]

[0081] in is the random noise sampled from the Gumbel distribution, τ is the temperature parameter that affects the Gumbel distribution and controls the smoothness of the distribution of the Softmax output.

[0082] The spatiotemporal memory reading step first calculates the pixel-by-pixel similarity between the memory frame image features and the query frame image features, and then performs softmax normalization to obtain a similarity matrix w. This matrix w is then used as weight information to weight the memory frame image features, yielding enhanced memory frame image feature information. The query frame image features are then used to retrieve target-related information from the memory frame image features. Using an attention mechanism, different levels of importance are assigned to the memory information. The weighted memory frame image features are then concatenated with the query frame image features to yield a shared feature y with a feature dimension of 256.

[0083] In the network prediction step, the network uses a dual-branch structure, with tracking and segmentation branches sharing features. First, the shared features y are randomly dropped in the spatial dimension to reduce the risk of overfitting. Then, three dilated convolutional layers with different expansion rate parameters are connected in a residual manner to achieve multi-scale feature fusion. In the tracking branch, the classification network and the regression network respectively perform convolution operations on the fused features y to generate the final classification response map R. cls , Centrality Response Chart R ctr and the regression offset R reg The center response is also calculated from the classification features. In the segmentation branch, the decoder structure contains several upsampling modules and residual modules, which combine the fused feature y with the feature F obtained in the feature extraction step, whose feature dimensions are 48 and 128 respectively. q4 and F q8 Restore to 1 / 4 resolution and obtain the single-channel mask prediction result.

[0084] The input feature y of the classification regression network of the tracking branch in the network prediction step remains unchanged after multi-scale feature fusion, and the cls feature and reg feature are extracted respectively through the convolution layer, and then the dimensions are reduced to 1 and 4 respectively through the lightweight convolution layer. The final classification response map R cls , Centrality Response Chart R ctr and the regression offset R reg The sizes of are 1×1×H×W, 1×1×H×W, and 1×4×H×W, respectively. The calculation of the centrality response and the classification response share features, and the two are multiplied together during the inference phase to suppress the high confidence scores of pixels that are off-center, ensuring that the model pays more attention to pixels in the central area.

[0085] like Figure 4 As shown, in the embodiment of the present invention, the decoder of the segmentation branch in the network prediction step first uses the residual module to perform channel compression operation on the fusion feature map y, maintaining the original resolution but halving the feature dimension, and then uses a convolution kernel of size 3×3 and a filling rate of 1 to perform a convolution operation on it to obtain y1. q8 Add element by element to get the feature map y2. Then use the bilinear interpolation method to upsample the feature map y2 to restore the resolution to 1 / 4, and add it to the 1 / 4 resolution feature map F after adjusting the number of channels. q4 Element-wise addition yields y3. Passing the feature map y3 through a convolutional layer and activation function in a ReLU layer generates a single-channel segmentation mask. The resulting predicted segmentation mask has a resolution of 1 / 4 that of the original image, ensuring efficient model operation.

[0086] like Figure 5As shown, in an embodiment of the present invention, the template updating step is to dynamically select the memory frame template according to the confidence score and the time distance constraint. First, the classification response graph R in the network prediction step is cls , Centrality Response Chart R ctr The dimension 1×1×H×W is reshaped into 1×(H×W)×1, and the sigmoid activation function is used to map its value to between 0 and 1, and then the two are multiplied to obtain the classification confidence score S1. The decoded segmentation mask map and the query frame image feature map are passed through the backbone network again. Extract feature F qm After that, the mask confidence score S2 in the range of 0 to 1 is obtained through the convolution layer, the fully connected layer and the activation layer. m , confidence score S C The final score S of the current query frame is calculated as follows:

[0087] S m =r -|fidx-m|

[0088] S C =S1×S2

[0089] S=S m +S C

[0090] Where fidx is the frame number of the current query frame, and m is the frame number of the memory frame stored in the template information library.

[0091] In the template updating step, the template information library is updated with a key-value pair (fidx, S C ) to store the frame number and confidence score of each template frame. When the confidence score of the query frame is greater than the threshold, the score S of each frame in the template information library is calculated and sorted by size. The template frame with the lowest score is replaced with the current query frame to form a new template information library.

[0092] The training datasets used in the examples of the present invention include OTB-100, GOT-10k, and TrackingNet, all official large-scale object tracking datasets. The model was also tested on the GOT-10k test set, which contains 180 videos covering 84 categories and 32 motion patterns, with no overlap in target categories with the training set. This dataset uses two evaluation metrics: average overlap (AO) and success rate (SR), with success rate thresholds set at 0.5 and 0.75.

[0093] As shown in Table 1, the target tracking method based on the gated attention mechanism and spatiotemporal memory network proposed in an embodiment of the present invention has better performance than the currently advanced target tracking algorithms SiamRPN++, ATOM, SiamFC++, DiMP-50, STMTrack, SiamR-CNN, AutoMatch, TrSiam, etc.

[0094]

[0095]

[0096] like Figure 6 As shown, in the first set of images, the upper half of the target has a high resemblance to the background and is also reflective, so the STMTrack method fails to track the entire target, while the tracking frames of the TrSiam and AutoMatch methods are inaccurate. In the third set of images, the target moves rapidly and is subject to interference from background light, leading the other methods to mistake a motorcycle for part of the target. In the fifth set of images, the target is largely occluded and bears a high resemblance to a penguin. The TrSiam method's tracking frame is often too large to include non-target objects, while the AutoMatch method only recognizes a portion of the target under occlusion. Compared to other tracking methods, the embodiments of the present invention can achieve more stable and accurate results when tracking complex targets, such as those that are occluded, have a high resemblance to the background and other objects, or are deformed due to rapid motion.

[0097] In summary, compared with the prior art, the advantages of the present invention are as follows: The present invention proposes an adaptive gated attention module that dynamically weights memory frame and query frame information by using a gated unit to filter attention mechanisms, effectively enhancing the target region of interest in the features, fully utilizing spatiotemporal memory information, strengthening the fusion of memory frame and query frame image features, and improving the template's adaptability to target changes. Different information can select different attention mechanisms to enhance its key information, maximizing the utilization of spatiotemporal memory information. At the same time, compared with the design of a multi-branch attention module, the computational complexity is significantly reduced while maintaining the same performance. Target tracking and segmentation are two tasks with strong correlations in the visual field. Tracking can provide more accurate target location information for segmentation, and segmentation can provide more detailed instance information for tracking. Introducing a segmentation branch with shared features as an auxiliary branch in the target tracking task provides detailed pixel-level segmentation results for the bounding box, enabling the model to learn richer feature representations and improving the tracking model's ability to distinguish similar instances. Compared with joint learning, segmentation as an auxiliary branch reduces the resolution of the mask, reducing the impact of excessive parameter count on real-time tracking. The memory frame templates are dynamically screened using confidence scores and temporal distance constraints, which improves the quality of template information and enhances the accuracy and robustness of tracking.

[0098] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0099] The above-mentioned specific implementation methods are used to illustrate the present invention and are only preferred embodiments of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit of the present invention and the scope of protection of the claims shall fall within the scope of protection of the present invention.

Claims

1. A target tracking method based on gated attention mechanism and spatiotemporal memory network, characterized in that: The following steps are involved: S1 Feature Extraction: The memory frame image and the query frame image are respectively sent to the backbone network to extract the corresponding memory frame image features and query frame image features; S2 feature enhancement: The adaptive gated attention mechanism is used to enhance the memory frame image features and the query frame image features. The dynamic gating unit adaptively selects a pair of memory frame image features and query frame image features from the three attention modules: spatial attention, channel attention, and channel-spatial mixed attention, and performs weighted processing. S3 spatiotemporal memory reading: Calculate the similarity matrix between the enhanced memory frame image features and the query frame image features as weight information to weight the memory frame image features and obtain fusion features; S4 network prediction: The fused features are fed into the classification regression network and decoder to predict the tracking and segmentation results respectively; S5 template update: Dynamically filter memory frame templates based on confidence scores and time distance constraints; in: The feature enhancement step uses an adaptive gated attention mechanism to enhance the target area of ​​interest in the memory frame image features and the query frame image features. The feature enhancement step is specifically as follows: S21: Generate one-hot vectors by dynamic gating units, where the memory frame gated one-hot vector is , the gated one-hot vector of the query frame is ; S22: Select the attention module according to the one-hot vector, and the attention mechanism is ; S23: Use the selected attention mechanism and one-hot vector to assign different weights to the information at each position in the feature, enhance the information of the target-related area in the feature, and finally enhance the memory frame image features after the gated attention mechanism. and query frame image features The calculation formulas are:

2. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: MobileNet is used as the backbone network in the feature extraction step , the memory frame image and the query frame image use the backbone network structure respectively and , but the weights are not shared between the two network branches; The memory frame template contains T memory frame images, which are input into the backbone network Then it is spliced ​​with the T mask images after mapping, and features are gradually extracted through four stages including depth-separable convolution modules. The final output dimension is Memory frame image feature F m ; A query frame image is input into the backbone network The dimension is The query frame image feature F q , and retain the intermediate features F extracted by the first and second stages q4 and F q8 , the dimensions are and .

3. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: In the spatiotemporal memory reading step, the similarity between each pixel of the memory frame image features and the query frame image features obtained in the feature enhancement step is first calculated, and a similarity matrix w is obtained after softmax normalization; the similarity matrix w is used as weight information to weight the memory frame image features to obtain enhanced memory frame image feature information; the query frame image features are used to retrieve target-related information in the memory frame image features, and different importance is assigned to the memory information according to the attention mechanism, and the weighted memory frame image features and the query frame image features are spliced ​​to obtain a shared feature y with a feature dimension of 256.

4. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: In the network prediction step, the network uses a dual-branch structure, and the tracking branch and the segmentation branch share features. First, the shared feature y obtained in the spatiotemporal memory reading step is randomly discarded in the spatial dimension to reduce the risk of overfitting. Then, it passes through three dilated convolution layers with different expansion rate parameters and is connected in a residual manner to achieve multi-scale feature fusion. In the tracking branch, the classification network and the regression network respectively perform convolution operations on the fused feature y to generate the final classification response map R. cls , Centrality Response Chart R ctr and the regression offset R reg , where the center response is also calculated from the classification features; in the segmentation branch, the decoder structure contains several upsampling modules and residual modules, which combine the fused feature y and the feature F obtained in the feature extraction step with feature dimensions of 48 and 128 respectively q4 and F q8 Restore to 1 / 4 resolution and obtain the single-channel mask prediction result.

5. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The input feature y of the classification regression network of the tracking branch in the network prediction step remains unchanged after multi-scale feature fusion, and the cls feature and reg feature are extracted respectively through the convolution layer, and then the dimensions are reduced to 1 and 4 respectively through the lightweight convolution layer; the final classification response map R cls , Centrality Response Chart R ctr and the regression offset R reg The sizes are 、 and The calculation of the centrality response and the calculation of the classification response share features, and the two are multiplied together during the inference phase to suppress the high confidence scores of pixels that deviate from the center of the target, ensuring that the model pays more attention to pixels in the central area.

6. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The decoder of the split branch in the network prediction step first uses the residual module to perform channel compression operation on the fused feature map y, maintaining the original resolution but halving the feature dimension, and then uses the size of , convolution operation is performed on it with a convolution kernel with a filling rate of 1 to obtain y1; y1 is combined with the 1 / 8 resolution feature map F q8 Add element by element to get the feature map y2; then use the bilinear interpolation method to upsample the feature map y2 to restore the resolution to 1 / 4, and add it to the 1 / 4 resolution feature map F after adjusting the number of channels. q4 Element-by-element addition yields y3; the feature map y3 passes through the convolutional layer and is processed by the activation function of the ReLU layer to generate a single-channel segmentation mask; the resolution of the final predicted segmentation mask is 1 / 4 of the original image, ensuring the operating efficiency of the model.

7. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The template updating step is to dynamically select the memory frame template according to the confidence score and time distance constraint; first, the classification response map R cls , Centrality Response Chart R ctr By Dimension reshape , and use the sigmoid activation function to map its value to between 0 and 1, and then multiply the two to get the classification confidence score S1; the decoded segmentation mask map and the query frame image feature map are passed through the backbone network again Extract feature F qm After that, the mask confidence score S2 in the range of 0~1 is obtained through the convolution layer, the fully connected layer and the activation layer; the time distance constraint , confidence score The final score S of the current query frame is calculated as follows:

Images