Target tracking method based on gated attention mechanism and space-time memory network

Abstract

The invention relates to the technical field of target tracking in computer vision, and particularly discloses a target tracking method based on a gated attention mechanism and a space-time memory network, which mainly comprises a feature extraction module, a gated attention mechanism feature fusion module, a space-time memory feature reading module and a double-branch prediction module. The method comprises the following steps: inputting a memory frame image and a current query frame image into a backbone network to extract features; fusing the enhanced memory frame image features and the query frame image features by adopting a gating attention mechanism; respectively predicting a target position frame and a segmentation mask result by using a double-branch detection module; and dynamically screening the memory frame template according to the confidence score and the time distance constraint. According to the method, fusion of memory frame image features and query frame image features is enhanced through a gating attention mechanism, space-time memory information is fully utilized, interference caused by conditions that a target is shielded for a long time and non-target similarity is high is effectively reduced, and accuracy and robustness of a tracking model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a target tracking method based on a gated attention mechanism and a spatiotemporal memory network. Background Art

[0002] Vision Object Tracking is one of the important research directions in the field of computer vision. Its task is to mark the position and state of the target to be tracked in the initial frame or a certain frame of the video sequence, so that the tracker can automatically track the target in all subsequent video sequences. Vision tracking technology has a wide range of applications in many aspects such as autonomous driving, video surveillance, motion analysis, medical diagnosis, etc.

[0003] The research methods of visual target tracking mainly include early classic algorithms such as optical flow method, algorithms based on correlation filters and current mainstream deep learning methods. Among them, deep learning methods are divided into two mainstream frameworks: the method based on the twin network transforms the tracking task into a similarity matching problem between images; the method based on the Transformer structure introduces the self-attention mechanism to strengthen the target area of ​​interest in a dynamic weighted manner.

[0004] Most tracking methods based on twin networks use the initial frame image or the previous frame image as the template image, which has poor adaptability to complex situations such as target shape, changes in lighting conditions, and occlusion of interfering objects; and the continuous increase in the number of templates will consume a lot of computing resources, affecting the real-time performance of tracking. The Transformer structure can capture long-distance dependencies and global context information, and can better adapt to changes in targets and scenes, but its training complexity is high and it is easily affected by input data noise. Spatiotemporal memory information refers to the information about the timing and position of the target stored in the model. Introducing spatiotemporal memory information into the network can improve the performance of target tracking. The existing tracking methods based on spatiotemporal memory networks store multi-frame historical information of the target by introducing a memory mechanism, perform feature matching globally, and the template can be dynamically updated, which improves the adaptability of the tracker to the target to a certain extent. However, the spatiotemporal memory information between memory frames is still not fully utilized. In the case of interference such as long-term large-area occlusion of the target and high non-target similarity, it is easy to lose the target, the template is contaminated, resulting in target frame switching, inaccuracy, etc. Summary of the invention

[0005] In view of the shortcomings of existing tracking technology, the present invention provides a target tracking method based on a gated attention mechanism and a spatiotemporal memory network, which can make full use of the spatiotemporal memory information, focus on the target area of ​​interest in the memory information, and effectively improve the performance of the tracker in complex scenarios such as target shape changes, long-term occlusion, and high similarity with interfering objects.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows: a target tracking method based on a gated attention mechanism and a spatiotemporal memory network, the method steps are as follows:

[0007] S1 feature extraction: the memory frame image and the query frame image are sent to the backbone network to extract the corresponding memory frame image features and query frame image features respectively;

[0008] S2 feature enhancement: The memory frame image features and query frame image features are enhanced using the adaptive gated attention mechanism. The dynamic gating unit adaptively selects a pair of memory frame image features and query frame image features from the three attention modules: spatial attention, channel attention, and channel-spatial mixed attention, and performs weighted processing.

[0009] S3 spatiotemporal memory reading: Calculate the similarity matrix between the enhanced memory frame image features and the query frame image features as weight information to weight the memory frame image features and obtain fusion features;

[0010] S4 network prediction: The fused features are sent to the classification regression network and decoder to predict the tracking and segmentation results respectively;

[0011] S5 template update: Dynamically filter memory frame templates based on confidence scores and temporal distance constraints.

[0012] MobileNet is used as the backbone network in the feature extraction step The memory frame and query frame use the backbone network structure respectively. and However, the two network branches do not share weights; the memory frame template contains T memory frame images, which are input into the backbone network After that, it is concatenated with the T mask images after the mapping process, and features are gradually extracted through four stages including the depthwise separable convolution module, and finally the memory frame image feature F with a dimension of 3×1024×20×20 is output. m ; A query frame image is similarly input into the backbone network Get the query frame image feature F with a dimension of 1×1024×20×20 q , and retain the intermediate features F extracted by the first and second stages q4 and F q8 , the dimensions are 1×48×40×40 and 1×128×20×20 respectively.

[0013] The feature enhancement step uses an adaptive gated attention mechanism to enhance the target area of ​​interest in the memory frame image features and the query frame image features. The feature enhancement step is specifically as follows:

[0014] S21: Generate a one-hot vector by the dynamic gating unit, where the memory frame gated one-hot vector is α mi =[α mi1 ,α mi2 ,α mi3 ] T , the gated one-hot vector of the query frame is α q ;

[0015] S22: Select the attention module according to the one-hot vector, and get the attention mechanism as A s =[A1,A2,A3] T ;

[0016] S23: Use the selected attention mechanism and one-hot vector to assign different weights to the information at each position in the feature, enhance the information in the target-related area in the feature, and finally obtain the memory frame image feature F' after being enhanced by the gated attention mechanism. m and query frame image features F' q The calculation formulas are:

[0017] F' mi =(A s T α mi )F mi

[0018] F' m =concat(F' m1 ,F' m2 ,F' m3 )

[0019] F' q =(A s T α q )F q

[0020] Where i=1,2,3, and concat() is a function for connecting multiple vectors.

[0021] The specific steps of generating a one-hot vector by the dynamic gating unit are as follows:

[0022] S211: First, a global average pooling layer is used to compress the global information of the feature F with an input dimension of C×H×W into the channel descriptor to obtain a feature vector F with a length of C. k , where C is the number of channels, H and W are the height and width;

[0023] S212: Then send it to the first fully connected layer, F k The feature dimension is compressed to C / r, where r is a constant and the calculation formula is:

[0024] F k′ =ω1a(F k )+b1

[0025] Where a is the global average pooling operation, ω1 is the weight matrix, and b1 is the bias vector;

[0026] S213: The output of the first fully connected layer is sent to the ReLU layer, and after nonlinear activation, it is sent to the second fully connected layer to map the number of channels of the feature vector back to the original feature dimension C. The calculation formula is:

[0027] ε k =ω2(σ(ω1a(F k )+b1))+b2

[0028] Where σ is the ReLU activation function, ω2 is the weight matrix, and b2 is the bias vector;

[0029] S214: The gate signal ε is obtained by k The GumbelSoftmax function generates the one-hot vector α of the gated unit, which is calculated as:

[0030]

[0031] in is the random noise sampled from the Gumbel distribution, τ is the temperature parameter that affects the Gumbel distribution and controls the smoothness of the distribution of the Softmax output.

[0032] In the spatiotemporal memory reading step, the similarity between each pixel of the memory frame image feature and the query frame image feature obtained in the feature enhancement step is first calculated, and the similarity matrix w is obtained through softmax normalization; the memory frame image feature is weighted by using the similarity matrix w as weight information to obtain enhanced memory frame image feature information; the query frame image feature is used to retrieve the target-related information in the memory frame image feature, different importance is assigned to the memory information according to the attention mechanism, and the weighted memory frame image feature is concatenated with the query frame image feature to obtain a shared feature y with a feature dimension of 256.

[0033] In the network prediction step, the network uses a dual-branch structure, and the tracking branch and the segmentation branch share features. First, the shared feature y obtained in the spatiotemporal memory reading step is randomly discarded in the spatial dimension to reduce the risk of overfitting, and then three dilated convolution layers with different expansion rate parameters are connected in a residual manner to achieve multi-scale feature fusion. In the tracking branch, the classification network and the regression network respectively perform convolution operations on the fused feature y to generate the final classification response map R. cls, Centrality response map R ctr and the regression offset R reg , where the centrality response is also calculated from the classification features; in the segmentation branch, the decoder structure includes several upsampling modules and residual modules, which combine the fused feature y and the feature F obtained in the feature extraction step with feature dimensions of 48 and 128 respectively q4 and F q8 Restore to 1 / 4 resolution and get the single-channel mask prediction result.

[0034] The input feature y of the classification regression network of the tracking branch in the network prediction step remains unchanged after multi-scale feature fusion, and the cls feature and reg feature are extracted respectively through the convolution layer, and then the dimensions are reduced to 1 and 4 respectively through the lightweight convolution layer; the final classification response map R cls , Centrality response map R ctr and the regression offset R reg The sizes of are 1×1×H×W, 1×1×H×W, and 1×4×H×W respectively; the calculation of the centrality response and the calculation of the classification response share features, and the two are multiplied together in the inference stage to suppress the high confidence scores of pixels that deviate from the center of the target, ensuring that the model pays more attention to the pixels in the central area.

[0035] The decoder of the segmentation branch in the network prediction step first uses the residual module to perform channel compression operation on the fused feature map y, maintaining the original resolution but halving the feature dimension, and then uses a convolution kernel of size 3×3 and a filling rate of 1 to obtain y1; y1 is combined with the 1 / 8 resolution feature map F q8 Add element by element to get the feature map y2; then use the bilinear interpolation method to upsample the feature map y2 to restore the resolution to 1 / 4, and add it to the 1 / 4 resolution feature map F after adjusting the number of channels. q4 Add element by element to get y3; pass the feature map y3 through the convolution layer and after being processed by the activation function of the ReLU layer, a single-channel segmentation mask is generated; the resolution of the final predicted segmentation mask map is 1 / 4 of the original image, which ensures the operation efficiency of the model.

[0036] The template updating step is to dynamically select the memory frame template according to the confidence score and the time distance constraint; first, the classification response map R cls , Centrality response map R ctr The dimension 1×1×H×W is reshaped into 1×(H×W)×1, and the sigmoid activation function is used to map its value to between 0 and 1, and then the two are multiplied to obtain the classification confidence score S1; the decoded segmentation mask map and the query frame image feature map are passed through the backbone network again Extract feature F qmAfter that, the mask confidence score S2 in the range of 0 to 1 is obtained through the convolution layer, the fully connected layer and the activation layer; the time distance constraint S m , confidence score S C The final score S of the current query frame is calculated as follows:

[0037] S m =e -|fidx-m|

[0038] S C =S1×S2

[0039] S=S m +S C

[0040] Where fidx is the frame number of the current query frame, and m is the frame number of the memory frame stored in the template information library.

[0041] The template information library is represented by key-value pairs (fidx, S C ) in the form of storing the frame number and confidence score of each frame template; when the confidence score of the query frame is greater than the threshold, the score S of each frame in the template information library is calculated and sorted by size, and the frame template with the lowest score is replaced with the current query frame to form a new template information library.

[0042] Compared with the prior art, the target tracking method based on the gated attention mechanism and spatiotemporal memory network provided by the present invention has the following beneficial effects:

[0043] (1) An adaptive gated attention module is proposed. By using the gated unit to filter the attention mechanism, the memory frame and query frame information are dynamically weighted, which effectively enhances the target area of ​​interest in the feature, makes full use of the spatiotemporal memory information, strengthens the fusion of the memory frame and query frame image features, and improves the template's ability to adapt to target changes.

[0044] (2) Compared with the design of multi-branch attention modules, the present invention significantly reduces the amount of computation while maintaining the same performance.

[0045] (3) This paper proposes to introduce a segmentation branch of shared features in the target tracking task, providing fine pixel-level segmentation results for the bounding box, enabling the model to learn richer feature representations, and dynamically screening memory frame templates using confidence scores and time distance constraints, thereby improving the quality of template information and the accuracy and robustness of tracking. The tracking effect is improved in complex scenarios such as target shape changes, long-term occlusion, and high similarity with interfering objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a schematic diagram of the overall network structure of the target tracking method based on the gated attention mechanism and spatiotemporal memory network of the present invention.

[0048] Figure 2 The present invention is a flow chart of a target tracking method based on a gated attention mechanism and a spatiotemporal memory network.

[0049] Figure 3 Schematic diagram of the structure of the adaptive gated attention module of the present invention.

[0050] Figure 4 It is a schematic diagram of the structure of the decoder in the segmentation branch of the present invention.

[0051] Figure 5 A flow chart of updating the template information library of the present invention.

[0052] Figure 6 This is a comparison chart of the tracking results of the present invention and other methods. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with examples and accompanying drawings. It should be understood that the examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit and scope of the present invention, variations and advantages that those skilled in the art can expect are all included in the present invention, and are protected by the appended claims and equivalents.

[0054] It is understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number. In the present invention, except for the names and terms that have been clearly defined by the inventor, other names and terms are common names in the art.

[0055] In one embodiment of the present invention, a target tracking method based on a gated attention mechanism and a spatiotemporal memory network is provided. Figure 1 As shown, the overall network structure of an embodiment of the present invention includes a backbone network, an adaptive gated attention mechanism module, a spatiotemporal memory readout module, a classification regression network and a decoder.

[0056] In another embodiment of the present invention, a target tracking method based on a gated attention mechanism and a spatiotemporal memory network is provided. Figure 2 As shown, the steps of this method are as follows:

[0057] S1 feature extraction: the memory frame image and the query frame image are sent to the backbone network to extract the corresponding memory frame image features and query frame image features respectively;

[0058] S2 feature enhancement: The memory frame image features and query frame image features are enhanced using the adaptive gated attention mechanism. The dynamic gating unit adaptively selects a pair of memory frame image features and query frame image features from the three attention modules: spatial attention, channel attention, and channel-spatial mixed attention, and performs weighted processing.

[0059] S3 spatiotemporal memory reading: Calculate the similarity matrix between the enhanced memory frame image features and the query frame image features as weight information to weight the memory frame image features and obtain fusion features;

[0060] S4 network prediction: The fused features are sent to the classification regression network and decoder to predict the tracking and segmentation results respectively;

[0061] S5 template update: Dynamically filter memory frame templates based on confidence scores and temporal distance constraints.

[0062] MobileNet is used as the backbone network in the feature extraction step The memory frame and query frame use a similar backbone network structure and However, the two network branches do not share weights. T memory frame images are input to the backbone network After that, it is concatenated with the T mask images after the mapping process, and features are gradually extracted through four stages including the depthwise separable convolution module, and finally the memory frame image feature F with a dimension of 3×1024×20×20 is output. m A query frame image is similarly input into the backbone network. Get the query frame image feature F with a dimension of 1×1024×20×20 q , and retain the intermediate features F extracted by the first and second stages q4 and F q8 , the dimensions are 1×48×40×40 and 1×128×20×20 respectively.

[0063] The adaptive gated attention mechanism in the feature enhancement step consists of three optional attention modules and a dynamic gating unit. Specifically, Figure 3As shown in the figure, the three optional attention modules are spatial attention, channel attention, and channel-spatial mixed attention. A k-dimensional one-hot vector in the gated unit selects one of the multiple attention mechanisms to enhance the image features of the memory frame. The feature enhancement steps are as follows:

[0064] S21: Generate a one-hot vector by the dynamic gating unit, where the memory frame gated one-hot vector is α mi =[α mi1 ,α mi2 ,α mi3 ] T , the gated one-hot vector of the query frame is α q ;

[0065] S22: Select the attention module according to the one-hot vector, and get the attention mechanism as A s =[A1,A2,A3] T ;

[0066] S23: Use the selected attention mechanism and one-hot vector to assign different weights to the information at each position in the feature, enhance the information in the feature related to the target area, and finally enhance the memory frame image feature F' after the gated attention mechanism. m and query frame image features F' q The calculation formulas are:

[0067] F' mi =(A s T α mi )F mi

[0068] F' m =concat(F' m1 ,F' m2 ,F' m3 )

[0069] F' q =(A s T α q )F q

[0070] Where i=1,2,3, and concat() is a function for connecting multiple vectors.

[0071] The specific steps of generating a one-hot vector by the dynamic gating unit are as follows:

[0072] S211: First, a global average pooling layer is used to compress the global information of the feature F with an input dimension of C×H×W into the channel descriptor to obtain a feature vector F with a length of C. k , where C is the number of channels, H and W are the height and width;

[0073] S212: Then send it to the first fully connected layer, F k The feature dimension is compressed to C / r, where r is a constant and the calculation formula is:

[0074] F k′ =ω1a(F k )+b1

[0075] Where a is the global average pooling operation, ω1 is the weight matrix, and b1 is the bias vector;

[0076] S213: The output of the first fully connected layer is sent to the ReLU layer, and after nonlinear activation, it is sent to the second fully connected layer to map the number of channels of the feature vector back to the original feature dimension C. The calculation formula is:

[0077] ε k =ω2(σ(ω1a(F k )+b1))+b2

[0078] Where σ is the ReLU activation function, ω2 is the weight matrix, and b2 is the bias vector;

[0079] S214: The gate signal ε is obtained by k The GumbelSoftmax function generates the one-hot vector α of the gated unit, which is calculated as:

[0080]

[0081] in is the random noise sampled from the Gumbel distribution, τ is the temperature parameter that affects the Gumbel distribution and controls the smoothness of the distribution of the Softmax output.

[0082] In the spatiotemporal memory reading step, the similarity between each pixel of the memory frame image feature and the query frame image feature is first calculated, and the similarity matrix w is obtained after softmax normalization. The similarity matrix w is used as weight information to weight the memory frame image feature to obtain enhanced memory frame image feature information. The query frame image feature is used to retrieve the target-related information in the memory frame image feature, and different importance is assigned to the memory information according to the attention mechanism, and the weighted memory frame image feature and the query frame image feature are spliced ​​to obtain a shared feature y with a feature dimension of 256.

[0083] In the network prediction step, the network uses a dual-branch structure, and the tracking branch and the segmentation branch share features. First, the shared feature y is randomly discarded in the spatial dimension to reduce the risk of overfitting, and then it passes through three dilated convolution layers with different expansion rate parameters and is connected in a residual manner to achieve multi-scale feature fusion. In the tracking branch, the classification network and the regression network respectively perform convolution operations on the fused feature y to generate the final classification response map R cls , Centrality response map R ctr and the regression offset R reg The center response is also calculated from the classification features. In the segmentation branch, the decoder structure includes several upsampling modules and residual modules. The fused feature y and the feature F obtained in the feature extraction step are 48 and 128 in dimension respectively. q4 and F q8 Restore to 1 / 4 resolution and get the single-channel mask prediction result.

[0084] The input feature y of the classification regression network of the tracking branch in the network prediction step remains unchanged after multi-scale feature fusion, and the cls feature and reg feature are extracted respectively through the convolution layer, and then the dimensions are reduced to 1 and 4 respectively through the lightweight convolution layer. The final classification response map R cls , Centrality response map R ctr and the regression offset R reg The sizes of are 1×1×H×W, 1×1×H×W, and 1×4×H×W respectively. The calculation of the centrality response and the calculation of the classification response share features, and the two are multiplied together in the inference stage to suppress the high confidence scores of pixels that deviate from the center of the target, ensuring that the model pays more attention to the pixels in the central area.

[0085] like Figure 4 As shown, in the embodiment of the present invention, the decoder of the segmentation branch in the network prediction step first uses the residual module to perform channel compression operation on the fused feature map y, maintains the original resolution but halves the feature dimension, and then uses a convolution kernel of size 3×3 and a filling rate of 1 to perform a convolution operation on it to obtain y1. q8 The feature map y2 is obtained by adding each element. Then the bilinear interpolation method is used to upsample the feature map y2 to restore its resolution to 1 / 4, and then the feature map F with 1 / 4 resolution after adjusting the number of channels is added. q4 Add element by element to get y3. Pass the feature map y3 through the convolution layer and the activation function of the ReLU layer to generate a single-channel segmentation mask. The resolution of the final predicted segmentation mask is 1 / 4 of the original image, ensuring the running efficiency of the model.

[0086] like Figure 5As shown, in an embodiment of the present invention, the template updating step is to dynamically select the memory frame template according to the confidence score and the time distance constraint. First, the classification response graph R in the network prediction step is cls , Centrality response map R ctr The dimension 1×1×H×W is reshaped into 1×(H×W)×1, and the sigmoid activation function is used to map its values ​​to between 0 and 1, and then the two are multiplied to obtain the classification confidence score S1. The decoded segmentation mask map and the query frame image feature map are passed through the backbone network again Extract feature F qm After that, the mask confidence score S2 in the range of 0 to 1 is obtained through the convolution layer, the fully connected layer and the activation layer. m , confidence score S C The final score S of the current query frame is calculated as follows:

[0087] S m =r -|fidx-m|

[0088] S C =S1×S2

[0089] S=S m +S C

[0090] Where fidx is the frame number of the current query frame, and m is the frame number of the memory frame stored in the template information library.

[0091] In the template updating step, the template information library is updated with key-value pairs (fidx, S C ) is used to store the frame number and confidence score of each frame template. When the confidence score of the query frame is greater than the threshold, the score S of each frame in the template information library is calculated and sorted by size. The frame template with the lowest score is replaced with the current query frame to form a new template information library.

[0092] The data sets used for training in the embodiments of the present invention include OTB-100, GOT-10k and TrackingNet, which are all official large-scale target tracking data sets, and the model is experimented on the GOT-10k test set, which has 180 videos, including 84 categories, 32 motion modes, and does not overlap with the target categories of the training set. The data set uses two evaluation indicators, the average overlap AO and the success rate SR, where the thresholds selected for the success rate are 0.5 and 0.75.

[0093] As shown in Table 1, the target tracking method based on the gated attention mechanism and the spatiotemporal memory network proposed in the embodiment of the present invention has better performance than the current advanced target tracking algorithms SiamRPN++, ATOM, SiamFC++, DiMP-50, STMTrack, SiamR-CNN, AutoMatch, TrSiam, etc.

[0094]

[0095]

[0096] like Figure 6 As shown, in the first group of pictures, the upper body of the target has a high similarity with the background and there are reflective reflections. The STMTrack method fails to track the entire target, and the tracking frames of the TrSiam and AutoMatch methods are not accurate enough; in the third group of pictures, the target moves quickly and is interfered by background light, and the other methods mistake the motorcycle for part of the target; in the fifth group of pictures, the target is blocked by a large area and has a high similarity to penguins. The tracking frame of the TrSiam method is often too large to include non-target objects, and the AutoMatch method only recognizes a part of the target under occlusion. The embodiments of the present invention can obtain more stable and accurate results compared to other tracking methods when the target is tracked in complex situations such as being blocked, having a high similarity with the background and other objects, and being deformed due to rapid movement.

[0097] In summary, compared with the prior art, the advantages of the present invention are: the present invention proposes an adaptive gated attention module, which dynamically weights the memory frame and query frame information by using the gated unit to screen the attention mechanism, effectively enhances the target area of ​​interest in the feature, makes full use of the spatiotemporal memory information, strengthens the fusion of the memory frame and query frame image features, and improves the template's adaptability to target changes. Different information can choose different attention mechanisms to enhance its key information, thereby maximizing the utilization of spatiotemporal memory information. At the same time, compared with the design of the multi-branch attention module, the amount of calculation is greatly reduced under the same performance. Target tracking and segmentation are two tasks with strong correlation in the visual field. Tracking can provide more accurate location information of the target for segmentation, and segmentation can provide more detailed instance information for tracking. In the target tracking task, the segmentation branch of the shared feature is introduced as an auxiliary to provide a fine pixel-level segmentation result for the bounding box, so that the model learns a richer feature representation and improves the tracking model's ability to distinguish similar instances. Compared with joint learning, segmentation as an auxiliary branch reduces the resolution of the mask and reduces the impact of excessive parameters on real-time tracking. The confidence scores and temporal distance constraints are used to dynamically filter the memory frame templates, which improves the quality of template information and enhances the tracking accuracy and robustness.

[0098] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0099] The above-mentioned specific implementation methods are used to explain the present invention and are only preferred embodiments of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A target tracking method based on gated attention mechanism and spatiotemporal memory network, characterized in that: The following steps are involved: S1 feature extraction: the memory frame image and the query frame image are sent to the backbone network to extract the corresponding memory frame image features and query frame image features respectively; S2 feature enhancement: The memory frame image features and query frame image features are enhanced using the adaptive gated attention mechanism. The dynamic gating unit adaptively selects a pair of memory frame image features and query frame image features from the three attention modules: spatial attention, channel attention, and channel-spatial mixed attention, and performs weighted processing. S3 spatiotemporal memory reading: Calculate the similarity matrix between the enhanced memory frame image features and the query frame image features as weight information to weight the memory frame image features and obtain fusion features; S4 network prediction: The fused features are sent to the classification regression network and decoder to predict the tracking and segmentation results respectively; S5 template update: Dynamically filter memory frame templates based on confidence scores and temporal distance constraints.

2. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: MobileNet is used as the backbone network in the feature extraction step The memory frame image and query frame image use the backbone network structure respectively and However, the two network branches do not share weights; the memory frame template contains T memory frame images, which are input into the backbone network After that, it is concatenated with the T mask images after the mapping process, and features are gradually extracted through four stages including the depthwise separable convolution module, and finally the memory frame image feature F with a dimension of 3×1024×20×20 is output. m ; A query frame image is input into the backbone network Get the query frame image feature F with a dimension of 1×1024×20×20 q , and retain the intermediate features F extracted by the first and second stages q4 and F q8 , the dimensions are 1×48×40×40 and 1×128×20×20 respectively.

3. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The feature enhancement step uses an adaptive gated attention mechanism to enhance the target area of ​​interest in the memory frame image features and the query frame image features. The feature enhancement step is specifically as follows: S21: Generate a one-hot vector by the dynamic gating unit, where the memory frame gated one-hot vector is α mi =[α mi1 ,α mi2 ,α mi3 ] T , the gated one-hot vector of the query frame is α q ; S22: Select the attention module according to the one-hot vector, and get the attention mechanism as A s =[A1,A2,A3] T ; S23: Use the selected attention mechanism and one-hot vector to assign different weights to the information at each position in the feature, enhance the information in the target-related area in the feature, and finally obtain the memory frame image feature F' after being enhanced by the gated attention mechanism. m and query frame image features F' q The calculation formulas are: F' mi =(A s T a mi )F mi F’ m =concat(F’ m1 ,F’ m2 ,F’ m3 ) F' q =(A s T a q )F q Where i=1,2,3, and concat() is a function for connecting multiple vectors.

4. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 3, characterized in that: The specific steps of generating a one-hot vector by the dynamic gating unit are as follows: S211: First, a global average pooling layer is used to compress the global information of the image feature F with an input dimension of C×H×W into the channel descriptor to obtain a feature vector F with a length of C. k , where C is the number of channels, H and W are the height and width; S212: Then send it to the first fully connected layer, F k The feature dimension is compressed to C / r, where r is a constant and the calculation formula is: F k′ =ω1a(F k )+b1 Where a is the global average pooling operation, ω1 is the weight matrix, and b1 is the bias vector; S213: The output of the first fully connected layer is sent to the ReLU layer, and after nonlinear activation, it is sent to the second fully connected layer to map the number of channels of the feature vector back to the original feature dimension C. The calculation formula is: e k =ω2(σ(ω1a(F k )+b1))+b2 Where σ is the ReLU activation function, ω2 is the weight matrix, and b2 is the bias vector; S214: The gate signal ε is obtained by k The GumbelSoftmax function generates the one-hot vector α of the gated unit, which is calculated as: in is the random noise sampled from the Gumbel distribution, τ is the temperature parameter that affects the Gumbel distribution and controls the smoothness of the distribution of the Softmax output.

5. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: In the spatiotemporal memory reading step, the similarity between each pixel of the memory frame image feature and the query frame image feature obtained in the feature enhancement step is first calculated, and the similarity matrix w is obtained through softmax normalization; the memory frame image feature is weighted by using the similarity matrix w as weight information to obtain enhanced memory frame image feature information; the query frame image feature is used to retrieve the target-related information in the memory frame image feature, different importance is assigned to the memory information according to the attention mechanism, and the weighted memory frame image feature is concatenated with the query frame image feature to obtain a shared feature y with a feature dimension of 256.

6. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: In the network prediction step, the network uses a dual-branch structure, and the tracking branch and the segmentation branch share features. First, the shared feature y obtained in the spatiotemporal memory reading step is randomly discarded in the spatial dimension to reduce the risk of overfitting, and then three dilated convolution layers with different expansion rate parameters are connected in a residual manner to achieve multi-scale feature fusion. In the tracking branch, the classification network and the regression network respectively perform convolution operations on the fused feature y to generate the final classification response map R. cls , Centrality response map R ctr and the regression offset R reg , where the centrality response is also calculated from the classification features; in the segmentation branch, the decoder structure includes several upsampling modules and residual modules, which combine the fused feature y and the feature F obtained in the feature extraction step with feature dimensions of 48 and 128 respectively q4 and F q8 Restore to 1 / 4 resolution and get the single-channel mask prediction result.

7. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The input feature y of the classification regression network of the tracking branch in the network prediction step remains unchanged after multi-scale feature fusion, and the cls feature and reg feature are extracted respectively through the convolution layer, and then the dimensions are reduced to 1 and 4 respectively through the lightweight convolution layer; the final classification response map R cls , Centrality response map R ctr and the regression offset R reg The sizes of are 1×1×H×W, 1×1×H×W, and 1×4×H×W respectively; the calculation of the centrality response and the calculation of the classification response share features, and the two are multiplied together in the inference stage to suppress the high confidence scores of pixels that deviate from the center of the target, ensuring that the model pays more attention to the pixels in the central area.

8. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The decoder of the segmentation branch in the network prediction step first uses the residual module to perform channel compression operation on the fused feature map y, maintaining the original resolution but halving the feature dimension, and then uses a convolution kernel of size 3×3 and a filling rate of 1 to obtain y1; y1 is combined with the 1 / 8 resolution feature map F q8 Add element by element to get the feature map y2; then use the bilinear interpolation method to upsample the feature map y2 to restore the resolution to 1 / 4, and add it to the 1 / 4 resolution feature map F after adjusting the number of channels. q4 Add element by element to get y3; pass the feature map y3 through the convolution layer and after being processed by the activation function of the ReLU layer, a single-channel segmentation mask is generated; the resolution of the final predicted segmentation mask map is 1 / 4 of the original image, which ensures the operation efficiency of the model.

9. The target tracking method based on gated attention mechanism and spatiotemporal memory network according to claim 1, characterized in that: The template updating step is to dynamically select the memory frame template according to the confidence score and the time distance constraint; first, the classification response map R cls , Centrality response map R ctr The dimension 1×1×H×W is reshaped into 1×(H×W)×1, and the sigmoid activation function is used to map its values ​​to between 0 and 1, and then the two are multiplied to obtain the classification confidence score S1; the decoded segmentation mask map and the query frame image feature map are passed through the backbone network again Extract feature F qm After that, the mask confidence score S2 in the range of 0 to 1 is obtained through the convolution layer, the fully connected layer and the activation layer; the time distance constraint S m , confidence score S C The final score S of the current query frame is calculated as follows: S m ＝r -|fidx-m| S C =S1×S2 S=S m +S C Where fidx is the frame number of the current query frame, m is the frame number of the memory frame stored in the template information library, The template information library is represented by key-value pairs (fidx, SC ) The frame number and confidence score of each frame template are stored in the form of; When the confidence score of the query frame is greater than the threshold, the score S of each frame in the template information library is calculated and sorted by size. The frame template with the lowest score is replaced with the current query frame to form a new template information library.

Images