Target tracking method based on long short-term context decoupling and natural language query

Abstract

A target tracking method based on long-short context decoupling and natural language query, the method comprises the following steps: step one, using a ResNet network pre-trained on an ImageNet dataset and an encoder with a Transfomer structure to extract video frame-level features; step two, splicing the video frame-level features to obtain video-level features; step three, using a text encoder to obtain text query features; step four, capturing dynamic context information through a short-term context matching branch; step five, capturing static context information through a long-term context perception branch; step six, adaptively regulating the required long-term or short-term context information through a long-short context modulation module; step seven, estimating the position of the target through a prediction module; and step eight, model optimization. Compared with existing natural language query-based target tracking methods, the accuracy of target positioning of the present application is significantly improved, and good tracking robustness is exhibited.

Description

Technical Field

[0001] This invention relates to a target tracking method, specifically a target tracking method based on long and short time context decoupling and natural language query. Background Technology

[0002] The main challenge of natural language query-based target tracking methods lies in predicting target displacement changes by fusing two heterogeneous pieces of information. One is the static description of the main video features contained in the text query, i.e., long-term context; the other is an image patch containing the target and its surrounding environment, obtained by cropping from the current video frame, i.e., the search region. Currently, most methods simply fuse these two without considering the rationality of the fusion method. This is because the textual information in the natural language query and the visual information in the search region may sometimes be inconsistent. In such cases, directly fusing the two may cause conflicts, leading to incorrect estimations of the target's tracking position. Summary of the Invention

[0003] To address the inconsistency between textual and visual information, this invention provides a target tracking method based on long and short-term context decoupling and natural language query.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A target tracking method based on long and short time context decoupling and natural language query includes the following steps:

[0006] Step 1: Extract video frame-level features using a ResNet network and Transformer encoder pre-trained on the ImageNet dataset. The specific steps are as follows:

[0007] Step 1: By using 1×1 convolution modules, the output dimension of ResNet's Res4f is reduced from 1024 dimensions to 256 dimensions.

[0008] Steps 1 and 2: Convert video frames Inputting a ResNet network, the encoder with a Transformer structure extracts features from the Res4f convolutional modules of the ResNet, denoted as . in: The Transformer-structured encoder contains a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer contains 8 heads, and the feedforward neural network contains 2 fully connected layers and a ReLU activation layer.

[0009] Step 13: Deform f' using a stretching operation to obtain video frame tokens. Where: N S=w×h;

[0010] Step 2: Concatenate video frame-level features to obtain video-level features. The specific steps are as follows:

[0011] Step 2: Given multiple video frame-level features with a time length within the range of T, concatenate these multiple video frame-level features along the tokens count axis to obtain video-level features;

[0012] Step 22: Use a feedforward neural network to obtain refined video-level features.

[0013] Step 3: Use a text encoder to obtain text query features. The specific steps are as follows:

[0014] Step 3: 1. Use a text encoder to convert each word in the text query into a one-hot vector. The text encoder includes a token embedding layer and a text transformer, in which 12 transformer encoding layers are stacked sequentially.

[0015] Step 3.2: Generate corresponding text tokens for each one-hot vector. Where: C l and N l These represent the channel dimension and number of text tokens, respectively.

[0016] Step 4: Capture dynamic context information through short-term context matching branches. The specific steps are as follows:

[0017] Step 41: Through a fully connected layer and a ReLU activation layer, Size adjustment Used to match the size of video frame-level tokens. It is a video-level feature f v Compressed representation;

[0018] Step 42, f s and Let z and x be denoted as z and x respectively. The context matching task uses the InfoNCE loss L. short Optimize:

[0019]

[0020] Where: L short This is the short-time context matching loss, where N is the number of samples in a training batch, and cos(·) is used to calculate f. s and The cosine similarity between them, where τ is a hyperparameter;

[0021] Step 5: Capture static context information through long-term context-aware branches. The specific steps are as follows:

[0022] Step 51: In the context-aware task, language tokens extracted from text queries are used as supervision;

[0023] Step 52: Transfer the text tokens f l Compress to f l 'to match f s The size is determined by passing through a fully connected layer and a ReLU activation layer. Size adjustment

[0024] Step 53, f l 'and Let z and x be the values ​​respectively, and let L be the loss through InfoNCE. long To optimize context-aware tasks:

[0025]

[0026] Where: L long It is a long-term context-aware loss;

[0027] Step 6: Adaptively adjust the required long-time or short-time context information through the long-short-time modulation module. The specific steps are as follows:

[0028] Step 61: Given the context, match the short-term tokens of the branch. Long-term tokens with context-aware branches Enhance f using long-short-time modulation module s Context-dependent features in;

[0029] Step 62: In order to adaptively adjust the required amount of long-term and short-term information, respectively... and Encoding is performed, which involves modulating video frame-level tokens using both scaling and shifting methods;

[0030] Step 63: Store in The static context information in the image is projected onto a scaling vector, and the static context information in the image is projected onto a scaling vector. The dynamic context information contained therein is projected onto a shift vector:

[0031]

[0032]

[0033] Among them: W γ W β bγ and b β Indicates learnable parameters;

[0034] Step 64: Adjust f using control vectors γ and β s The representation of:

[0035] f s '=F(f s )⊙γ+β

[0036] Where: ⊙ represents the dot product operation, and F contains a 1×1 convolutional layer and an instance normalization layer;

[0037] Step 65: Stack multiple long-short-time modulation modules together to further optimize the frame representation. The final output is denoted as...

[0038] Step 7: Estimate the target's location using the prediction module, which contains two branches: one branch estimates the probability that a candidate belongs to the foreground or background, and the other branch regresses the target's bounding box.

[0039]

[0040]

[0041] in: and P represents the feedforward neural networks used for classification and regression tasks, respectively. cls The output is a 2D vector representing the classification score of the corresponding position as either foreground or background, P. reg Output a 4-dimensional vector representing the distance from the corresponding position to the four sides of the bounding box;

[0042] Step 8: Model optimization, the specific steps are as follows:

[0043] Step 81: For context matching and context awareness tasks, L long With L short The two losses are linearly combined:

[0044] L context =(1-λ)L short +λL long

[0045] Where: λ is a hyperparameter;

[0046] Step 82: For the classification and regression branches in the prediction module, apply cross-entropy loss L respectively. cls The loss L is the sum of the sums ... GIoU The overall loss of the model is expressed as:

[0047] L=α·L context +(L cls +L GIoU )

[0048] Where α is the regularization parameter.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] This invention decouples the contextual information involved in natural language query-based target tracking into two forms: long-term context and short-term context. Short-term contextual information is obtained by a context matching task, while long-term information is captured by a context-aware task. This invention embeds these two types of contextual information into a visual tracking framework and adaptively adjusts the required contextual information to achieve better tracking performance. Compared with existing natural language query-based target tracking methods, the accuracy of target localization is significantly improved, demonstrating good tracking robustness. Attached Figure Description

[0051] Figure 1 This is a flowchart of the target tracking method based on long and short time context decoupling and natural language query of the present invention;

[0052] Figure 2 It is a long-short-time modulation module;

[0053] Figure 3 To improve the performance of other natural language query-based tracing methods by applying the long and short time context decoupling strategy;

[0054] Figure 4 This paper compares the method of the present invention with advanced visual target tracking methods on the OTB-lang and LaSOT datasets;

[0055] Figure 5 This paper compares the method of the present invention with advanced visual target tracking methods on the TNL2k dataset. Detailed Implementation

[0056] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0057] This invention provides a target tracking method based on long-short-term context decoupling and natural language query. This method introduces video segments containing short-term contextual information into a natural language query-based target tracking framework and proposes an effective fusion mechanism to reduce the impact of inconsistencies between visual and textual information. Specifically, this invention designs two joint optimization tasks: a short-term context matching task and a long-term context awareness task. The short-term context matching task collects dynamic contextual information over a period of time, while the long-term context awareness task collects global static contextual information. Furthermore, this method designs a long-short-term modulation module to effectively fuse the two different types of contextual information. The method of this invention can accurately locate targets given a natural language query and has achieved competitive results on multiple target tracking datasets containing natural language annotations.

[0058] Figure 1 The diagram shows the overall structure of this method, which can be roughly divided into five parts, as detailed below:

[0059] The first part is the encoding of the search area, video clip, and natural language query. For example... Figure 1 As shown, firstly, the encoding of the search region: This invention uses a ResNet network pre-trained on the ImageNet dataset and a Transformer encoder to extract video frame-level features. The output dimension of ResNet's Res4f is reduced from 1024 dimensions to 256 dimensions by utilizing 1×1 convolutional modules. The Transformer encoder contains a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer contains 8 heads, and the feedforward neural network contains 2 fully connected layers and a ReLU activation layer. The video frames are then processed... Input a ResNet network and extract the features of the Res4f convolutional module of ResNet, denoted as . in Next, the video frame tokens are obtained by stretching f'. Where, N S = w × h. Secondly, video segment encoding: Given multiple video frame-level features with a duration within the range of T, this invention concatenates these features along the tokens count axis to obtain video-level features. Next, this invention uses a feedforward neural network to obtain refined video-level features. Finally, natural language query encoding: The text encoder includes a token embedding layer and a text transformer. The text transformer contains 12 stacked transformer encoding layers. This invention converts each word in the text query into a one-hot vector, and then generates corresponding text tokens for each one-hot vector. Where C l and N l These represent the channel dimension and number of text tokens, respectively.

[0060] The second part captures dynamic context information through short-time context matching branches. First, a fully connected layer and a ReLU activation layer are used to... Size adjustment Used to match the size of video frame-level tokens. (f) s and Let z and x be denoted as z and x respectively. The context matching task is optimized using the InfoNCE loss:

[0061]

[0062] Where N is the number of samples in a training batch, and cos(·) is used to calculate f. s and The cosine similarity between the features is τ, which is a hyperparameter. Supervision for the context matching task is the mean of the video segment features.

[0063] The third part involves capturing static context information through a long-term context-aware branch. Compared to dynamic short-term context information, static long-term context information contains more discriminative information. Therefore, this invention designs a context-aware task that uses the current video segment to predict the global static information contained in the text query. In the context-aware task, language tokens extracted from the text query are used as supervision. Similar to the context matching task, the text tokens are... l Compress to f l 'to match f s The size. Furthermore, the present invention uses with... same process generated Next, the context-aware task uses another InfoNCE loss L long To optimize, L long With L short They have the same form, the difference being that z and x represent f respectively. l 'and In this way, L long It can guide the model to perceive static information in the entire video that also has discriminative capabilities.

[0064] The fourth part involves adaptively adjusting the required long-time or short-time context information through a long-short-time modulation module. For example... Figure 2 As shown, this module contains three inputs: short-term tokens. Long-term tokens and video frame-level tokens Specifically, video frame-level tokens are used as the query(Q) embedding of the multi-head cross-attention module. Short-time and long-time tokens are symmetrically embedded as the key(K) and value(V) of the multi-head cross-attention module, outputting short-time and long-time tokens highly correlated with the video frame-level tokens. Next, shift vectors β and scaling vectors γ are generated using the short-time and long-time tokens respectively. Using feature offsets and feature channel weights, the short-time and long-time context information highly correlated with the current video frame-level tokens is integrated into the video frame-level tokens. This adaptively adjusts the required short-time and long-time context information of the video frame-level tokens, i.e., enhancing f through a long-short-time modulation module. s The invention incorporates context-dependent features. Specifically, by adding two additional multi-head cross-attention modules between the multi-head self-attention module and the feedforward network of the video frame encoder, long-term and short-term contextual information is integrated into the video frame-level tokens. Taking the long-term portion as an example, the output of the multi-head self-attention module is used as the query embedding of the multi-head cross-attention module. In this way, the text-enhanced tokens derived from the multi-head cross-attention module possess both the main features throughout the video and the frame-level features of the current frame. Furthermore, to adaptively adjust the required amount of long-term and short-term information, the invention separately... and Encoding is performed, specifically by modulating video frame-level tokens using both scaling and shifting methods. This invention stores... The static context information in the image is projected onto a scaling vector, and the static context information in the image is projected onto a scaling vector. The dynamic context information contained therein is projected onto a shift vector:

[0065]

[0066]

[0067] Among them: W γ W β b γ and b β This represents the learnable parameters. Next, f is adjusted using the two control vectors mentioned above. s The representation of:

[0068] f s '=F(f s )⊙γ+β

[0069] Here, ⊙ represents the dot product operation, and F contains a 1×1 convolutional layer and an instance normalization layer. This invention stacks multiple long-short-time modulation modules together to further optimize frame representation. In this way, the final output is denoted as... It should be noted that this invention adds learnable positional encoding as input to each transformer encoding layer to preserve positional information.

[0070] The fifth part estimates the target's location using a prediction module. For example... Figure 1 As shown, the prediction module of this invention includes two branches: one branch is used to estimate the probability that a candidate belongs to the foreground or background, and the other branch is used to regress the bounding box of the target.

[0071]

[0072]

[0073] in, and P represents the feedforward neural networks used for classification and regression tasks, respectively. cls The output is a 2D vector representing the classification score of the corresponding position as either foreground or background, P. reg Output a 4-dimensional vector representing the distance from the corresponding position to the four sides of the bounding box.

[0074] Throughout the training phase of the model, for both context matching and context-aware tasks, this invention linearly combines these two losses:

[0075] L context =(1-λ)L short +λL long

[0076] Where λ is a hyperparameter. For the classification and regression branches in the prediction module, this invention uses cross-entropy loss Li respectively. cls The loss L is the sum of the sums ... GIoU The overall loss of the model is expressed as:

[0077] L=α·L context +(L cls +L GIoU )

[0078] Where α is the regularization parameter.

[0079] Experimental results

[0080] This invention collects video and natural language query pairs from three datasets: VisualGenome, MSCOCO, and Youtube-BoundingBox, and trains the entire model using training sets from LaSOT and OTB-lang. Furthermore, to ensure fairness in comparisons, when conducting performance tests on the TNL2k dataset, this invention only uses the TNL2k training set to train the model.

[0081] This invention is implemented in PyTorch and achieves a speed of 32 FPS on hardware equipped with an Intel(R) 10700k CPU and an Nvidia 3090 GPU. The model input is uniformly adjusted to 256×256, and the maximum length of natural language queries is set to 40. If the query length is less than 40, an empty token is added after the [SEP] token. If the query length is greater than 38, the query is truncated. This invention stacks a total of four long short-time modulation (LSTM) modules. Training is performed using the ADAM optimizer with an initial learning rate of 1e-4 and a learning rate decay factor of 0.2 every 10 epochs. The entire model is trained for 47 epochs, with each epoch containing 1000 iterations.

[0082] During the performance testing phase, this invention used three publicly available datasets with natural language query annotations for performance testing and comparison with other methods. These three datasets included OTB-lang, LaSOT, and TNL2k, and all three datasets used the same evaluation metrics, including precision (P) and success rate (AUC).

[0083] Figure 3 This paper demonstrates the performance improvements achieved by applying a short-term and long-term context decoupling strategy to other natural language query-based tracking methods. Since text queries contain global information about the video, inconsistencies may exist between the video frame representation and the text embedding. These inconsistencies can lead to drift when the target undergoes significant appearance changes, occlusion, or reappearance. To address this issue, this invention proposes a context decoupling strategy that models short-term and long-term context information through short-term context matching and long-term context awareness. Most importantly, this strategy ensures that the necessary short-term and long-term context information is collected to improve tracking accuracy. Figure 3 As shown in (a), this invention adds the proposed context decoupling strategy to existing natural language query-based trackers, including TNL, SNLT, CapsuleTNL, and AdaSwitcher, provided that only natural language is used for model initialization and the implementation details of the above methods are strictly followed. Thanks to the adaptive context information learned by the context decoupling strategy, the above methods can generate more accurate feature maps. Furthermore, as... Figure 3 As shown in (b) and 3(c), these methods achieve better performance and faster convergence.

[0084] Table 1 shows a performance comparison of the present invention and other natural language query-based target tracking methods on the OTB-lang, LaSOT, and TNL2k datasets. AUC and P represent the success rate and accuracy metrics, respectively.

[0085] Table 1 compares the performance on three datasets: OTB-lang, LaSOT, and TNL2k.

[0086]

[0087] In Table 1, the light gray section shows the performance of the method using only natural language queries for initialization, while the dark gray section shows the performance of the method using both natural language queries and bounding box initialization.

[0088] OTB-lang: As shown in Table 1, this invention achieved the best performance on the OTB-lang dataset, with a success rate of 0.738 and an accuracy of 0.948.

[0089] LaSOT: On the LaSOT dataset, this invention was compared with six other methods in terms of performance, and the success rate and accuracy reached 0.712 and 0.753, respectively, far exceeding the other methods.

[0090] TNL2k: On the more challenging TNL2k dataset, this invention also achieves state-of-the-art performance compared to other methods.

[0091] Furthermore, the performance of this invention was compared with several advanced visual object tracking methods on the OTB-lang, LaSOT, and TNL2k datasets, and the results are as follows: Figure 4 and Figure 5 As shown. These datasets contain a variety of challenging scenarios, such as fast movement, occlusion, scale changes, and motion blur. (By...) Figure 4 and Figure 5 As can be seen, thanks to the adaptive extraction strategy for both long-term and short-term contextual information, this invention achieves the highest performance score and can accurately locate targets even in relatively complex scenarios.

Claims

1. A target tracking method based on long and short-term context decoupling and natural language query, characterized in that... The method includes the following steps: Step 1: Extract video frame-level features using a ResNet network pre-trained on the ImageNet dataset and a Transformer-based encoder. The specific steps are as follows: Step 1: By using 1×1 convolution modules, the output dimension of ResNet's Res4f is reduced from 1024 dimensions to 256 dimensions. Steps 1 and 2: Convert video frames Inputting a ResNet network, the encoder with a Transformer structure extracts features from the Res4f convolutional modules of the ResNet, denoted as . in: Step 13: Deform f' using a stretching operation to obtain video frame tokens. Where: N S =w×h; Step 2: Concatenate video frame-level features to obtain video-level features. The specific steps are as follows: Step 2: Given multiple video frame-level features with a time length within the range of T, concatenate these multiple video frame-level features along the tokens count axis to obtain video-level features; Step 22: Use a feedforward neural network to obtain refined video-level features. Step 3: Use a text encoder to obtain text query features. The specific steps are as follows: Step 3:

1. Use a text encoder to convert each word in the text query into a one-hot vector; Step 3.2: Generate corresponding text tokens for each one-hot vector. Where: C l and N l These represent the channel dimension and number of text tokens, respectively. Step 4: Capture dynamic context information through short-term context matching branches. The specific steps are as follows: Step 41: Through a fully connected layer and a ReLU activation layer, Size adjustment Used to match the size of video frame-level tokens. It is a video-level feature f v Compressed representation; Step 42, f s and Let z and x be denoted as z and x respectively. The context matching task uses the InfoNCE loss L. short Optimize: Where: L short This is the short-time context matching loss, where N is the number of samples in a training batch, and cos(·) is used to calculate f. s and The cosine similarity between them, where τ is a hyperparameter; Step 5: Capture static context information through long-term context-aware branches. The specific steps are as follows: Step 51: In the context-aware task, language tokens extracted from text queries are used as supervision; Step 52: Transfer the text tokens f l Compress to f' l To match f s The size is determined by passing through a fully connected layer and a ReLU activation layer. Size adjustment Step 53, f' l and Let z and x be the values ​​respectively, and let L be the loss through InfoNCE. long To optimize context-aware tasks: Where: L long It is a long-term context-aware loss; Step 6: Adaptively adjust the required long-time or short-time context information through the long-short-time modulation module. The specific steps are as follows: Step 61: Given the context, match the short-term tokens of the branch. Long-term tokens with context-aware branches Enhance f using long-short-time modulation module s Context-dependent features in; Step 62: In order to adaptively adjust the required amount of long-term and short-term information, respectively... and Encoding is performed, which involves modulating video frame-level tokens using both scaling and shifting methods; Step 63: Store in The static context information in the image is projected onto a scaling vector, and the static context information in the image is projected onto a scaling vector. The dynamic context information contained therein is projected onto a shift vector: Among them: W γ W β b γ and b β Indicates learnable parameters; Step 64: Adjust f using control vectors γ and β s The representation of: f' s =F(f s )⊙γ+β Where: ⊙ represents the dot product operation, and F contains a 1×1 convolutional layer and an instance normalization layer; Step 65: Stack multiple long-short-time modulation modules together to further optimize the frame representation. The final output is denoted as... Step 7: Estimate the target's location using the prediction module, which contains two branches: one branch estimates the probability that a candidate belongs to the foreground or background, and the other branch regresses the target's bounding box. in: and P represents the feedforward neural networks used for classification and regression tasks, respectively. cls The output is a 2D vector representing the classification score of the corresponding position as either foreground or background, P. reg Output a 4-dimensional vector representing the distance from the corresponding position to the four sides of the bounding box; Step 8: Model optimization, the specific steps are as follows: Step 81: For context matching and context awareness tasks, L long With L short The two losses are linearly combined: THE context =(1-λ)L short +λL long Where: λ is a hyperparameter; Step 82: For the classification and regression branches in the prediction module, apply cross-entropy loss L respectively. cls The loss L is the sum of the sums ... GIoU The overall loss of the model is expressed as: L=α·L context +(L cls +L GIoU ) Where α is the regularization parameter.

2. The target tracking method based on long and short-term context decoupling and natural language query according to claim 1, characterized in that... The encoder of the Transformer structure includes a multi-head self-attention layer and a feedforward neural network. The multi-head self-attention layer contains 8 heads, and the feedforward neural network contains 2 fully connected layers and a ReLU activation layer.

3. The target tracking method based on long and short-term context decoupling and natural language query according to claim 1, characterized in that... The text encoder includes a token embedding layer and a text transformer, in which 12 transformer encoding layers are stacked sequentially.

4. The target tracking method based on long and short-term context decoupling and natural language query according to claim 1, characterized in that... The long-short-time modulation module includes short-time modulation. Long-term tokens and video frame level The system takes three inputs: a video frame-level token as the query(Q) embedding of the multi-head cross-attention module; short-time tokens and long-time tokens as the key(K) and value|(V) embedding of the multi-head cross-attention module in a symmetrical manner; and outputs short-time and long-time tokens that are highly correlated with the video frame-level token. Shift vectors β and scaling vectors γ are generated using the short-time and long-time tokens, respectively. Short-time and long-time context information that is highly correlated with the current video frame-level token is incorporated into the video frame-level tokens using two methods: feature offset and feature channel weight. This adaptively adjusts the short-time and long-time context information required by the video frame-level tokens.

Images