Rgb-l tracking method based on target-prior autoregressive query

Abstract

The application discloses an RGBL tracking method based on target prior autoregressive query, which comprises collecting and aligning an RGBL dataset, constructing a training set and a test set; taking an autoregressive query RGB tracking network AQATrack as the basis, constructing an RGBL tracking model based on target prior autoregressive query; introducing language and visual semantic tokens to learn target features of each mode; designing a language semantic token enhancement module to strengthen the target features of the language semantic tokens, and designing a target feature extraction module and a visual semantic token feature enhancement module to improve the target features of the visual semantic tokens; fusing the language and visual semantic token features in the decoder, outputting multi-modal semantic token features, and taking the multi-modal semantic token features as target prior to initialize the query with a value of zero, and capturing spatiotemporal information in an autoregressive learning mode. The application can more effectively capture target features in spatiotemporal information in the initial stage by introducing target prior, thereby speeding up the target positioning and tracking process of the tracker.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to an RGBL tracking method based on a prior autoregressive query of a target. Background Technology

[0002] Compared to tracking methods that rely solely on visual information, natural language-based tracking combines visual and linguistic information. This allows the tracking system to not only "see" the images in the video but also "understand" and respond to human language commands, thus overcoming the limitations of relying solely on visual modalities. Therefore, visual-language combined tracking technology shows enormous commercial application potential in multiple fields, such as human-computer interaction, video surveillance, and autonomous driving. However, visual-language tracking still faces many challenges, such as interference from factors like changes in lighting, camera movement, and rapid target movement. These factors cause continuous changes in the target's appearance features, making it difficult for trackers to accurately identify and continuously track the target.

[0003] To address the aforementioned challenges, incorporating temporal information to mitigate the impact of spatial appearance feature variations on tracking results is crucial. In visual-language tracking, many methods focus on modeling the interaction between visual and linguistic modalities, often neglecting the importance of spatiotemporal information. In contrast, the field of pure visual tracking has seen a wealth of excellent research exploring the extraction of spatiotemporal information; these methods can be broadly categorized into three types.

[0004] The first type of method analyzes the relationship between confidence scores and thresholds to determine whether the target template or features need to be updated to obtain a new target appearance. Although this method is effective in some cases, it usually requires manually designing update strategies and introducing hyperparameters, thus increasing the complexity of hyperparameter tuning.

[0005] The second type of method integrates the appearance information of historical frames by splicing, weighting, or using memory networks, which can better fuse spatiotemporal information. However, these methods often consume more computational resources, and may lead to error accumulation during long-term tracking, thus affecting tracking accuracy.

[0006] The third type of method effectively learns spatiotemporal information using an autoregressive approach. Taking AQATrack as an example, it inputs a set of learnable target queries and visual spatial features into the decoder, combining autoregressive learning and a sliding window mechanism to capture the spatiotemporal relationships between consecutive frames. Although this method performs well in terms of efficiency, its drawback is that it initializes the target query with only empty queries with element values ​​of zero. This results in poor fusion of cross-attention and spatiotemporal information in the initial stage, making it difficult to effectively extract target features. Furthermore, since visual spatial features contain a large amount of background information, the lack of target prior queries for autoregressive learning ultimately leads to low relevance between the captured spatiotemporal information and the target.

[0007] Therefore, how to effectively integrate spatiotemporal information, reduce background interference, and accurately extract target features remains a key problem that needs to be solved in the field of visual language tracking. Summary of the Invention

[0008] To address the problem that in existing technologies, when learning spatiotemporal information using autoregressive methods, traditional methods only initialize the target query with an empty query whose element value is zero. This results in poor fusion of cross-attention and spatiotemporal information in the initial stage, making it difficult to effectively extract target features. This invention provides an RGBL tracking method based on target prior autoregressive queries. By introducing linguistic and visual semantic tokens, it learns and fuses target features from various modalities, thereby more effectively capturing spatiotemporal information and more effectively dealing with challenges such as camera motion, motion blur, and scale changes, significantly improving tracking accuracy.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An RGBL tracking method based on a target prior autoregressive query includes the following steps: S10. Collect the labeled and aligned RGBL dataset, and construct the training and test sets from it; S20. Based on the autoregressive query RGB tracking network AQATrack, a text encoder is added to extract target text features, and a multimodal encoder is introduced to fuse target text features and visual features to construct an autoregressive query RGBL target tracking baseline network. S30. Introduce linguistic semantic tokens and visual semantic tokens into the RGBL target tracking baseline network with autoregressive query to construct an RGBL tracking network with target prior autoregressive query. S40. An optimization algorithm is used to optimize and train the RGBL tracking network with target prior autoregressive query to obtain an RGBL tracking model based on target prior autoregressive query. S50. Perform online tracking, load the trained RGBL tracking model based on the target prior autoregressive query, and test the tracking effect.

[0010] Specifically, step S30, which involves introducing linguistic semantic tokens and visual semantic tokens into the RGBL target tracking baseline network with autoregressive queries, and constructing an RGBL tracking network with prior autoregressive queries for targets, includes the following steps: S31. By introducing a language semantic token into the text branch and concatenating it with the text features, the language semantic token interacts with the target text features through an attention mechanism to capture the language semantic token features of the target. S32. By introducing a visual semantic token into the visual branch and concatenating it with the visual features, the visual semantic token interacts with the target visual features through an attention mechanism to capture the visual semantic token features of the target. S33. Enhance the language semantic token features using the Language Semantic Token Enhancement Module LSTFE; enhance the visual semantic token features using the Target Feature Extraction Module TFE and the Visual Semantic Token Feature Enhancement Module VSTFE. S34. Input the language semantic token features and the visual semantic token features into the multimodal spatiotemporal decoder for fusion to obtain the multimodal semantic token features, and use them as the target prior to initialize the query, where the initial value is zero. Capture the spatiotemporal information in the historical target query and visual features through autoregression. S35. The final output target query is used to update historical target queries and target predictions.

[0011] Specifically, the framework of the RGBL tracking network with target prior autoregressive query includes: a multimodal feature learning and token feature enhancement part, a target query adaptive regression part, and a target prediction part, wherein the input data each time includes the first frame image of the video sequence, the current frame image, and a prompt word statement describing the target in the first frame.

[0012] Specifically, the process of enhancing the language semantic token features using the Language Semantic Token Enhancement Module (LSTFE) in step S33 is as follows: First, the text features are adaptively enhanced using the Sparse Multi-Head Self-Attention (SMHSA) mechanism to mine key semantic information, and then standardized using layer normalization. Next, the cosine similarity between the visual semantic token features and the enhanced text features is calculated and normalized using an activation function. Then, the most relevant text features are selected based on the similarity score, i.e., the index position with the highest similarity is extracted. Finally, the corresponding language semantic token features are extracted from the enhanced text features according to this index position, and after linear transformation, element-wise summation is performed with the original language semantic token features, followed by layer normalization for standardization, thereby strengthening the target semantic features contained in the language semantic token. This process is expressed by the following formula: In the above formula, f l The text features represent the output of the text encoder, SMHSA represents sparse multi-head self-attention, LN represents layer normalization, and f represents the text features output by the text encoder. l_out T represents text features enhanced through sparse multi-head attention. v_outThis represents the visual semantic token feature output by the visual encoder. CS(·) represents the cosine similarity function, sigmoid(·) represents the activation function, Argmax(·) represents the function that takes the maximum index, and gather(·) represents the function that extracts the token feature according to the specified index. T represents a linear layer. l and T l_out These represent the semantic token features of the language before and after enhancement.

[0013] Specifically, the process of enhancing the visual semantic token features using the target feature extraction module TFE and the visual semantic token feature enhancement module VSTFE in step S33 is as follows: First, the target feature extraction module TFE, composed of two-dimensional convolution, layer normalization, cropping operation, and linear layer, is used to extract target features from the central region of the template image. Two-dimensional convolution is used to extract the global representation of the template image, layer normalization standardizes the feature distribution, the cropping operation extracts the core target region of the image based on the physical center coordinates, and the linear layer performs dimensional transformation and semantic alignment on the cropped target features, providing a structured target representation for the subsequent visual semantic token enhancement module. Then, the visual semantic token features are combined with the aforementioned cropping... The obtained target features are subjected to cosine similarity calculation, and then normalized using an activation function. Based on the normalized similarity score, the visual semantic feature most relevant to the target features is selected, i.e., the index position with the highest similarity is extracted. Finally, based on this index position, the most relevant visual token feature is selected from the visual features, and element-wise summed with the visual semantic token feature output by the previous visual semantic token feature enhancement module VSTFE. Subsequently, the summed feature is input into a linear transformation layer for feature mapping, and the result is standardized through layer normalization. Finally, it is element-wise added to the original visual semantic token feature, thereby effectively enhancing the visual semantic token's ability to represent the target semantics. This process is expressed by the formula: In the above formula, z represents the template image. This represents a 2D convolution with a kernel size of 16×16, LN represents layer normalization, and split(·) represents the slicing function. f represents a linear layer t f represents the target features extracted by the target feature extraction module TFE. xThe search terms represent the visual features of the search region. CS(·) represents the cosine similarity function, sigmoid(·) represents the activation function, mean(·) represents the average value function, Argmax(·) represents the function that takes the maximum value index, and gather(·) represents the function that extracts token features according to a specified index. T pv_out T represents the visual semantic token feature output by the previous visual semantic token feature enhancement module VSTFE. v and T v_out These represent the visual semantic token features before and after enhancement, respectively.

[0014] Specifically, step S34 includes: in the multimodal spatiotemporal decoder, multimodal semantic token features are obtained by fusing linguistic semantic token features and visual semantic token features, and these are used as target priors to initialize the query, with an initial value of zero, to capture spatiotemporal information from historical target queries and visual features in an autoregressive manner; this process is expressed by the formula: In the above formula, f l_out f represents the text features output by the LSTFE module. v F represents the visual features concatenated from the template region and the search region extracted by the visual spatial encoder. vl T represents the visual language features derived from the fused spatial information output by the multimodal encoder. v_out and T l_out They represent visual semantic token features and linguistic semantic token features, respectively; MHCA represents multi-head cross-attention; T vl T represents the multimodal semantic token feature. vl_hq T represents the characteristics of historical target tokens. vl_t and T vl_t_s T represents a multimodal target query that includes time information and spatiotemporal information, respectively. vl_t_s_out represents the output features of the multimodal spatiotemporal decoder, [:] represents the concatenation operation, LN represents layer normalization, and FFN represents the feedforward neural network. This indicates an element-wise multiplication operation.

[0015] Specifically, step S35, target prediction, includes: fusing target query features with visual search region features through a spatiotemporal information fusion module; then processing the visual search region features through a classification and regression prediction head to accurately locate the target; and finally outputting a tracking target bounding box. This process can be expressed by the following formula: In the above formula, T represents the visual search region features. vl This represents a multimodal semantic token feature, where P represents the classification and regression prediction head. This represents the dot product operation. This indicates an element-wise multiplication operation, and B represents the final output target box.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention introduces language semantic tokens and visual semantic tokens into the RGBL target tracking baseline network with autoregressive query, learns the target features of each modality, constructs an RGBL tracking network with target prior autoregressive query, generates multimodal semantic token features by fusing language and visual semantic tokens, and uses them as target priors for queries with initial values ​​of zero, thereby capturing spatiotemporal information more effectively through autoregressive learning.

[0017] (2) This invention improves the target prior in the language semantic token by designing the language semantic token enhancement module LSTFE; at the same time, it enhances the target prior in the visual semantic token by designing the target feature extraction module TFE and the visual semantic token feature enhancement module VSTFE; with the help of the LSTFE, TFE and VSTFE modules, the tracking method proposed in this invention can capture the multimodal spatiotemporal information of the target more efficiently, thereby more effectively dealing with challenges such as camera motion, motion blur and scale changes, and significantly improving tracking accuracy. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the network structure of an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the network structure of the language semantic token feature enhancement module in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the network structure of the target feature extraction module and the visual semantic token feature enhancement module in an embodiment of the present invention.

[0022] Figure 5This is a schematic diagram of the network structure of the multimodal spatiotemporal decoder in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments. The embodiments of the present invention include, but are not limited to, the following embodiments.

[0024] like Figures 1 to 5 As shown, this RGBL tracking method based on target prior autoregressive query includes the following steps: S10. Collect the labeled and aligned RGBL dataset, and construct the training and test sets from it.

[0025] S20. Based on the autoregressive query RGB tracking network AQATrack, a text encoder is added to extract target text features, and a multimodal encoder is introduced to fuse target text features and visual features, thereby constructing an autoregressive query RGBL target tracking baseline network.

[0026] S30. Introduce linguistic semantic tokens and visual semantic tokens into the RGBL target tracking baseline network with autoregressive queries to construct an RGBL tracking network with target prior autoregressive queries. The specific process is as follows: First, by introducing a linguistic semantic token into the text branch and concatenating it with text features, the token is input into the text encoder. The linguistic semantic token interacts with the target text features through an attention mechanism, capturing the target's linguistic semantic token features. The linguistic semantic token is configured as a learnable vector, representing the target features as linguistic semantic token features.

[0027] Next, by introducing a visual semantic token into the visual branch and concatenating it with the visual features, the visual semantic token is input into the visual encoder. The visual semantic token interacts with the target visual features through an attention mechanism, capturing the visual semantic token features of the target. The visual semantic token is configured as a learnable vector, and the target features it represents are the visual semantic token features.

[0028] Simultaneously, the target features of the language semantic token are enhanced using the language semantic token enhancement module LSTFE, i.e., language semantic token features; the target features of the visual semantic token are enhanced using the target feature extraction module TFE and the visual semantic token feature enhancement module VSTFE, i.e., visual semantic token features.

[0029] Finally, the linguistic semantic token features and visual semantic token features are input into the multimodal spatiotemporal decoder for fusion to obtain multimodal semantic token features, which are used as target priors to initialize queries (with an initial value of zero). Then, spatiotemporal information in historical target queries and visual features is captured through autoregressive learning.

[0030] The framework for an RGBL tracking network with a target prior autoregressive query is described in the following reference. Figure 2 It can be divided into three parts: multimodal feature learning and token feature enhancement, target query adaptive regression, and target prediction. The data input to the model each time includes the first frame image of the video sequence, the current frame image, and a prompt describing the target in the first frame.

[0031] The main functions of the multimodal feature learning and token feature enhancement section include: First, extracting textual features of the target through a text encoder, and simultaneously introducing a learnable vector, which is concatenated with the textual features and input into the text encoder. This vector interacts with the target-related textual features through an attention mechanism, thereby effectively capturing the semantic information of the target. The feature represented by this vector is named the linguistic semantic token feature. Subsequently, the representational ability of this linguistic semantic token feature is further enhanced by the linguistic semantic token enhancement module LSTFE. In the visual modality, extracting visual features of the target through a visual encoder, and similarly introducing a learnable vector, which is concatenated with the visual features and input into the visual encoder, allows it to interact with the visual features through an attention mechanism to mine the discriminative information of the target. The feature represented by this vector is named the visual semantic token feature, and it is further enhanced by the target feature extraction module TFE and the visual semantic token feature enhancement module VSTFE. Finally, a multimodal encoder is used to fuse the visual features and textual features of the target, achieving joint modeling of multimodal information.

[0032] The text encoder can use RoBERTa-Base, and the visual encoder can use HiViT-Base. To achieve subsequent interactive fusion of multimodal features and cosine similarity calculation, a linear transformation layer is introduced to map the features output by the text encoder, ensuring that its dimension is consistent with the visual features.

[0033] The specific process of enhancing the target features of a language semantic token using the Language Semantic Token Enhancement Module (LSTFE) is as follows: First, the text features are adaptively enhanced using the Sparse Multi-Head Self-Attention (SMHSA) mechanism to extract key semantic information, and then standardized using layer normalization. Next, the cosine similarity between the dimension-aligned visual semantic token features and the enhanced text features is calculated to measure their semantic relevance, and then normalized using an activation function. Then, the most relevant text feature is selected based on the similarity score, i.e., the index position with the highest similarity is extracted. Finally, the corresponding language semantic token feature is extracted from the enhanced text features according to this index position, and after linear transformation, it is summed element-wise with the original language semantic token features, and then standardized using layer normalization, thereby strengthening the target semantic features contained in the language semantic token. This process can be represented by the following formula: In the above formula, f l The text features represent the output of the text encoder, SMHSA represents sparse multi-head self-attention, LN represents layer normalization, and f represents the text features output by the text encoder. l_out T represents text features enhanced through sparse multi-head attention. v_out This represents the visual semantic token feature output by the visual encoder. CS(·) represents the cosine similarity function, sigmoid(·) represents the activation function, Argmax(·) represents the function that takes the maximum index, and gather(·) represents the function that extracts the token feature according to the specified index. T represents a linear layer. l and T l_out These represent the linguistic semantic token features before and after enhancement, respectively. The visual semantic token features, ultimately output by the visual encoder, are obtained by introducing a learnable vector, concatenating it with the original visual features, and then inputting them together into the visual encoder. This learnable vector interacts with the target visual features through an attention mechanism, thereby effectively learning the visual semantic information of the target. The representation result is the visual semantic token feature.

[0034] The specific process of enhancing the target features of the visual semantic token using the Target Feature Extraction (TFE) module and the Visual Semantic Token Feature Enhancement (VSTFE) module is as follows: First, the TFE module, composed of two-dimensional convolution, layer normalization, cropping, and linear layers, extracts target features from the central region of the template image. Two-dimensional convolution is used to extract the global representation of the template image, layer normalization standardizes the feature distribution, the cropping operation extracts the core target region of the image based on the physical center coordinates, and the linear layer performs dimensional transformation and semantic alignment on the cropped target features, providing a structured target representation for the subsequent visual semantic token enhancement module. Then, the visual semantic token features are combined with the aforementioned cropping... The obtained target features are subjected to cosine similarity calculation, and then normalized using an activation function. Based on the normalized similarity score, the visual semantic feature most relevant to the target features is selected, i.e., the index position with the highest similarity is extracted. Finally, based on this index position, the most relevant visual token feature is selected from the visual features, and element-wise summed with the visual semantic token feature output by the previous visual semantic token feature enhancement module VSTFE. Subsequently, the summed feature is input into a linear transformation layer for feature mapping, and the result is standardized through layer normalization. Finally, it is element-wise added to the original visual semantic token feature, thereby effectively enhancing the visual semantic token's ability to represent the target semantics. This process can be expressed by the following formula: In the above formula, z represents the template image. This represents a 2D convolution with a kernel size of 16×16, LN represents layer normalization, and split(·) represents the slicing function. f represents a linear layer t f represents the target features extracted by the target feature extraction module TFE. x The search terms represent the visual features of the search region. CS(·) represents the cosine similarity function, sigmoid(·) represents the activation function, mean(·) represents the mean function, Argmax(·) represents the function that takes the maximum index, and gather(·) represents the function that extracts token features according to a specified index. T pv_out T represents the visual semantic token feature output by the previous visual semantic token feature enhancement module VSTFE. v and T v_out These represent the visual semantic token features before and after enhancement, respectively. In the first VSTFE module, T pv_out Initialized as an all-zero feature vector, starting from the second VSTFE module, borrowing the idea of ​​residual connections, the visual semantic token feature T in the current module... v The T output from the previous module will be merged. v_out (i.e., T) pv_out This is used to enhance visual semantic features, thereby achieving layer-by-layer optimization.

[0035] The main functions of the adaptive regression part for target queries include: in the multimodal spatiotemporal decoder, multimodal semantic token features are obtained by fusing linguistic and visual semantic token features, and these are used as target priors to initialize the query (initial value is zero). This captures spatiotemporal information from historical target queries and visual features in an autoregressive manner. The final output target query is used not only to update historical target queries but also for target prediction. The feature processing flow of the multimodal spatiotemporal decoder can be represented by the following formula: In the above formula, f l_out f represents the text features output by the LSTFE module. v F represents the visual features concatenated from the template region and the search region extracted by the visual spatial encoder.vl T represents the visual language features derived from the fused spatial information output by the multimodal encoder. v_out and T l_out They represent visual semantic token features and linguistic semantic token features, respectively; MHCA represents multi-head cross-attention; T vl T represents the multimodal semantic token feature. vl_hq T represents the characteristics of historical target tokens. vl_t and T vl_t_s T represents a multimodal target query that includes time information and spatiotemporal information, respectively. vl_t_s_out represents the output features of the multimodal spatiotemporal decoder, [:] represents the concatenation operation, LN represents layer normalization, and FFN represents the feedforward neural network. This indicates an element-wise multiplication operation. The historical target token feature is formed by concatenating the target token features output by the multimodal spatiotemporal decoder during historical frame tracking, with a maximum length of 4 tokens. As tracking continues, the target token feature generated by the multimodal spatiotemporal decoder in the current frame will replace the oldest one in the historical target token sequence to maintain the timeliness and representational power of the feature.

[0036] The main functions of the target prediction section include: fusing target query features with visual search region features through a spatiotemporal information fusion module; then processing the visual search region features through a classification and regression prediction head to accurately locate the target; and finally outputting the tracking target bounding box. This process can be represented by the following formula: In the above formula, T represents the visual search region features. vl This represents a multimodal semantic token feature, where P represents the classification and regression prediction head. This represents the dot product operation. This indicates an element-wise multiplication operation, and B represents the final output target box.

[0037] S40. After completing the construction of the RGBL tracking network with target prior autoregressive query, the optimization algorithm is used to optimize and train the RGBL tracking network with target prior autoregressive query to obtain the RGBL tracking model based on target prior autoregressive query.

[0038] S50. After training is complete, perform online tracking, load the trained RGBL tracking model based on the target prior autoregressive query, and test the tracking effect.

[0039] In this embodiment, the designed RGBL tracking model based on template prior autoregressive query was trained on the training set for 150 rounds with a learning rate of 0.0002. Starting from the 120th round, the learning rate decayed to 0.00002.

[0040] Table 1 shows the performance of the method of this invention on the TNL2K and LaSOT test sets, and compares it with some existing RGBL tracking methods. From the evaluation results on the TNL2K and LaSOT datasets, using the success rate (AUC) and normalized accuracy (P0.05)... Norm As performance indicators, the method of the present invention performs well in all aspects and is superior to existing tracking methods, demonstrating its superiority.

[0041] Table 1. Comparison of the present invention with similar methods on the TNL2K and LaSOT test sets. The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any changes made based on the design principles of the present invention, or any non-creative modifications made thereon, shall fall within the scope of protection of the present invention.

Claims

1. An RGBL tracking method based on target prior autoregressive query, characterized in that, Includes the following steps: S10. Collect the labeled and aligned RGBL dataset, and construct the training and test sets from it; S20. Based on the autoregressive query RGB tracking network AQATrack, a text encoder is added to extract text features, and a multimodal encoder is introduced to fuse text features and visual features to construct an autoregressive query RGBL target tracking baseline network. S30. Introduce linguistic semantic tokens and visual semantic tokens into the RGBL target tracking baseline network with autoregressive queries to construct an RGBL tracking network with target prior autoregressive queries; this process includes: S31. By introducing a language semantic token into the text branch and concatenating it with the text features, the language semantic token interacts with the target text features through an attention mechanism to capture the target's language semantic token features. S32. By introducing a visual semantic token into the visual branch and concatenating it with the visual features, the visual semantic token interacts with the target visual features through an attention mechanism to capture the visual semantic token features of the target. S33. Enhance the language semantic token features using the Language Semantic Token Enhancement Module LSTFE; enhance the visual semantic token features using the Target Feature Extraction Module TFE and the Visual Semantic Token Feature Enhancement Module VSTFE. S34. Input the language semantic token features and the visual semantic token features into the multimodal spatiotemporal decoder for fusion to obtain the multimodal semantic token features, and use them as the target prior to initialize the query, where the initial value is zero. Capture the spatiotemporal information in the historical target query and visual features through autoregression. S35. The final output target query is used to update historical target queries and target predictions; S40. An optimization algorithm is used to optimize and train the RGBL tracking network with target prior autoregressive query to obtain an RGBL tracking model based on target prior autoregressive query. S50. Perform online tracking, load the trained RGBL tracking model based on the target prior autoregressive query, and test the tracking effect.

2. The RGBL tracking method based on target prior autoregressive query as described in claim 1, characterized in that, The framework of the RGBL tracking network with target prior autoregressive query includes: a multimodal feature learning and token feature enhancement part, a target query adaptive regression part, and a target prediction part. The input data each time includes the first frame image of the video sequence, the current frame image, and a prompt word describing the target in the first frame.

3. The RGBL tracking method based on target prior autoregressive query according to claim 2, characterized in that, The process of enhancing the language semantic token features using the Language Semantic Token Enhancement Module (LSTFE) in step S33 is represented as follows: In the above formula, f l The text features output by the text encoder are represented by SMHSA, LN represents layer normalization, and f is the text feature output by the text encoder. l_out T represents text features enhanced through sparse multi-head attention. v_out This represents the visual semantic token feature output by the visual encoder. CS(·) represents the cosine similarity function, sigmoid(·) represents the activation function, Argmax(·) represents the function that takes the maximum index, and gather(·) represents the function that extracts the token feature according to the specified index. T represents a linear layer. l and T l_out These represent the semantic token features of the language before and after enhancement.

4. The RGBL tracking method based on target prior autoregressive query according to claim 3, characterized in that, The process of enhancing the visual semantic token features using the target feature extraction module TFE and the visual semantic token feature enhancement module VSTFE in step S33 is represented as follows: In the above formula, z represents the template image. This represents a 2D convolution with a kernel size of 16×16, LN represents layer normalization, and split(·) represents the slicing function. f represents a linear layer t f represents the target features extracted by the target feature extraction module TFE. x The search terms represent the visual features of the search region. CS(·) represents the cosine similarity function, sigmoid(·) represents the activation function, mean(·) represents the average value function, Argmax(·) represents the function that takes the maximum value index, and gather(·) represents the function that extracts token features according to a specified index. T pv_out T represents the visual semantic token feature output by the previous visual semantic token feature enhancement module VSTFE. v and T v_out These represent the visual semantic token features before and after enhancement, respectively.

5. The RGBL tracking method based on target prior autoregressive query according to claim 4, characterized in that, The process of step S34 is represented as follows: In the above formula, f l_out f represents the text features output by the LSTFE module. v F represents the visual features concatenated from the template region and the search region extracted by the visual encoder. vl T represents the visual language features derived from the fused spatial information output by the multimodal encoder. v_out and T l_out They represent visual semantic token features and linguistic semantic token features, respectively; MHCA represents multi-head cross-attention; T vl T represents the multimodal semantic token feature. vl_hq T represents the characteristics of historical target tokens. vl_t and T vl_t_s T represents a multimodal target query that includes time information and spatiotemporal information, respectively. vl_t_s_out represents the output features of the multimodal spatiotemporal decoder, [:] represents the concatenation operation, LN represents layer normalization, and FFN represents the feedforward neural network. This indicates an element-wise multiplication operation.

6. The RGBL tracking method based on target prior autoregressive query according to claim 5, characterized in that, The process of target prediction in step S35 is represented as follows: In the above formula, T represents the features of the visual search region. vl This represents a multimodal semantic token feature, where P represents the classification and regression prediction head. This represents the dot product operation. This indicates an element-wise multiplication operation, and B represents the final output target box.

Images