Adaptive vision-language matching detection and tracking method

Abstract

This invention belongs to the fields of computer vision, vision-language match detection, and multimodal target tracking, and discloses an adaptive vision-language match detection and tracking method. First, the method leverages the modal alignment and representation advantages of a pre-trained multimodal large model to construct a vision-language multimodal tracking large model. A visual context-aware cue learning algorithm is proposed to extract visual context features from the current search features and integrate them into the text encoding process using cue learning, enabling dynamic perception of visual changes by text features. Furthermore, a language-adaptive implicit update mechanism is designed. By embedding learnable vectors in the text encoding process and combining them with dynamic visual context cues, text features are implicitly updated during tracking, enabling static language to dynamically match visual appearance, thereby improving vision-language detection and tracking performance.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, vision-language matching detection, and multimodal target tracking, and relates to an adaptive vision-language matching detection and tracking method. Background Art

[0002] Vision-language detection and tracking aims to first detect and match a specified target in the initial frame, and then locate the specific target in each frame of the video sequence based on the language description of the initial target state. Compared with traditional pure visual tracking that relies on bounding boxes in the initial frame to identify the target, vision-language tracking has a wider range of practical application scenarios, such as human-computer interaction, autonomous driving, and robot tracking. In addition, the rich semantic information provided by natural language helps to reduce the ambiguity problems common in pure visual tracking and enhance the discriminative ability of tracking models. Despite its huge potential, the research progress of vision-language detection and tracking lags significantly behind that of pure visual tracking. Recently, some vision-language detection and tracking methods have adopted a single-stream Transformer architecture similar to the cutting-edge models in the field of pure visual tracking. They typically use language models such as BERT as text encoders, concatenate text tags with visual tags, and then perform attention calculations through a single-stream Transformer.

[0003] TNLS-II is the first proposed visual language tracking baseline framework. In this framework, an LSTM network is used to encode text queries, and the subsequent tracking process is similar to the tracking paradigm based on Siamese networks, where language embeddings and template image features are used as references for cross-correlation operations. RTTNLD is a detection-based visual language tracking method that generates language-related candidate regions through a candidate box generation network and uses an LSTM-based tracker for subsequent prediction. The GTI method decomposes the visual language tracking task into three subtasks and constructs a "semantic localization-tracking-fusion" framework. The core component is a fusion module that guides the fusion process of semantic localization and tracking. The CTRNLT method introduces language information from the perspective of cross-modal retrieval and proposes an adaptive retrieval switching module to switch between local tracking and global search.

[0004] Unlike the aforementioned frameworks that separate task modules, the JointNLT method proposes a joint detection and tracking framework that reconstructs semantic localization and tracking into a unified task that can adapt to the different characteristics of the two reference information. The MMTrack method regards vision-language detection and tracking as a word vector generation task, aiming to learn a unified multimodal representation. Its core idea is to serialize language descriptions and bounding boxes into a sequence of discrete word vectors and process them using an autoregressive model. The UVLTrack method designs a modality-unified feature extractor, a multimodal contrast loss function, and a modality-adaptive bounding box regression head to uniformly handle three different input reference settings and keep parameters consistent. OneTracker further proposes a general framework for unifying various multimodal tracking tasks, including vision-language detection and tracking. It regards the visible light tracker as a pre-trained base model and other modal information as prompt information, integrating them through efficient parameter fine-tuning.

[0005] However, even with the addition of language cues, existing vision-based tracking methods still underperform vision-only tracking, with the performance gap being even more pronounced for language-only tracking. The advantages of language in tracking tasks have not been fully explored for two main reasons.

[0006] First, existing vision-to-language trackers lack an effective cross-modal alignment mechanism. The significant modality gap between language and images makes cross-modal alignment challenging. Current mainstream vision-to-language tracking methods often use BERT for text encoding, but this encoding process does not consider alignment with image encoding in a shared embedding space. In contrast, large-scale vision-to-language models such as CLIP achieve well-aligned embedding representations through pre-training on contrasting image-text pairs. Their excellent transfer capabilities have been demonstrated in downstream tasks such as image description generation, visual question answering, and semantic segmentation. However, the application of such general-purpose vision-to-language models in vision-to-language tracking remains very limited. While the recent work CiteTracker introduces CLIP to enhance pure visual tracking, it restricts its generated language to a predefined pool of attributes, limiting the potential of vision-to-language models. How to fully leverage the powerful alignment capabilities of vision-to-language models to advance vision-to-language tracking remains an unresolved challenge.

[0007] Furthermore, static language descriptions are prone to deviations from the dynamic search image. During tracking, the initial fixed language description of the target struggles to keep pace with the continuously changing search image. For example, the target may undergo significant changes in position (e.g., left / right) or appearance attributes (e.g., shape, color, texture). Vision-language tracking requires adaptive mechanisms to address these deviations, but existing methods often overlook this dynamic adaptation challenge due to the complexity of the problem.

[0008] Based on the above analysis, the present invention constructs an adaptive vision-language matching detection and tracking method based on contextual cues. The CLIP text encoder is used to obtain text representations aligned with vision, and a contextual cue module is used to guide CLIP to perform cue learning. This enables the model to obtain updated language representations that match vision in real time according to changes in target appearance during the tracking process, effectively alleviating the alignment deviation problem between semantics and vision. Summary of the Invention

[0009] The present invention aims to address the problems of inter-modal feature misalignment and weak matching between static language and dynamic visual information in existing visual-language detection and tracking frameworks, and to build a high-performance visual-language target tracking model that can be based on context-adaptive perception. Specifically, the present invention first utilizes the modal alignment and representation advantages of the pre-trained multimodal large model to build a visual-language multimodal tracking large model; and proposes a visual context-aware cue learning algorithm to extract visual context clues from the current search image features, and uses cue learning to integrate them into the text encoding process to achieve dynamic perception of visual changes by text features; in addition, a language-adaptive implicit update mechanism is designed to embed learnable vectors in the text encoding process and combine them with dynamic visual context clues to achieve implicit updates of text features during tracking, so that static language can dynamically match the visual appearance, thereby improving visual-language detection and tracking performance.

[0010] The technical solution of the present invention:

[0011] An adaptive vision-language matching detection and tracking method, the steps are as follows:

[0012] Step 1: Build a large vision-language multimodal tracking model;

[0013] The large model of visual-language multimodal tracking includes an image branch, a text branch, an association model, and a tracking module. The image branch and the text branch are used to process image and text inputs, respectively. The image branch includes an image encoder, through which the template image and the search image are encoded to obtain template features and search features. The text branch contains a context prompter and a text encoder. After each word of the language instruction is encoded into a text word vector through the word segmentation module of the CLIP pre-trained model, the search features are fused into the text word vector as prompts through the context prompter. The fused text word vector extracts text features through the text encoder. The obtained template features, search features, and text features are then input into the association model for multimodal attention interaction fusion to obtain fused template features, search features, and text features. Finally, the fused search features are input into the tracking module for tracking prediction.

[0014] The overall process of the vision-language multimodal tracking model is as follows:

[0015]

[0016]

[0017]

[0018]

[0019] in, and Represent the template image and the current t-th frame search image respectively; Indicates language instructions; and Represented by the image encoder The template features and search features corresponding to the extracted template image and the current t-th frame search image; For image encoder The corresponding parameters of Represented by the text encoder The corresponding text features extracted; For text encoder The corresponding parameters of Indicates that the parameter is Context prompter; Indicates the number of Predefined learnable word vectors for ; Represents an association model; Represents the tracking module; and These are the corresponding parameters of the association module and the tracking module respectively;

[0020] The fused text word vector enhanced by visual context features is then passed through the text encoder Encoded as text features ;

[0021] Step 2: Training process based on cue learning;

[0022] The large-scale vision-language multimodal tracking model uses large-scale vision-language training samples for contrastive learning, resulting in a pre-trained model with strong modality alignment and representation capabilities. The CLIP pre-trained model is used as the text encoder, while the image encoder and association model are initialized using the Fast-iTPN pre-trained model trained using CLIP supervision.

[0023] Due to the lack of visual-language tracking training samples, directly fine-tuning the text encoder can easily lead to overfitting. Therefore, during training, the present invention freezes the CLIP text encoder parameters, while the contextual cue generator is fine-tuned through cue learning. Specifically, text encoding optimization is achieved by simply updating a small number of predefined vector parameters and a lightweight context network. Through this efficient cue learning approach, optimized model parameters adapted for the object tracking task are learned during the training phase using limited visual-language tracking sample pairs.

[0024] Step 3: Reasoning process of text adaptive implicit update;

[0025] In the tracking and inference stage, the previous visual-language target tracking method often only extracts text features once in the initial frame during the inference stage, without updating the text features, resulting in a mismatch between the unchanging text and the dynamically changing visual information, thereby misleading the tracking process. The present invention uses the context prompter trained in step 2 to extract the visual context features of the current frame during the tracking process of each frame, and inserts them into the text word vector sequence through a learnable vector to achieve adaptive implicit update of the text features. The updated text features will be able to effectively adapt to the dynamic changes in visual appearance, thereby avoiding the problem of misleading text.

[0026] Beneficial effects of the present invention:

[0027] (1) The context-based adaptive visual-language tracking method proposed in this paper can effectively utilize the modal alignment and representation advantages of a large multimodal model, and improve the flexibility and robustness of visual target tracking by proposing visual context-aware cue learning and language adaptive implicit updating, thereby solving the problems of feature misalignment and information mismatch between modalities in existing visual-language multimodal tracking, and providing a new idea and basic framework for high-performance visual-language multimodal tracking.

[0028] (2) In applications in the field of embodied intelligence, this invention can enable intelligent agents (such as service robots, autonomous driving systems, etc.) to understand human natural language instructions in complex scenarios in real time, match and detect the targets described by the instructions, and dynamically track the target objects. For example, in a home service scenario, the robot can adaptively integrate visual context (such as lighting changes, occlusions, or target motion) based on the user's verbal instructions (such as "track the moving blue cup") to achieve highly robust continuous tracking, thereby enhancing the interaction ability of the intelligent agent with the physical environment and promoting the development of embodied intelligence towards more natural and efficient human-machine collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a framework diagram of the adaptive vision-language matching detection and tracking method.

[0030] Figure 2 This is the structural diagram of the adaptive vision-language matching detection and tracking model. DETAILED DESCRIPTION

[0031] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0032] like Figure 1 As shown, the present invention constructs a concise and effective vision-language detection and tracking framework. Image and text inputs are first segmented and encoded using an image encoder and a text encoder, respectively. The resulting visual and language word vectors are concatenated into a unified representation and fed into a Transformer-based association model for multi-layer attention interaction. The present invention uses the CLIP text encoder to extract aligned text encodings, maintaining its pre-trained parameters unchanged during training. To enable prompt learning of the text encoder and adaptive language encoding, the present invention proposes a contextual prompter that introduces several learnable vectors. These learnable word vectors replace some invalid word vectors in the text word vector sequence, thereby facilitating CLIP prompt learning. Furthermore, during inference, these word vectors are updated based on the search features output by the image encoder, incorporating current visual context information (i.e., search features) into the language encoding process. This dynamic adaptive mechanism enables the final language encoding to adjust based on the current tracking state, effectively alleviating the problem of misalignment between vision and language.

[0033] The specific structure of the adaptive vision-language matching detection and tracking model is as follows: Figure 2 As shown. The visual-language tracking framework proposed in the present invention is implemented as a unified word vector model. The image and language inputs are encoded by the image encoder and the text encoder respectively. The output template features and search features are spliced ​​with the text word vectors and then unified and fused through the Transformer-based association model. In response to the challenges of modal alignment and the possible mismatch between fixed language descriptions and dynamic visual changes, the present invention proposes a context prompter, which can achieve both modal alignment and adaptive adjustment of text embedding. The overall process of the visual-language multimodal tracking model is shown as follows:

[0034]

[0035]

[0036]

[0037]

[0038] in, and Represent the template image and the current t-th frame search image respectively; Indicates language instructions; and Represented by the image encoder The template features and search features corresponding to the extracted template image and the current t-th frame search image; For image encoder The corresponding parameters of Represented by the text encoder The corresponding text features extracted; For text encoder The corresponding parameters of Indicates that the parameter is Context prompter; Indicates the number of Predefined learnable word vectors for ; Represents an association model; Represents the tracking module; and They are the corresponding parameters of the association module and the tracking module respectively.

[0039] The text branch of the present invention includes two steps, namely, the context-aware language segmentation process and the text encoding process. Specifically, the input language description First, the word segmentation module of the CLIP pre-trained model is decomposed and encoded into text word vectors. In order to improve the multimodal alignment capability and achieve dynamic adaptation to visual changes, the context prompter of the present invention Introduce the current visual context features into text encoding, that is, from the current search features Extract adaptive visual information and insert multiple learnable word vectors To reconstruct the initial text word vector sequence. The reconstructed text word vector enhanced by visual context clues is then passed through the text encoder Encoded as text features , the text encoder is implemented using CLIP's text encoder, whose parameters are kept frozen during training, while the contextual prompter of the present invention is fine-tuned through prompt learning.

[0040] The image encoder and association module of the present invention are implemented using the HiViT model; the first two stages of the HiViT model act as image encoders for image encoding, and the third stage acts as an association module to process multimodal attention interaction fusion; the word segmentation and text encoder of language instructions are implemented using the CLIP pre-trained model;

[0041] The context prompter contains several learnable word vectors and a context network, which is implemented by a convolutional neural network. The language instruction is segmented by the word segmentation module of the CLIP pre-trained model to obtain a text word vector. The text word vector is surrounded by a start word vector [SOS] and an end word vector [EOS], which respectively identify the start and end positions of the text word vector. To ensure that the length of the text word vector encoding sequence is fixed, the CLIP pre-trained model adds several padding word vectors after the end word vector [EOS]. The context prompter inserts multiple learnable vectors carrying visual context features at the end of the text word vector and immediately before the end word vector [EOS]. , while removing the last Fill word vectors to keep the sequence length unchanged; for the inserted learnable vectors , through the context network in the context prompter, the visual context features are first extracted from the current search features, and then the obtained visual context features are superimposed on each learnable vector Finally, a fused text word vector is constructed; the context prompter integrates the current visual context information into the text encoding process, so that the text features can dynamically respond to visual changes, thereby achieving robust tracking performance.

[0042] To verify the effectiveness of the present invention, the tracking performance of the present invention was tested in four challenging visual-language tracking datasets, including LaSOT, LaSOT EXT , TNL2K, and OTB99. The evaluation metric is tracking success rate. Table 1 shows the tracking performance of our method in each dataset and compares it with other cutting-edge tracking methods. It can be seen that our method surpasses other cutting-edge methods in all four datasets, achieving optimal performance.

[0043] Table 1 Tracking performance of the present invention in various data sets and comparison with other cutting-edge tracking methods

[0044] .

Claims

1. An adaptive vision-language matching detection and tracking method, characterized in that: Here are the steps: Step 1: Build a large vision-language multimodal tracking model; The large-scale vision-language multimodal tracking model includes an image branch, a text branch, an association model, and a tracking module. The image branch and the text branch are used to process image and text inputs, respectively. The image branch includes an image encoder, which encodes the template image and the search image to obtain template features and search features. The text branch includes a contextual prompter and a text encoder. First, each word in the language instruction is encoded into a text word vector using the word segmentation module of the CLIP pre-trained model. The contextual prompter then fuses the search features as prompts into the text word vector. The fused text word vector is then passed through the text encoder to extract text features. The acquired template features, search features, and text features are then fed into the association model for multimodal attention interaction fusion, resulting in fused template features, search features, and text features. Finally, the fused search features are fed into the tracking module for tracking prediction. The overall process of the vision-language multimodal tracking model is as follows: ； ； ； ； in, and Represent the template image and the current t-th frame search image respectively; Indicates language instructions; and Represented by the image encoder The template features and search features corresponding to the extracted template image and the current t-th frame search image; For image encoder The corresponding parameters of Represented by the text encoder The corresponding text features extracted; For text encoder The corresponding parameters of Indicates that the parameter is Context prompter; Indicates the number of Predefined learnable word vectors for ; Represents an association model; Represents the tracking module; and These are the corresponding parameters of the association module and the tracking module respectively; The fused text word vector enhanced by visual context features is then passed through the text encoder Encoded as text features ; Step 2: Training process based on cue learning; The CLIP pre-trained model is used as the text encoder, and the image encoder and association model are initialized using the Fast-iTPN pre-trained model trained using CLIP supervision; The parameters of the text encoder are frozen, and the context prompter adjusts its parameters through prompt learning; the text encoding is optimized only by updating the parameters of the learnable vector and the context network of the context prompter; Step 3: Reasoning process of text adaptive implicit update; In the tracking and inference stage, the context prompter trained in step 2 is used to extract the visual context features of the current frame during the tracking process of each frame, and inserts them into the text word vector sequence through the learnable vector to achieve adaptive implicit update of the text features.

2. The adaptive vision-language matching detection and tracking method according to claim 1, characterized in that: The image encoder and association module are implemented using the HiViT model; the first two stages of the HiViT model serve as image encoders for image encoding, and the third stage serves as an association module to handle multimodal attention interaction fusion; the word segmentation and text encoder of language instructions are implemented by the CLIP pre-trained model.

3. The adaptive vision-language matching detection and tracking method according to claim 1, characterized in that: The context prompter contains several learnable word vectors and a context network, which is implemented by a convolutional neural network. The language instruction is segmented by the word segmentation module of the CLIP pre-trained model to obtain a text word vector. The text word vector is surrounded by a start word vector [SOS] and an end word vector [EOS], which respectively identify the start and end positions of the text word vector. To ensure that the length of the text word vector encoding sequence is fixed, the CLIP pre-trained model adds several padding word vectors after the end word vector [EOS]. The context prompter inserts multiple learnable vectors carrying visual context features at the end of the text word vector and immediately before the end word vector [EOS]. , while removing the last Fill word vectors to keep the sequence length unchanged; for the inserted learnable vectors , through the context network in the context prompter, the visual context features are first extracted from the current search features, and then the obtained visual context features are superimposed on each learnable vector Finally, the fused text word vector is constructed.

Images