Multi-modal tracking method based on apparent time sequence modeling
By introducing dynamic templates and appearance temporal modeling into the multimodal tracking method, and using visual Transformer and Mamba models to generate spatiotemporal context tokens, the problem of decreased tracking accuracy caused by neglecting temporal information in existing methods is solved, and stable target tracking in complex scenarios is achieved.
Patent Information
- Application Number
- CN202511606927.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-05
- Publication Date
- 2026-03-13
AI Technical Summary
Existing multimodal tracking methods rely on target appearance information for matching and tracking, ignoring temporal information, which leads to decreased tracking accuracy in complex scenes, especially in cases of occlusion and deformation, making it difficult to identify the target position.
By introducing dynamic templates and appearance temporal modeling through a visual Transformer-based multimodal tracking framework, information is collected from target change cues using attention mechanisms and Mamba models to generate spatiotemporal context tokens. Combined with initial and dynamic template tokens, the feature drift and forgetting problems are mitigated.
It improves the robustness and accuracy of multimodal tracking, enables stable target tracking in complex scenarios, enhances the utilization of temporal context information, and improves the tracker's recognition capabilities.
Smart Images
Figure CN121661095A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal tracking methods, and in particular to a multimodal tracking method based on appearance time series modeling. Background Technology
[0002] Visual object tracking (VOT) is a fundamental task in computer vision, aiming to predict the position of a target in subsequent frames given its initial state. This technology has significant value in various practical applications such as video surveillance and security, drone reconnaissance, and human-computer interaction. However, RGB trackers are limited by visible light imaging mechanisms and often perform poorly in complex scenarios with changing lighting and occlusion. Therefore, multimodal tracking is attracting increasing attention.
[0003] Multimodal tracking leverages the complementarity between different modalities to achieve more robust tracking. Common modal combinations include RGB+thermal imaging (RGB-T), RGB+depth (RGB-D), and RGB+events (RGB-E). Most existing multimodal tracking methods focus on feature fusion and interaction methods between different modalities. Although these methods have achieved promising results, they rely entirely on spatial information such as target appearance for matching, localization, and tracking, neglecting the important role of temporal information such as context in the video.
[0004] To address the aforementioned problems, this invention proposes a multimodal target tracking method based on appearance temporal modeling. This method collects reliable target change cues from dynamically updated template images, continuously acquiring the target's appearance features throughout the video sequence tracking process. This information is then compressed into the hidden state space, establishing long-range contextual relationships between video frames. The continuous updating of reference information during tracking enhances the tracker's perception and utilization of spatiotemporal contextual information, thereby improving the accuracy of model tracking.
[0005] "CAO B, GUO J, ZHU P et al., 2024. Bi-directional Adapter for Multimodal Tracking[J / OL]. Proceedings of the AAAI Conference on Artificial Intelligence, 38(2): 927-935. DOI:10.1609 / aaai.v38i2.27852." This paper proposes a multimodal Transformer tracking visual cue framework based on a bi-directional adapter. The framework mainly consists of three parts: image pair input, backbone network, and prediction head. The image pair contains target templates and search region images from two modalities. The backbone network is a dual-stream Transformer encoder embedded with a bi-directional adapter. The dual-stream Transformer encoder can perform unified feature extraction and fusion of the input target template and the search image. The bi-directional adapter can perceive the dynamic changes of the dominant modality in an open scene and effectively fuse multimodal information in an adaptive manner. Finally, the features output from the two modalities are added together and fed into the prediction head for target localization.
[0006] This invention is also based on the visual Transformer multimodal tracking framework. Unlike existing methods that rely on spatial information or only on sparse temporal information, this invention designs an appearance information aggregation mechanism based on attention mechanism and Mamba model. Through Mamba's selective scanning mechanism, it collects target change cues from continuously updated target appearance information, continuously compresses reliable target appearance information into the hidden state space to update target context information, and simultaneously aggregates target context history information into the current tracking reference information to assist current tracking, thereby establishing long-distance contextual relationships between video frames.
[0007] The problem with multimodal offline tracking methods, exemplified by BAT (Baidu, Alibaba, Tencent), is that they rely on target appearance information for matching and tracking, focusing on feature fusion and interaction between different modalities, while neglecting the utilization of temporal information. In complex real-world scenarios, targets may encounter deformation, occlusion, and other complex conditions. As tracking progresses, the target's appearance will shift relative to the initial template, making it difficult for offline trackers to identify the latest target position. In such cases, introducing temporal context information—updating target reference information by recording changes in the target's appearance—can improve the tracker's ability to identify targets, thereby enhancing the model's tracking accuracy.
[0008] Taking the BAT tracking algorithm as an example, as an offline tracker, its initial method relies solely on target appearance information for tracking without utilizing any temporal information. We visualized its attention map during the tracking process of the yellowcar sequence on the RGBT234 dataset. Figure 1 As shown, the first row contains RGB images, and the second row is a visualization of the BAT tracker's attention map. In the RGB images, red boxes indicate the actual location of the target, and the blue numbers in the upper corner of each image represent its index in the sequence. Figure 1 As shown, as tracking progresses, the target will face complex scenarios such as occlusion and interference from similar objects, which will reduce the similarity with the initial template. The tracker's attention will be diverted by other interference objects and will not be able to focus on the real target, ultimately resulting in errors in the target tracking results.
[0009] As a contrast, we added a dynamic template to the BAT method, thereby providing the tracker with some sparse temporal information. After identical training, we tested the performance of both tracking methods on the LasHeR dataset. Figure 1 As shown, "BAT" represents the original BAT method, and its result is represented by the green line. "BAT + DynamicTemplete" represents the BAT tracing method with dynamic templates added, and its result is represented by the red line. It can be seen that even with just the addition of dynamic templates, the performance of the BAT tracing method is significantly improved. Summary of the Invention
[0010] To address the problems existing in the prior art, the present invention aims to provide a multimodal tracking method based on apparent time series modeling.
[0011] To achieve the above objectives, the technical solution of the present invention is as follows: This invention proposes a multimodal tracking method based on appearance temporal modeling, which is based on a visual Transformer multimodal tracking framework. First, a dynamic template token is updated based on a confidence threshold update mechanism to obtain the latest reference information of the target. Then, using an initial template token containing initial reference information and a dynamic template token containing the latest reference information, the target appearance information is continuously compressed into the hidden state space through an appearance information aggregation module using the Mamba selection scanning mechanism, thereby generating a spatiotemporal context token. Finally, the initial template token, dynamic template token, and spatiotemporal context token form the target appearance temporal information, which alleviates the problems of feature drift and forgetting in long-sequence multimodal tracking tasks, thereby promoting the robustness of multimodal tracking. Furthermore, the steps of this method are as follows: Step 1: Input the first frame of the video; A video sequence contains several frames, each frame containing an RGB modal image (visible light image) and an X modal image (an image in addition to the RGB modal), and each frame contains at least one target. The target's position in the first frame is known, while the target's position in each of the remaining frames is unknown. The number of video frames is a positive integer greater than 1. The top-left corner of each frame in the video frame sequence is the origin. The width and height are respectively and Input the first frame image and automatically or manually mark the rectangular area of the tracked target. That is, the selected tracking target; among which, This represents the coordinates of the top-left corner of the rectangular region. Let the width and height of the rectangular region be respectively; Indicates the current frame number, the first frame number. The target selected in the first frame is also called the tracking result of the current frame. = ; Step 2: Generate the initial template token sequence; Zoom in on the rectangular area of the selected target in the first frame. The magnified area corresponds to a change in image size. , as the target initial template image for both modalities and Where the height and width are equal, that is , Indicates the number of channels in an image, indicated by a superscript. Indicates the initial template; right and The image is divided into blocks, each with a pixel resolution of [missing value]. It can form a sequence of template image blocks with two modalities. and ,in This indicates the number of template image patches for a given modality. This represents the token embedding dimension corresponding to an image patch, indicated by the superscript. Represents an image block; and After linear projection and add position embedding and It can generate initial template token sequences for two modalities. and ,Right now and superscript Indicates the target template. Indicates the initial template; Step 3: Initialize the dynamic template token sequence; make and These represent the dynamic template token sequences for two modalities, with superscripts indicating the two modes. This indicates a dynamic template; for the first frame... In this case, the dynamic template token sequence is initialized with the target initial template token sequence, i.e. , ; Step four generates a spatiotemporal context token sequence; Because RGB modal data and X modal data have different characteristics, the spatiotemporal context token sequences for the two modalities are generated independently; based on the initial template token sequence of the RGB modality... and dynamic template token sequence It can generate spatiotemporal context token sequences of RGB modality. Based on the initial template token sequence of the RGB modality and dynamic template token sequence It can generate spatiotemporal context token sequences for mode X. The appearance information aggregation module can generate spatiotemporal context token sequences, which include two sub-steps: context alignment fusion and context information aggregation. The generation methods for the spatiotemporal context token sequences of the two modalities are the same. The following only uses the RGB modality as an example to illustrate its generation method. 4.1 Context alignment and blending; To alleviate problems such as feature drift and forgetting in long sequence tracking tasks, this invention uses an initial template token sequence based on a cross-attention method. For dynamic template token sequences The process of performing context-aligned fusion can be represented as follows: ; in and These represent the layer normalization operation and the activation function, respectively. , and For the learnable parameters corresponding to the query, key, and value, Scaling factor for The token embedding dimension, This represents the target template token sequence after alignment and fusion. 4.2 Contextual information aggregation; Will Sending it to the Mamba layer enables the aggregation of context information, and the specific process is as follows: (1) To RMS normalization was performed to obtain , can be represented as: ; (2) After linear projection operation and Silu activation function Linear processing results can be obtained. ,in The token embedding dimension after linear projection dimension expansion can be represented as: ; (3) After linear projection operation Positive features can be obtained. Further reverse operation Inverse features can be obtained. , can be represented as: ; ; and After each forward convolution operation and backward convolution operation Silu activation function And it is sent to the state space model. Extracting and updating context information can be represented as follows: ; ; in , This represents the forward and backward hidden states in Mamba. For sequence length, The internal feature dimension of the hidden state. For the state dimension of Mamba, , Represents the updated hidden state; state-space model It is a key part of the Mamba layer, which models the dynamics of sequential data through hidden states. , These are the output features of the state-space model; (4) Process the results using linear methods Output features of the state-space model and Through dot product operation Perform a weighted summation, and then perform an element-wise addition operation on the weighted result. get , can be represented as: ; The linear projection layer can reduce The feature dimensions, and then with By performing residual joins, the output of the Mamba layer can be obtained. This output aggregates the context information stored in the hidden state space of the Mamba layer, and can be represented as: ; This refers to the generated RGB modal spatiotemporal context token sequence; Step 5: Input the next frame and generate the search area token sequence; Input a new video frame as the current frame, and increment the current frame number by 1. ; Track the results of the previous frame Enlarged rectangular area b 2 The magnified area corresponds to a change in image size. The search region image serves as both modalities. and ,in and These represent the height and width of the search region image, respectively. , The number of channels in an image, symbol Indicates the search area; right and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and ,in This indicates the number of image patches in a modal search region. This indicates the token embedding dimension, which is the same as the template image patch embedding dimension. and After linear projection and add position embedding and It can generate two modal search region token sequences. and ,Right now and ; Step Six: Feature Extraction and Fusion; The target initial template token sequence of the two modalities and Dynamic template token sequence and Spatiotemporal context token sequence and and search area token sequence and Connect them to form a token sequence , can be represented as: ; Will By feeding the data into the backbone network for feature extraction and fusion, the output features of the backbone network can be obtained. , can be represented as: ; in , , , These represent the initial template feature sequence, dynamic template feature sequence, spatiotemporal context feature sequence, and search region feature sequence of the RGB modal output, respectively. , , , These represent the initial template feature sequence, dynamic template feature sequence, spatiotemporal context feature sequence, and search region feature sequence of the X modality output, respectively. The backbone network of this invention can adopt a single-branch ViT network, a multimodal branch ViT network, or an improved network thereof; wherein a single-branch ViT network refers to a network that... The first type of network is a single ViT network for unified feature extraction and fusion. The second type is a multimodal branch ViT network, which means that token sequences of different modalities are fed into different ViT networks for feature extraction and fusion respectively. The third type is an improved network based on the first two types of networks. For example, based on the multimodal branch ViT network, bidirectional adapters can be embedded between encoders of the same layer of different modalities to realize feature cues between modalities. Step 7: Determine the tracking results; By fusing the feature sequences of the two modalities of the search region output by the backbone network, we can obtain the fused search region feature information. Taking the element-wise addition fusion method as an example, the search region feature information... It can be represented as: ; Using search area feature information as the basis for target localization is a standard processing method in the target tracking and localization stage. This mainly includes reinterpreting the search area feature information into a two-dimensional feature map, and then... Conv-BN- ReLU The tracking model uses a multi-layered, fully convolutional network to determine the target location based on classification score maps and local offsets; the total training loss is: ; Among them, classification loss use Weighed Focal Loss Classification is performed, and regression loss is adopted. L1 loss and Generalized IoU loss Perform bounding box regression. and They are respectively Generalized IoU Loss and L1 The regularization parameter corresponding to the loss; the target localization result is used as the tracking result of the current frame. ; Step 8: Determine if it is the last frame; If the current frame is the last frame, the tracking ends; otherwise, continue with the subsequent steps. Step 9: Determine whether the confidence level of the tracking result has reached the threshold; The confidence level corresponding to the target center position is obtained from the classification score map in the current frame tracking process and compared with the confidence level threshold. If the confidence level corresponding to the target center position does not reach the confidence level threshold, proceed to step five; otherwise, continue to execute the subsequent steps. Step 10: Update the dynamic template token sequence; Tracking results of the current frame Image magnification of corresponding rectangular region Times, deformation Size, serving as a dynamic template image for both modalities. and Where the height and width are equal, that is , Represents the number of image channels; for and The image is divided into blocks, each with a pixel resolution of [missing value]. To form a dynamic template image block sequence and After linear projection Add position embedding and It can generate dynamic template token sequences with two modalities. and ,Right now and After the dynamic template token sequence is updated, proceed to step four.
[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: In real-world target tracking systems with complex scenarios, both the target and the background are constantly changing. Relying solely on target appearance information for matching and tracking often results in interference from target deformation and occlusion, thus affecting the tracker's accuracy. In short, methods that rely solely on target appearance information for target matching and tracking through spatial feature fusion and interaction between different modalities severely underutilize temporal information, leading to decreased tracking accuracy. Therefore, this invention proposes a multimodal tracking method based on appearance temporal series modeling. This method continuously collects temporal contextual information by aggregating reliable cues of target appearance changes. During tracking, it simultaneously utilizes early target reference information, the latest target reference information, and historical contextual information aggregating cues of target appearance changes. The combination of these three elements provides the tracker with rich spatiotemporal contextual information, helping it to perform better tracking. Attached Figure Description
[0013] Figure 1 BAT Attention Map Visualization Figure 2 Performance comparison chart of BAT with dynamic templates and the original method on the LasHeR dataset; Figure 3 : Schematic diagram of the tracking method of the present invention; Figure 4 Radar charts of accuracy and success rate for 19 challenging scenarios on the LasHeR dataset; Figure 5 : A schematic diagram of the network model of this invention; Figure 6 : Schematic diagram of the appearance information aggregation module of this invention; Figure 7 : ab_bikeoccluded The selection of the tracking region in the first frame of the sequence; Figure 8 Partial tracking results; Detailed Implementation
[0014] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: like Figure 1-8 As shown, To evaluate model performance and verify its general applicability, this invention compares recent high-performance multimodal trackers on multiple multimodal benchmark datasets, including three main categories: RGB-T tracking, RGB-E tracking, and RGB-D tracking.
[0015] (1) Tracking performance on the RGB-T tracking dataset Table 1 shows the test results of this invention and recent high-performance multi-mode trackers on the LasHeR and RGBT234 datasets. On LasHeR, the method of this invention achieved scores of 75.8 and 60.3 on PR and SR, respectively, demonstrating highly competitive results compared to other trackers. On the RGBT234 dataset, this invention also achieved scores of 89.4 and 66.3 on the two core metrics of MPR and MSR, proving the effectiveness and versatility of the method.
[0016]
[0017] Table 1 shows the comparison results between the current state-of-the-art RGB-T tracker and the LasHer and RGBT234 datasets.
[0018] (2) Tracking performance on the RGB-D tracking dataset; In the RGB-D domain, the method of this invention was compared with recent high-performance trackers on DepthTrack and VOT-RGBD2022, and the results are shown in Table 2. Benefiting from its long-range temporal modeling capabilities, the method of this invention achieved an EAO score of 76.3, an Acc score of 82.2, and a Rob score of 92.3 on the VOT-RGBD dataset. On the DepthTrack dataset, the method of this invention achieved scores of 62.6, 62.7, and 62.6 on the F-score, Re, and Pr metrics, respectively. The scores achieved by the method of this invention on these two datasets surpass all compared methods, verifying the effectiveness and robustness of this invention in the RGB-D tracking domain.
[0019]
[0020] Table 2 shows the comparison results with the current state-of-the-art RGB-D trackers on the VOT-RGBD22 and DepthTrack datasets.
[0021] (3) Tracking performance on the RGB-E tracking dataset; In RGB-E tracking, this invention was compared with recent high-performance RGB-E trackers on the VisEvent dataset. The results are shown in Table 3. The method of this invention achieved a PR score of 78.3 and an SR score of 61.9 on the VisEvent dataset, surpassing all the compared methods.
[0022]
[0023] Table 3 shows the comparison results with the current state-of-the-art RGB-E trackers on the VisEvent dataset.
[0024] In summary, the method proposed in this invention has been validated in experiments across multiple multimodal tracking fields, demonstrating its excellent performance and versatility. It also proves the potential for future applications of this invention in multiple multimodal target tracking fields.
[0025] (4) Tracking performance under various complex environments; To comprehensively analyze the performance of this invention across various scenarios, a full range of attribute evaluations and visualizations were performed on the LasHeR dataset. In nineteen typical challenge scenarios within the LasHeR dataset, the accuracy and success rate of the model across multiple attribute dimensions were systematically evaluated. Experimental results demonstrate significant advantages; compared to current state-of-the-art methods such as BAT, SDSTrack, ViPT, and UnTrack, the method of this invention achieves superior performance in most attribute evaluations.
[0026] Specifically, in lighting-related challenges (LI, HI, AIV), low illumination (LI) easily leads to loss of RGB modal details, high illumination (HI) easily causes feature overexposure, and abrupt changes in illumination (AIV) can cause sudden and fragmented changes in the appearance of the target. This invention integrates high-confidence target features selected under different lighting conditions by aggregating appearance information, avoiding feature deviations caused by extreme lighting in a single frame, and ultimately achieving a stable score improvement in lighting-related attributes.
[0027] To address the challenges posed by motion and morphological disturbances (MB, DEF, SV), this invention leverages Mamba's efficient temporal modeling capabilities to achieve continuous feature judgment based on historically reliable appearances. In scenarios involving environmental and modal disturbances (BC, CM, TC), the appearance aggregation mechanism proactively filters out historical features free from background interference, with stable perspectives, and modal alignment, constructing a clean target feature library. Then, the contextual associations of these features are further aggregated through the Mamba temporal modeling layer, strengthening the discriminative power between the target and background, mitigating information conflicts caused by modal misalignment, and enabling the model to stably lock onto the target even under complex environments and modal disturbances.
[0028] In summary, the core of these performance improvements lies in the appearance information aggregation mechanism of this invention. It compensates for the feature defects of a single frame by aggregating reliable information from multiple frames, captures the target change patterns by using Mamba temporal modeling, and ensures feature matching accuracy by relying on attention alignment. The three work together to construct a robust multimodal feature representation system, which effectively alleviates the feature degradation problem under various complex challenges and provides a reliable guarantee for high-precision multimodal target tracking.
[0029] According to the technical solution of this invention, the LasHeR dataset is selected. ab_bikeoccluded Tracking video sequences presents two challenging properties: targets disappearing over long periods of time and scale variations.
[0030] Step 1: Input the first frame of the video; A video sequence contains several frames, each frame having an RGB modal image (visible light image) and an X modal image (an image in addition to the RGB modal), and containing at least one target. The target's location in the first frame is known, while the target's location in each of the remaining frames is unknown. ab_bikeoccluded Taking a video sequence as an example, it contains 294 frames of images, with the top left corner of each frame in the video frame sequence being the origin of the coordinate system. The width and height are respectively and .
[0031] Input the first frame image and automatically or manually mark the rectangular area of the tracked target. This refers to the selected tracking target. Here, (550, 341) represents the coordinates of the top-left corner of the rectangular region, and 50, 118 represent the width and height of the rectangular region, respectively. Let... Indicates the current frame number, the first frame number. The target selected in the first frame is also called the tracking result of the current frame. .
[0032] Step 2: Generate the target initial template token sequence; The rectangular region of the selected target in the first frame is magnified, and the image size corresponding to the magnified region is distorted. , as the target initial template image for both modalities and Where the height and width are equal, that is 3 indicates the number of channels in the image, superscript This represents the initial template.
[0033] right and The image is divided into blocks, each with a pixel resolution of [missing value]. It can form a sequence of template image blocks with two modalities. and , where 64 represents the number of target image patches in a modality and 768 represents the token embedding dimension. and After linear projection and add position embedding and It can generate initial template token sequences for two modalities. and ,Right now and .
[0034] Step 3: Initialize the dynamic template token sequence; make and These represent the dynamic template token sequences for the two modalities, respectively. For the first frame, i.e. In this case, the dynamic template token sequence is initialized with the target initial template token sequence, i.e. , .
[0035] Step four generates a spatiotemporal context token sequence; Because RGB modal data and X modal data have different characteristics, the spatiotemporal context token sequences for the two modalities are generated independently. Based on the initial template token sequence of the RGB modality... and dynamic template token sequence It can generate spatiotemporal context token sequences of RGB modality. Based on the initial template token sequence of the RGB modality and dynamic template token sequence It can generate spatiotemporal context token sequences for mode X. The generation of spatiotemporal context token sequences includes two sub-steps: context alignment and fusion, and context information aggregation. The generation methods for the spatiotemporal context token sequences are the same for both modalities; the following explanation uses the RGB modality as an example.
[0036] 4.1 Context alignment and blending; To alleviate problems such as feature drift and forgetting in long sequence tracking tasks, this invention uses an initial template token sequence based on a cross-attention method. For dynamic template token sequences The process of performing context-aligned fusion can be represented as follows: ; in and These represent the layer normalization operation and the activation function, respectively. , and For the learnable parameters corresponding to the query, key, and value, This is the scaling factor, which is 8 in this case. This represents the target template token sequence after alignment and fusion.
[0037] 4.2 Contextual information aggregation; Will Sending it to the Mamba layer enables the aggregation of context information, and the specific process is as follows: (1) To RMS normalization was performed to obtain , can be represented as: ; (2) After linear projection operation and Silu activation function Linear processing results can be obtained. Where 1024 represents the token embedding dimension after linear projection dimension expansion, which can be expressed as: ; (3) After linear projection operation Positive features can be obtained. Further reverse operation Inverse features can be obtained. , can be represented as: ; ; and After each forward convolution operation and backward convolution operation Silu activation function And it is sent to the state space model. Extracting and updating context information can be represented as follows: ; ; in , This represents the forward and backward hidden states of Mamba, where 64 is the sequence length, 1024 is the internal feature dimension of the hidden state, and 16 is the state dimension of Mamba. , Represents the updated hidden state, state-space model It is a key part of the Mamba layer, which models the dynamics of sequential data through hidden states. , These are the output features of the state-space model.
[0038] (4) Process the results using linear methods Output features of the state-space model and Through dot product operation Perform a weighted summation, and then perform an element-wise addition operation on the weighted result. get , can be represented as: ; The linear projection layer can reduce The feature dimensions, and then with By performing residual joins, the output of the Mamba layer can be obtained. This output aggregates the context information stored in the hidden state space of the Mamba layer, and can be represented as: ; This is the generated RGB modal spatiotemporal context token sequence.
[0039] Step 5: Input the next frame and generate the search area token sequence; Input a new video frame as the current frame, and increment the current frame number by 1. The rectangular region (550, 341, 50, 118) from the previous frame's tracking result is enlarged, and the image size corresponding to the enlarged region is distorted to 256×256, serving as the search region image for both modalities. and ,in This represents the height and width of the image in the search region; 3 represents the number of channels in the image; the symbol... Indicates the search area.
[0040] right and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and , where 256 represents the number of image blocks in a modal search region, and 768 represents the token embedding dimension, which is the same as the template image block embedding dimension. and After linear projection and add position embedding and It can generate two modal search region token sequences. and ,Right now and .
[0041] Step Six: Feature Extraction and Fusion; The target initial template token sequence of the two modalities and Dynamic template token sequence and Spatiotemporal context token sequence and and search area token sequence and Connect them to form a token sequence , can be represented as ; Will Features are extracted and fused using a backbone network, which is a dual-stream Visual Transformer (ViT) network with embedded bidirectional adapters. Each modality uses a pre-trained ViT network with 12 encoder layers. Bidirectional adapters are embedded between the same encoder layer of different modalities to achieve feature cues between modalities. The template token sequence, dynamic template token sequence, spatiotemporal context token sequence, and search region token sequence in the backbone network undergo full attention interaction. Simultaneously, modal cue interaction is also achieved between different modalities through bidirectional adapters. Finally, the backbone network outputs the features. , can be represented as: ; in , , , These represent the target initial template feature sequence, target dynamic template feature sequence, target spatiotemporal context feature sequence, and search region feature sequence, respectively, output by the RGB modality. , , , These represent the target initial template feature sequence, target dynamic template feature sequence, target spatiotemporal context feature sequence, and search region feature sequence, respectively, for output in a mode other than RGB.
[0042] Step Nine: Determine the tracking results; Fusing the feature sequences of the two modalities of the search region output by the backbone network yields the fused search region feature information. Taking the element-wise addition fusion method as an example, the search region feature information... It can be represented as: ; Using search area feature information as the basis for target localization is a standard processing method in the target tracking and localization stage. This mainly includes reinterpreting the search area feature information into a two-dimensional feature map, and then... Conv-BN- ReLU The tracking model uses a multi-layered, fully convolutional network to determine the target location based on classification score maps and local offsets. The total training loss is: ; Among them, classification loss use Weighed Focal Loss Classification is performed, and regression loss is adopted. L1 loss and Generalized IoU loss Perform bounding box regression. and They are respectively Generalized IoU Loss and L1 The regularization parameter corresponding to the loss. The target localization result is used as the tracking result for the current frame. .
[0043] Step 8: Determine the last frame; If the current frame is the last frame, the tracking ends; otherwise, continue with the subsequent steps.
[0044] Step 9: Determine whether the confidence level of the tracking result has reached the threshold; The confidence score of the target center location obtained from the classification score map in the current frame tracking process is 0.76, which is compared with the confidence threshold of 0.65. Since the confidence score corresponding to the target center location has reached the confidence threshold, subsequent steps are executed.
[0045] Step 10: Update the dynamic template token sequence; Tracking results of the current frame Image size scaled to the corresponding rectangular area , serving as dynamic template images for two modalities and Where the height and width are equal, that is , Indicates the number of image channels. and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and 64 represents the number of target image patches in a modality, and 768 represents the token embedding dimension. and After linear projection and add position embedding and It can generate two modalities of target dynamic template token sequences. and ,Right now and After the dynamic template token sequence is updated, proceed to step four.
[0046] Finally, the examples of this invention were trained and tracked on an NVIDIA RTX 4090 GPU, and some tracking results are shown below. Figure 7As shown in the figure. In the figure, "RGB-T", "RGB-D", and "RGB-E" on the left represent three multimodal tracking tasks, respectively. The light blue label in the upper left corner is the video frame number, the green box indicates the real position of the video marker, the blue box indicates the tracking result of this invention, and the other colored boxes indicate other tracking models. See the figure caption for details.
[0047] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.
Claims
1. A multimodal tracking method based on apparent temporal modeling, characterized in that, Its multimodal tracking framework based on visual Transformer: First, the dynamic template token is updated based on the confidence threshold update mechanism to obtain the latest reference information of the target; then, the initial template token containing the initial reference information and the dynamic template token containing the latest reference information are used by the appearance information aggregation module to continuously compress the target appearance information into the hidden state space using the Mamba selection scanning mechanism, thereby generating the spatiotemporal context token. Finally, the initial template token, dynamic template token, and spatiotemporal context token form the target appearance temporal information, which alleviates the problems of feature drift and forgetting in long-sequence multimodal tracking tasks, thereby promoting the robustness of multimodal tracking.
2. The method according to claim 1, characterized in that, The steps of this method are as follows: Step 1: Input the first frame of the video; Step 2: Generate the initial template token sequence; Step 3: Initialize the dynamic template token sequence; Step four generates a spatiotemporal context token sequence; Because RGB modal data and X modal data have different characteristics, the spatiotemporal context token sequences of the two modalities are generated independently; Initial template token sequence based on RGB modality and dynamic template token sequence It can generate spatiotemporal context token sequences of RGB modality. Based on the initial template token sequence of the RGB modality and dynamic template token sequence It can generate spatiotemporal context token sequences for mode X. The appearance information aggregation module can generate spatiotemporal context token sequences, which include two sub-steps: context alignment fusion and context information aggregation. The generation methods for the spatiotemporal context token sequences of the two modalities are the same. The following only uses the RGB modality as an example to illustrate its generation method. 1) Context-aligned blending; To alleviate problems such as feature drift and forgetting in long sequence tracking tasks, this invention uses an initial template token sequence based on a cross-attention method. For dynamic template token sequences The process of performing context-aligned fusion can be represented as follows: ； in and These represent the layer normalization operation and the activation function, respectively. , and For the learnable parameters corresponding to the query, key, and value, Scaling factor for The token embedding dimension, This represents the target template token sequence after alignment and fusion. 2) Contextual information aggregation; Will Sending it to the Mamba layer enables the aggregation of context information, and the specific process is as follows: (1) To RMS normalization was performed to obtain , can be represented as: ； (2) After linear projection operation and Silu activation function Linear processing results can be obtained. ,in The token embedding dimension after linear projection dimension expansion can be represented as: ； (3) After linear projection operation Positive features can be obtained. Further reverse operation Inverse features can be obtained. , can be represented as: ； ； and After each forward convolution operation and backward convolution operation Silu activation function And it is sent to the state space model. Extracting and updating context information can be represented as follows: ； ； in , This represents the forward and backward hidden states in Mamba. For sequence length, The internal feature dimension of the hidden state. For the state dimension of Mamba, , Represents the updated hidden state; state-space model It is a key part of the Mamba layer, which models the dynamics of sequential data through hidden states. , These are the output features of the state-space model; (4) Process the results using linear methods Output features of the state-space model and Through dot product operation Perform a weighted summation, and then perform an element-wise addition operation on the weighted result. get , can be represented as: ； The linear projection layer can reduce The feature dimensions, and then with By performing residual joins, the output of the Mamba layer can be obtained. This output aggregates the context information stored in the hidden state space of the Mamba layer, and can be represented as: ； This refers to the generated RGB modal spatiotemporal context token sequence; Step 5: Input the next frame and generate the search area token sequence; Input a new video frame as the current frame, and increment the current frame number by 1. ; Tracking results from the previous frame Enlarged rectangular area b 2 The magnified area corresponds to a change in image size. The search region image serves as both modalities. and ,in and These represent the height and width of the search region image, respectively. , The number of channels in an image, symbol Indicates the search area; right and The image is divided into blocks, each with a pixel resolution of [missing value]. This can form a sequence of search region image patches in two modalities. and ,in This indicates the number of image patches in a modal search region. This indicates the token embedding dimension, which is the same as the template image patch embedding dimension. and After linear projection and add position embedding and It can generate two modal search region token sequences. and ,Right now and ; Step Six: Feature Extraction and Fusion; The target initial template token sequence of the two modalities and Dynamic template token sequence and Spatiotemporal context token sequence and and search area token sequence and Connect them to form a token sequence , can be represented as: ； Will By feeding the data into the backbone network for feature extraction and fusion, the output features of the backbone network can be obtained. , can be represented as: ； in , , , These represent the initial template feature sequence, dynamic template feature sequence, spatiotemporal context feature sequence, and search region feature sequence of the RGB modal output, respectively. , , , These represent the initial template feature sequence, dynamic template feature sequence, spatiotemporal context feature sequence, and search region feature sequence of the X modality output, respectively. The backbone network of this invention can adopt a single-branch ViT network, a multimodal branch ViT network, or an improved network thereof; wherein a single-branch ViT network refers to a network that... The data is fed into a single ViT network for unified feature extraction and fusion. Multimodal branch ViT network refers to feeding different modal token sequences into different modal ViT networks for feature extraction and fusion. Improved network refers to a network that improves upon the first two types of networks. For example, based on the multimodal branch ViT network, a bidirectional adapter can be embedded between encoders of the same layer in different modalities to achieve feature cues between modalities. Step 7: Determine the tracking results; By fusing the feature sequences of the two modal search regions output by the backbone network, we can obtain the feature information of the fused search region. Taking the element-wise addition fusion method as an example, the search region feature information It can be represented as: ； Using search area feature information as the basis for target localization is a standard processing method in the target tracking and localization stage. This mainly includes reinterpreting the search area feature information into a two-dimensional feature map, and then... Conv-BN-ReLU The tracking model uses a multi-layered, fully convolutional network to determine the target location based on classification score maps and local offsets; the total training loss is: ； Among them, classification loss use Weighed Focal Loss Classification is performed, and regression loss is adopted. L1 loss and Generalized IoU loss Perform bounding box regression. and They are respectively Generalized IoU Loss and L1 The regularization parameter corresponding to the loss; the target localization result is used as the tracking result of the current frame. ; Step 8: Determine if it is the last frame; If the current frame is the last frame, the tracking ends; otherwise, continue with the subsequent steps. Step 9: Determine whether the confidence level of the tracking result has reached the threshold; The confidence level corresponding to the target center position is obtained from the classification score map in the current frame tracking process and compared with the confidence level threshold. If the confidence level corresponding to the target center position does not reach the confidence level threshold, proceed to step five; otherwise, continue to execute the subsequent steps. Step 10: Update the dynamic template token sequence; Tracking results of the current frame Image magnification of corresponding rectangular region Times, deformation Size, serving as a dynamic template image for both modalities. and Where the height and width are equal, that is , Represents the number of image channels; for and The image is divided into blocks, each with a pixel resolution of [missing value]. To form a dynamic template image block sequence and After linear projection Add position embedding and It can generate dynamic template token sequences with two modalities. and ,Right now and After the dynamic template token sequence is updated, proceed to step four.
3. The method according to claim 2, characterized in that: Step one specifically involves: A video sequence contains several frames, each frame containing an RGB modal image (visible light image) and an X modal image (an image in addition to the RGB modal), and each frame contains at least one target. The target's position in the first frame is known, while the target's position in each of the remaining frames is unknown. The number of video frames is a positive integer greater than 1. The top-left corner of each frame in the video frame sequence is the origin. The width and height are respectively and ; Input the first frame image and automatically or manually mark the rectangular area of the tracked target. That is, the selected tracking target; among which, This represents the coordinates of the top-left corner of the rectangular region. Let the width and height of the rectangular region be respectively; Indicates the current frame number, the first frame number. The target selected in the first frame is also called the tracking result of the current frame. = .
4. The method according to claim 3, characterized in that, Step two specifically involves: enlarging the rectangular region of the selected target in the first frame. The magnified area corresponds to a change in image size. , as the target initial template image for both modalities and Where the height and width are equal, that is , Indicates the number of channels in an image, indicated by a superscript. Indicates the initial template; right and The image is divided into blocks, each with a pixel resolution of [missing value]. It can form a sequence of template image blocks with two modalities. and ,in This indicates the number of template image patches for a given modality. This represents the token embedding dimension corresponding to an image patch, indicated by the superscript. Represents an image block; and After linear projection and add position embedding and It can generate initial template token sequences for two modalities. and ,Right now and superscript Indicates the target template. This represents the initial template.
5. The method according to claim 4, characterized in that, Step three specifically involves: Let... and These represent the dynamic template token sequences for two modalities, respectively, with superscripts indicating the sequence. This indicates a dynamic template; for the first frame... In this case, the dynamic template token sequence is initialized with the target initial template token sequence, i.e. , .