Lightweight multi-modal target tracking method
By designing a lightweight multimodal target tracking method, combining D-MAE-Tiny pre-training algorithm and multi-expert model structure, the problem of insufficient multimodal tracking performance on resource-limited devices is solved, and real-time multi-dimensional on CPU devices is achieved Modal tracking improves robustness and efficiency.
Patent Information
- Application Number
- CN202411523201.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-10-30
AI Technical Summary
The prior art is difficult to achieve efficient and lightweight multimodal target tracking on devices with limited resources, especially when processing RGB-infrared, RGB-depth, and RGB-event data, which lacks concise and unified model design and efficient feature fusion. Methods, resulting in limited robust tracking performance.
A lightweight multimodal target tracking method was designed, combined with D-MAE-Tiny pre-training algorithm, and adopting a multi-expert model structure and a multi-modal feature spatial modulation module, which was realized by sharing self-attention and not sharing a multi-layer perception machine. Modal feature fusion, and the timing prompt word module is used to optimize the tracking results, and the teacher model distillation loss guidance training is used to achieve efficient and robust multimodal tracking.
Real-time multi-modal tracking on CPU devices and end-side devices is achieved, with a good balance of performance and speed, and can process RGB-infrared, RGB-depth, and RGB-event data at the same time, improving robustness and efficiency.
Smart Images

Figure CN119048874A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision and target tracking, and relates to a lightweight pre-training algorithm D-MAE-Tiny and a multimodal fusion algorithm; specifically, it is a lightweight multimodal target tracking method. Background Art
[0002] Given the target box annotation of the target to be tracked in the initial frame of the video, the target tracking algorithm will achieve continuous tracking of the specified target in subsequent frames by modeling the spatiotemporal information of the target and background. RGB video-based target tracking has developed rapidly in recent years, but RGB images may have difficulty in completing tracking tasks in some complex scenes, such as extreme lighting conditions and occlusion conditions, which limits its application in related fields that require high-robust tracking. In multiple visual perception fields such as segmentation, detection, and image restoration, multimodal fusion is receiving more and more attention. In the field of tracking, multimodal fusion can also enable the tracking algorithm to obtain more valuable information from auxiliary modalities other than RGB, achieve more comprehensive and complementary information extraction and fusion, and thus help improve the robustness of target tracking.
[0003] Recently, many works have applied the cue word learning technology to the visual field, and this technology has also shown its effectiveness in the field of multimodal tracking. Through this process design, the multimodal tracking algorithm model can inherit the pre-trained knowledge of the RGB tracking algorithm model, effectively making up for the shortcomings caused by the lack of multimodal tracking data sets. Multimodal tracking algorithms such as ViPT, SDSTrack, and Un-Track all use efficient parameter fine-tuning technology represented by cue word learning technology. This technology can effectively reduce the number of parameters that need to be adjusted during training, and achieve performance similar to or even better than full parameter fine-tuning with a small number of adjustment parameters. These works all focus on efficient training, but in practice, achieving efficient tracking reasoning on resource-limited devices, such as CPU devices and end-side GPU devices, also faces great challenges. Although some works have studied lightweight target tracking algorithms for pure RGB videos, such as LighTrack, HiT, and PromptVT, few works focus on efficient and lightweight reasoning for multimodal target tracking. At the same time, since auxiliary modalities may include multiple modalities such as infrared images, depth maps, event data, etc., there is also a lack of relevant research work on how to implement a lightweight multimodal tracking algorithm that adapts to multiple auxiliary modalities with fewer parameters.
[0004] Therefore, how to design a concise, unified, lightweight multimodal target tracking model, which can be combined with a lightweight backbone network to achieve fast and robust multimodal tracking on resource-limited devices, is one of the challenging problems that need to be solved in current tracking tasks. Summary of the invention
[0005] The present invention aims to provide a lightweight multimodal target tracking method, which embeds the structural design ideas of the multi-expert model, the multimodal feature space modulation module, and the time-series prompt word module into a concise and unified multimodal target tracking model framework, and uses the multimodal tracking model with excellent performance as the teacher model for distillation guidance to achieve an efficient and high-performance multimodal tracking algorithm. The present invention can simultaneously process multimodal tracking tasks of RGB-infrared, RGB-depth, and RGB-event data. Compared with the existing high-performance multimodal tracking algorithms, it can achieve real-time speed on CPU devices and end-side devices with less performance loss, achieving a better balance between performance and speed.
[0006] The technical solution of the present invention:
[0007] A lightweight multimodal target tracking method, the steps are as follows:
[0008] Step 1: Select the weights of the lightweight pre-trained algorithm D-MAE-Tiny (A Closer Look at Self-Supervised Lightweight Vision Transformers) as the initialization weights, use the training mode and tracking prediction network of the target tracking algorithm OSTrack, and use RGB modality target tracking data to fine-tune all parameters to obtain the RGB tracking algorithm model weights.
[0009] Step 2: Using the RGB tracking algorithm model weight obtained in step 1 as the initialization weight, select the single-stream tracking model structure in the target tracking algorithm OSTrack and its tracking prediction network to construct a multimodal target tracking model; the multimodal target tracking model includes an image block coding module, a multimodal feature space modulation module and a multi-expert coding layer module, and the multi-expert coding layer module includes a shared self-attention module and a non-shared multi-layer perceptron; in the multimodal target tracking model, image block coding modules with the same structure but non-shared weights are designed for different modalities, and after the image block coding module, a multimodal feature space modulation module is inserted to perform adaptive feature fusion on the RGB modality and the auxiliary modality; then, the feature fusion information of different auxiliary modalities and RGB modalities is processed respectively through the multi-expert coding layer module; in the multi-expert coding layer module, different auxiliary modalities share the self-attention module weights, and different auxiliary modalities do not share the multi-layer perceptron weights; the auxiliary modalities include infrared modalities, depth modalities and event modalities, and the auxiliary modalities and RGB modalities constitute different modalities;
[0010] (1) Multimodal feature space modulation module:
[0011] In the multimodal target tracking model, image block encoding modules with the same structure but not sharing weights are designed for different modalities to obtain image block encoding features respectively:
[0012]
[0013]
[0014] Among them, the superscript X represents a certain auxiliary mode. Represents the image patch encoding features in RGB mode, The image patch encoding features representing the auxiliary modality, The image patch encoding features representing the search area in RGB mode, Image patch encoding features representing the template region in RGB modality; The image patch encoding features representing the search area of the auxiliary modality, Image patches encoding features representing the template region of the auxiliary modality.
[0015] For the search area features, the multimodal feature space modulation module first encodes the image block features of the search area of the RGB mode. Image block encoding features of the search area and auxiliary modality Splice according to the channel dimension to obtain , and then use two sets of multi-layer perceptron networks and Acting on the splicing feature , generate corresponding spatial weights for the RGB modality and auxiliary modality image block encoding features respectively , to achieve spatial modulation of the coding features:
[0016]
[0017]
[0018] The template region features are processed in the same way as the search region features, but two sets of multi-layer perceptron networks are responsible for predicting the spatial modulation weights. , Two sets of multi-layer perceptrons with search area features , Weights are not shared:
[0019]
[0020]
[0021] in, It is the image patch encoding feature of the template area in RGB mode Image block encoding features of the template area of the auxiliary modality It is obtained by splicing according to the channel dimension;
[0022] The auxiliary modality image block coding feature through adaptive modulation is finally:
[0023]
[0024] The RGB modal image block encoding features through adaptive modulation are finally:
[0025]
[0026] Finally, the adaptive modulation results are added to obtain the fused image block coding features:
[0027] ;
[0028] (2) Coding layer module with multiple experts’ ideas:
[0029] Under the premise of a shared self-attention module, a specific multi-layer perceptron is configured for each RGB modality and auxiliary modality task to form a multi-expert coding layer module. The multi-layer perceptrons in the coding layer module of the multi-expert idea include infrared multi-layer perceptron, depth multi-layer perceptron, and event multi-layer perceptron. Each RGB-X multi-modal tracking task only activates the corresponding multi-layer perceptron module to achieve simultaneous joint unified training of multiple multi-modal tracking tasks:
[0030] ,
[0031]
[0032] Among them, T stands for infrared modality, D stands for depth modality, and E stands for event modality.
[0033] At the same time, in the joint unified training process, the self-attention map distillation method in the lightweight pre-training algorithm D-MAE-Tiny is used, and the multimodal tracking algorithm model ViPT (Visual Prompt Multi-Modal Tracking) is used as the teacher model to distill the last layer of the self-attention map of the current lightweight student model, providing effective information to guide the training of the larger model. The distillation loss and the tracking loss used by the target tracking algorithm OSTrack jointly guide the training of the model:
[0034]
[0035] in, is the distillation loss, MSE is the mean square error loss function, is the last layer of self-attention map of the student model, It is the last layer self-attention map of the teacher model. is the mapping layer of a certain auxiliary mode X;
[0036] Step 3: After the structural design and model training in step 2, a lightweight temporal cue word module is used to introduce the high-level semantic information of the previous frame into the current frame. Specifically:
[0037] The lightweight temporal cue word module is composed of a two-layer multi-layer perceptron module. It takes the last layer of features expressed by the joint modeling of the search area and the template area of the previous frame as input, and generates a cue word expression to add to the fused image block encoding features in step 2:
[0038] ,
[0039] Updated to:
[0040]
[0041] The right subscript t-1 represents the previous frame, and TP represents a lightweight temporal prompt word module. represents the fused image block encoding features with added temporal hints, It is a time sequence cue word expression.
[0042] In this stage of training, only the parameters of the lightweight temporal cue word module can be trained, and the parameters of the rest of the multimodal target tracking model are frozen;
[0043] Step 4: Given the initial frame and the initial frame target annotation box of the RGB modality video data and the corresponding infrared modality / depth modality / event modality video data, multimodal target tracking is performed, wherein for the encoding layer module of the multi-expert idea, only the parameters of the multi-layer perceptron of a specific auxiliary modality are activated.
[0044] Beneficial effects of the present invention:
[0045] (1) The concise and unified multimodal target tracking model proposed in this paper can simultaneously process the fusion tracking of RGB modality and multiple auxiliary modalities (infrared, depth, event, etc.). The multimodal feature space modulation module can realize early multimodal adaptive and efficient feature fusion. The temporal prompt word module can optimize the multimodal tracking results by using temporal information.
[0046] (2) The lightweight multimodal target tracking method proposed in the present invention can achieve a good balance between performance and speed, and can achieve real-time speed on CPU devices and end-side devices such as NVIDIA Jetson AGX Orin with less performance loss. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 This is a framework diagram of the multimodal target tracking model.
[0048] Figure 2 Schematic diagram of the structure of the multimodal feature space modulation module.
[0049] Figure 3 Schematic diagram of the coding layer module structure of multi-expert ideas.
[0050] Figure 4 Schematic diagram of the lightweight timing prompt word module structure. DETAILED DESCRIPTION
[0051] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.
[0052] A lightweight multimodal target tracking method, the steps are as follows:
[0053] Step 1: Select the weights of the lightweight pre-trained algorithm D-MAE-Tiny (A Closer Look at Self-Supervised Lightweight Vision Transformers) as the initialization weights, use the training mode and tracking prediction network of the target tracking algorithm OSTrack, and use RGB modality target tracking data to fine-tune all parameters to obtain the RGB tracking algorithm model weights.
[0054] Step 2: Using the RGB tracking algorithm model weight obtained in step 1 as the initialization weight, select the single-stream tracking model structure in the target tracking algorithm OSTrack and its tracking prediction network to construct a multimodal target tracking model; the multimodal target tracking model includes an image block coding module, a multimodal feature space modulation module and a multi-expert coding layer module, and the multi-expert coding layer module includes a shared self-attention module and a non-shared multi-layer perceptron; in the multimodal target tracking model, image block coding modules with the same structure but non-shared weights are designed for different modalities, and after the image block coding module, a multimodal feature space modulation module is inserted to perform adaptive feature fusion on the RGB modality and the auxiliary modality; then, the feature fusion information of different auxiliary modalities and RGB modalities is processed respectively through the multi-expert coding layer module; in the multi-expert coding layer module, different auxiliary modalities share the self-attention module weights, and different auxiliary modalities do not share the multi-layer perceptron weights; the auxiliary modalities include infrared modalities, depth modalities and event modalities, and the auxiliary modalities and RGB modalities constitute different modalities;
[0055] (1) Multimodal feature space modulation module:
[0056] In the single-stream tracking model structure of the conventional target tracking algorithm OSTrack, the features of the template area image and the search area image are obtained after being divided and encoded. After concatenation, the encoded features are obtained , C is the number of channels, is the number of image blocks in the search area, is the number of image blocks in the template area, the right subscript x refers to the search area, and the right subscript z refers to the template area. The encoded features will be fed into multiple Transformer encoding layers with self-attention modules and multi-layer perceptron modules as the core:
[0057] ,
[0058]
[0059] in, is the number of coding layers, It represents the result of Transformer encoding after layer l. represents the self-attention module, MLP represents the multi-layer perceptron module, Representation layer normalization.
[0060] In the multimodal target tracking model of the present invention, image block coding modules with the same structure but not sharing weights are designed for different modalities, and image block coding features are obtained respectively:
[0061]
[0062]
[0063] Among them, the superscript X represents a certain auxiliary mode. Represents the image patch encoding features in RGB mode, The image patch encoding features representing the auxiliary modality, The image patch encoding features representing the search area in RGB mode, Image patch encoding features representing the template region in RGB modality; The image patch encoding features representing the search area of the auxiliary modality, Image patches encoding features representing the template region of the auxiliary modality.
[0064] The multimodal feature space modulation module is used to explore the discrimination advantages of different modes in different scenarios and realize effective multimodal adaptive fusion. For the search area features, the multimodal feature space modulation module first encodes the image block features of the search area of the RGB mode. Image block encoding features of the search area and auxiliary modality Splice according to the channel dimension to obtain , and then use two sets of multi-layer perceptron networks and Acting on the splicing feature , generate corresponding spatial weights for the RGB modality and auxiliary modality image block encoding features respectively , to achieve spatial modulation of the coding features:
[0065]
[0066]
[0067] The template region features are processed in the same way as the search region features, but two sets of multi-layer perceptron networks are responsible for predicting the spatial modulation weights. , Two sets of multi-layer perceptrons with search area features , Weights are not shared:
[0068]
[0069]
[0070] in, It is the image patch encoding feature of the template area in RGB mode Image block encoding features of the template area of the auxiliary modality It is obtained by splicing according to the channel dimension;
[0071] The auxiliary modality image block coding feature through adaptive modulation is finally:
[0072]
[0073] The RGB modal image block encoding features through adaptive modulation are finally:
[0074]
[0075] Finally, the adaptive modulation results are added to obtain the fused image block coding features:
[0076]
[0077] (2) Coding layer module with multiple experts’ ideas:
[0078] In order to enable the model to handle the fusion tracking of multiple auxiliary modalities and RGB modalities at the same time, the multi-expert idea is introduced in the model structure design. Under the premise of a shared self-attention module, a specific multi-layer perceptron is configured for each RGB modality and auxiliary modality task to form a multi-expert idea encoding layer module. The multi-layer perceptrons in the encoding layer module of the multi-expert idea include infrared multi-layer perceptron, deep multi-layer perceptron, and event multi-layer perceptron. Each RGB-X multi-modal tracking task only activates the corresponding multi-layer perceptron module to achieve simultaneous joint and unified training of multiple multi-modal tracking tasks:
[0079] ,
[0080]
[0081] Among them, T stands for infrared modality, D stands for depth modality, and E stands for event modality.
[0082] At the same time, in the joint unified training process, the self-attention map distillation method in the lightweight pre-training algorithm D-MAE-Tiny is used, and the multimodal tracking algorithm model ViPT (Visual Prompt Multi-Modal Tracking) is used as the teacher model to distill the last layer of the self-attention map of the current lightweight student model, providing effective information to guide the training of the larger model. The distillation loss and the tracking loss used by the target tracking algorithm OSTrack jointly guide the training of the model:
[0083]
[0084] in, is the distillation loss, MSE is the mean square error loss function, is the last layer of self-attention map of the student model, It is the last layer self-attention map of the teacher model. is the mapping layer of a certain auxiliary mode X;
[0085] Step 3: After the structural design and model training in step 2, in order to further optimize the results of multimodal tracking, a lightweight temporal cue word module is used to introduce the high-level semantic information of the previous frame into the current frame. Specifically:
[0086] The lightweight temporal cue word module is composed of a two-layer multi-layer perceptron module. It takes the last layer of features expressed by the joint modeling of the search area and the template area of the previous frame as input, and generates a cue word expression to add to the fused image block encoding features in step 2:
[0087] ,
[0088] Updated to:
[0089]
[0090] The right subscript t-1 represents the previous frame, and TP represents a lightweight temporal prompt word module. represents the fused image block encoding features with added temporal hints, It is a time sequence cue word expression.
[0091] In this stage of training, only the parameters of the lightweight temporal cue word module can be trained, and the parameters of the rest of the multimodal target tracking model are frozen;
[0092] Step 4: Given the initial frame and the initial frame target annotation box of the RGB modality video data and the corresponding infrared modality / depth modality / event modality video data, multimodal target tracking is performed, wherein for the encoding layer module of the multi-expert idea, only the parameters of the multi-layer perceptron of a specific auxiliary modality are activated.
[0093] In a specific embodiment:
[0094] Figure 1 The multimodal target tracking model diagram is shown in Figure 2. The overall lightweight multimodal target tracking method is divided into two training stages. In the first training stage, the multimodal target tracking model is embedded in Figure 2 The multimodal feature space modulation module shown and Figure 3 The coding layer module of the multi-expert idea shown in the figure is used to jointly train the multimodal target tracking model with three multimodal tracking datasets, namely RGB-infrared, RGB-depth, and RGB-event. The training set includes RGB-infrared data LasHeR, RGB-depth data DepthTrack and RGBD-1K, and RGB-event data VisEvent, and the dataset sampling ratio is 2:1:1:2. The optimizer uses ADAMW, and a total of 60 epochs are trained. 60,000 sample pairs are sampled in each round. The learning rate decays by 0.1 times at the 48th epoch. The overall training uses the candidate box elimination mechanism of the RGB tracking algorithm OSTrack and the tracking loss function of OSTrack. The overall loss of the multimodal target tracking model is the sum of the tracking loss and the distillation loss, and the loss weights are all 1. The hidden dimension of the multilayer perceptron network in the multimodal feature space modulation module is 192.
[0095] In the second training phase, Figure 4Except for the lightweight temporal cue word module shown in the figure, all parameter weights are frozen, the overall loss of the multimodal target tracking model remains unchanged, the training data set remains unchanged, and a total of 10 epochs are trained. The triple sampling method is adopted, and each group of samples samples two search areas and one template area. For other training parameter settings, refer to the parameter settings of the first stage training. Among them, the hidden dimension of the multilayer perceptron module of the lightweight temporal cue word module is 32.
[0096] During the entire training process, the search area is 16 times the size of the target area and scaled to 256×256; the template area is 4 times the size of the target area and scaled to 128×128, and the batch size of each GPU is 64. During the inference process, according to the given RGB modality video data and the corresponding infrared modality / depth modality / event modality data, the parameters of the multi-layer perceptron of a specific auxiliary modality in the encoding layer module of the multi-expert idea are activated to perform multimodal target tracking.
[0097] Table 1: Performance comparison on the RGB-T tracking dataset LasHeR
[0098] Tracker Success rate SR Accuracy CPU Speed GPU Speed The present invention 53.3 65.9 83.8 29.1 SDSTrack 53.1 66.5 19.4 1.5 ViPT 52.5 65.1 39.5 6.3 Un-Track 51.3 54.6 - - ProTrack 42.0 53.8 - -
[0099] As represented by the performance on the LasHeR dataset in Table 1, the present invention shows good performance on multiple RGB-T, RGB-D, and RGB-E tracking datasets. The GPU is an RTX 3090 GPU and the CPU is an Intel Core i9-10850K CPU.
Claims
1. A lightweight multimodal target tracking method, It is characterized in that Here are the steps: Step 1: Select the weights of the lightweight pre-trained algorithm D-MAE-Tiny as the initialization weights, use the training mode and tracking prediction network of the target tracking algorithm OSTrack, and use the RGB modality target tracking data to fine-tune all parameters to obtain the RGB tracking algorithm model weights; Step 2: Using the RGB tracking algorithm model weight obtained in step 1 as the initialization weight, select the single-stream tracking model structure in the target tracking algorithm OSTrack and its tracking prediction network to build a multimodal target tracking model; the multimodal target tracking model includes an image block encoding module, a multimodal feature space modulation module and a multi-expert coding layer module, and the multi-expert coding layer module includes a shared self-attention module and an unshared multi-layer perceptron; in the multimodal target tracking model, image block encoding modules with the same structure but unshared weights are designed for different modalities, and after the image block encoding module, a multimodal feature space modulation module is inserted to perform adaptive feature fusion on the RGB modality and the auxiliary modality; Then, the feature fusion information of different auxiliary modalities and RGB modalities is processed separately through the encoding layer module of multi-expert ideas; In the encoding layer module of the multi-expert approach, different auxiliary modalities share the weights of the self-attention module, and different auxiliary modalities do not share the weights of the multi-layer perceptron; the auxiliary modalities include infrared modalities, depth modalities, and event modalities, and the auxiliary modalities and RGB modalities constitute different modalities; Step 3: After the structural design and model training in step 2, a lightweight temporal cue word module is used to introduce the high-level semantic information of the previous frame into the current frame; Step 4: Given the initial frame and the initial frame target annotation box of the RGB modality video data and the corresponding infrared modality / depth modality / event modality video data, multimodal target tracking is performed, wherein for the encoding layer module of the multi-expert idea, only the parameters of the multi-layer perceptron of a specific auxiliary modality are activated.
2. A lightweight multi-modal target tracking method according to claim 1, It is characterized in that The multimodal feature space modulation module in step 2: In the multimodal target tracking model, image block encoding modules with the same structure but not sharing weights are designed for different modalities to obtain image block encoding features respectively: Among them, the superscript X represents a certain auxiliary mode. Represents the image patch encoding features in RGB mode, The image patch encoding features representing the auxiliary modality, The image patch encoding features representing the search area in RGB mode, Image patch encoding features representing the template region in RGB modality; The image patch encoding features representing the search area of the auxiliary modality, An image patch encoding feature representing a template region of an auxiliary modality; For the search area features, the multimodal feature space modulation module first encodes the image block features of the search area of the RGB mode. Image block encoding features of the search area and auxiliary modality Splice according to the channel dimension to obtain , and then use two sets of multi-layer perceptron networks and Acting on the splicing feature , generate corresponding spatial weights for the RGB modality and auxiliary modality image block encoding features respectively , to achieve spatial modulation of the coding features: The template region features are processed in the same way as the search region features, but two sets of multi-layer perceptron networks are responsible for predicting the spatial modulation weights. , Two sets of multi-layer perceptrons with search area features , Weights are not shared: in, It is the image patch encoding feature of the template area in RGB mode Image block encoding features of the template area of the auxiliary modality It is obtained by splicing according to the channel dimension; The auxiliary modality image block coding feature through adaptive modulation is finally: The RGB modal image block encoding features through adaptive modulation are finally: Finally, the adaptive modulation results are added to obtain the fused image block coding features: 。 3. A lightweight multi-modal target tracking method according to claim 2, It is characterized in that The coding layer module of the multi-expert approach in step 2: Under the premise of a shared self-attention module, a specific multi-layer perceptron is configured for each RGB modality and auxiliary modality task to form a multi-expert coding layer module. The multi-layer perceptrons in the coding layer module of the multi-expert idea include infrared multi-layer perceptron, depth multi-layer perceptron, and event multi-layer perceptron. Each RGB-X multi-modal tracking task only activates the corresponding multi-layer perceptron module to achieve simultaneous joint and unified training of multiple multi-modal tracking tasks: , Among them, T represents infrared modality, D represents depth modality, and E represents event modality; At the same time, in the joint unified training process, the self-attention map distillation method in the lightweight pre-training algorithm D-MAE-Tiny is used, and the multimodal tracking algorithm model ViPT is used as the teacher model to distill the last layer of the self-attention map of the current lightweight student model, providing effective information to guide the training of the larger model. The distillation loss and the tracking loss used by the target tracking algorithm OSTrack jointly guide the training of the model: in, is the distillation loss, MSE is the mean square error loss function, is the last layer of self-attention map of the student model, It is the last layer of self-attention map of the teacher model; is the mapping layer of a certain auxiliary mode X.
4. A lightweight multi-modal target tracking method according to claim 3, It is characterized in that Step 3 is as follows: The lightweight temporal cue word module consists of a two-layer multi-layer perceptron module, which takes the last layer of features expressed by the joint modeling of the search area and the template area of the previous frame as input, and generates a cue word expression to add to the fused image block encoding features in step 2: , Updated to: The right subscript t-1 represents the previous frame, and TP represents a lightweight temporal prompt word module. represents the fused image block encoding features with added temporal hints, It is expressed as a time sequence cue word; In this stage of training, only the parameters of the lightweight temporal cue word module can be trained, and the parameters of the rest of the multimodal target tracking model are frozen.
Citation Information
Patent Citations
Multi-modal visual target tracking method based on self-distillation symmetric adapter
CN117710414A
Universal multi-modal target tracking model based on adaptive perception, training method and application
CN118674945A
System and method for tracking objects by fusing results of multiple sensing modalities
US6499025B1
Cited By
Multi-modal unified target tracking method based on modal unified representation
CN119672071A
Multi-modal target tracking method based on state space model
CN120070504A
Target tracking method based on computer vision
CN120782820A
Object tracking processing method and apparatus, device, and medium
WO2026138395A1
