A lightweight multimodal target tracking method

Abstract

The present invention provides a lightweight multimodal target tracking method, which belongs to the field of computer vision and target tracking. The structural design ideas of the multi-expert model, the multimodal feature space modulation module, and the time-series prompt word module are embedded in a concise and unified multimodal target tracking model framework, and the multimodal tracking model with excellent performance is used as the teacher model for distillation guidance to achieve an efficient and high-performance multimodal tracking algorithm. The present invention can simultaneously process multimodal tracking tasks of RGB-infrared, RGB-depth, and RGB-event data. Compared with the existing high-performance multimodal tracking algorithms, it can achieve real-time speed on CPU devices and end-side devices with less performance loss, achieving a better balance between performance and speed.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and target tracking, and relates to a lightweight pre-training algorithm D-MAE-Tiny and a multimodal fusion algorithm; specifically, it is a lightweight multimodal target tracking method. Background Art

[0002] Given the target box annotation of the target to be tracked in the initial frame of a given video, the target tracking algorithm will achieve continuous tracking of the specified target in subsequent frames by temporal and spatial modeling of the target and background information. Target tracking based on RGB video has developed rapidly in recent years, but RGB images may have difficulty completing tracking tasks in some complex scenes, such as extreme lighting conditions and occlusions, which limits its application in related fields that require highly robust tracking. In multiple visual perception fields such as segmentation, detection, and image restoration, multimodal fusion is receiving increasing attention. In the field of tracking, multimodal fusion can also enable tracking algorithms to obtain more valuable information from auxiliary modalities other than RGB, achieving more comprehensive and complementary information extraction and fusion, thereby helping to improve the robustness of target tracking.

[0003] Recently, many works have applied cue word learning technology to the vision field, and this technology has also demonstrated its effectiveness in multimodal tracking. This process design allows multimodal tracking algorithm models to inherit pre-trained knowledge from RGB tracking algorithm models, effectively addressing the shortcomings caused by insufficient multimodal tracking datasets. Multimodal tracking algorithms such as ViPT, SDSTrack, and Un-Track all utilize efficient parameter fine-tuning techniques, exemplified by cue word learning. This technology effectively reduces the number of parameters required for training, achieving performance similar to or even better than full parameter fine-tuning with a small number of parameters. While these works focus on efficient training, achieving efficient tracking inference on resource-constrained devices, such as CPUs and on-device GPUs, presents significant challenges. Although some work has studied lightweight object tracking algorithms for pure RGB video, such as LightTrack, HiT, and PromptVT, relatively little research has focused on efficient and lightweight inference for multimodal object tracking. At the same time, since auxiliary modalities may include multiple modalities such as infrared images, depth maps, event data, etc., there is a lack of relevant research work on how to implement a lightweight multimodal tracking algorithm that adapts to multiple auxiliary modalities with fewer parameters.

[0004] Therefore, how to design a concise and unified lightweight multimodal target tracking model, so as to cooperate with the lightweight backbone network to achieve fast and robust multimodal tracking on resource-limited devices, is one of the challenging problems that need to be solved in current tracking tasks. Summary of the Invention

[0005] The present invention aims to provide a lightweight multimodal target tracking method that embeds the structural design ideas of a multi-expert model, a multimodal feature space modulation module, and a time-series prompt word module into a concise and unified multimodal target tracking model framework. A high-performance multimodal tracking model is used as a teacher model for distillation guidance to achieve an efficient and high-performance multimodal tracking algorithm. The present invention can simultaneously process multimodal tracking tasks for RGB-infrared, RGB-depth, and RGB-event data. Compared with existing high-performance multimodal tracking algorithms, it can achieve real-time speeds on CPU devices and end-side devices with minimal performance loss, achieving a good balance between performance and speed.

[0006] The technical solution of the present invention:

[0007] A lightweight multimodal target tracking method, the steps are as follows:

[0008] Step 1: Select the weights of the lightweight pre-trained algorithm D-MAE-Tiny (A Closer Look at Self-Supervised Lightweight Vision Transformers) as the initial weights. Use the training mode and tracking prediction network of the target tracking algorithm OSTrack, and use RGB modality target tracking data to fine-tune all parameters to obtain the RGB tracking algorithm model weights.

[0009] Step 2: Using the RGB tracking algorithm model weight obtained in step 1 as the initialization weight, select the single-stream tracking model structure in the target tracking algorithm OSTrack and its tracking prediction network to construct a multimodal target tracking model; the multimodal target tracking model includes an image block coding module, a multimodal feature space modulation module and a multi-expert coding layer module, and the multi-expert coding layer module includes a shared self-attention module and a non-shared multi-layer perceptron; in the multimodal target tracking model, image block coding modules with the same structure but non-shared weights are designed for different modalities, and after the image block coding module, a multimodal feature space modulation module is inserted to perform adaptive feature fusion on the RGB modality and the auxiliary modality; then, the feature fusion information of different auxiliary modalities and RGB modalities is processed separately through the multi-expert coding layer module; in the multi-expert coding layer module, different auxiliary modalities share the self-attention module weights and different auxiliary modalities do not share the multi-layer perceptron weights; the auxiliary modalities include infrared modalities, depth modalities and event modalities, and the auxiliary modalities and RGB modalities constitute different modalities;

[0010] (1) Multimodal feature space modulation module:

[0011] In the multimodal target tracking model, image block encoding modules with the same structure but not shared weights are designed for different modalities to obtain image block encoding features respectively:

[0012]

[0013]

[0014] Among them, the upper right script X represents a certain auxiliary mode, Represents the image block encoding features of RGB modality, Image block encoding features representing the auxiliary modality, The image patch encoding features representing the search area in RGB mode, Image patch encoding features representing the template region in RGB mode; The image block encoding features representing the search area of ​​the auxiliary modality, Image patches encoding features representing the template region of the auxiliary modality.

[0015] For the search area features, the multimodal feature space modulation module first encodes the image block features of the search area of ​​RGB mode. Image block encoding features of the search area and auxiliary modality Splice according to the channel dimension to obtain , and then use two sets of multi-layer perceptron networks and Acting on splicing features , generate corresponding spatial weights for RGB modality and auxiliary modality image block encoding features respectively , to achieve spatial modulation of the coding features:

[0016]

[0017]

[0018] For the template region features, the processing method is the same as that for the search region features, but the two sets of multi-layer perceptron networks responsible for predicting the spatial modulation weights 、 Two sets of multilayer perceptrons with search area features 、 No shared weights:

[0019]

[0020]

[0021] in, It is the image block encoding feature of the template area in RGB mode Image block encoding features of the template area and auxiliary modality It is obtained by splicing according to the channel dimension;

[0022] The auxiliary modality image block coding feature through adaptive modulation is finally:

[0023]

[0024] The RGB modal image block encoding features through adaptive modulation are finally:

[0025]

[0026] Finally, the adaptive modulation results are added together to obtain the fused image block coding features:

[0027] ;

[0028] (2) Coding layer module of multi-expert thinking:

[0029] Under the premise of a shared self-attention module, a specific multi-layer perceptron is configured for each RGB modality and auxiliary modality task to form a multi-expert coding layer module. The multi-layer perceptrons in the multi-expert coding layer module include infrared multi-layer perceptron, depth multi-layer perceptron, and event multi-layer perceptron. Each RGB-X multimodal tracking task only activates the corresponding multi-layer perceptron module, achieving simultaneous joint and unified training of multiple multimodal tracking tasks:

[0030] ,

[0031]

[0032] Among them, T stands for infrared modality, D stands for depth modality, and E stands for event modality.

[0033] During the joint unified training process, the self-attention map distillation method used in the lightweight pre-training algorithm D-MAE-Tiny is adopted. The multimodal tracking algorithm model ViPT (Visual Prompt Multi-Modal Tracking) is used as the teacher model to distill the self-attention map of the last layer of the current lightweight student model. This provides effective information to guide the training of the larger model. The distillation loss is combined with the tracking loss used by the target tracking algorithm OSTrack to jointly guide the model training:

[0034]

[0035] in, is the distillation loss, MSE is the mean square error loss function, is the last layer of self-attention map of the student model, is the last layer of self-attention map of the teacher model. is the mapping layer of a certain auxiliary modality X;

[0036] Step 3: After the structural design and model training in Step 2, a lightweight temporal cue word module is used to introduce the high-level semantic information of the previous frame into the current frame. Specifically:

[0037] The lightweight temporal cue word module consists of a two-layer multilayer perceptron module. It takes as input the last layer of features expressed by the joint modeling of the search area and template area of ​​the previous frame, and generates a cue word expression that is added to the fused image block encoding features in step 2:

[0038] ,

[0039] Updated to:

[0040]

[0041] The right subscript t-1 represents the previous frame, and TP represents the lightweight temporal prompt word module. represents the fused image block encoding features with added temporal hints, It is a time sequence prompt word expression.

[0042] During this training phase, only the parameters of the lightweight temporal cue word module are trainable, while the parameters of the rest of the multimodal object tracking model are frozen.

[0043] Step 4: Given the initial frame and the initial frame target annotation box of the RGB modality video data and the corresponding infrared modality / depth modality / event modality video data, multimodal target tracking is performed. For the encoding layer module of the multi-expert idea, only the parameters of the multi-layer perceptron of a specific auxiliary modality are activated.

[0044] Beneficial effects of the present invention:

[0045] (1) The concise and unified multimodal target tracking model proposed in this paper can simultaneously handle the fusion tracking of RGB modality and multiple auxiliary modalities (infrared, depth, event, etc.). The multimodal feature space modulation module can achieve early multimodal adaptive and efficient feature fusion. The temporal prompt word module can optimize the multimodal tracking results by utilizing temporal information.

[0046] (2) The lightweight multimodal target tracking method proposed in this invention can achieve a good balance between performance and speed, and can achieve real-time speed on CPU devices and end-side devices such as NVIDIA Jetson AGX Orin with minimal performance loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a framework diagram of the multimodal target tracking model.

[0048] Figure 2 Schematic diagram of the multimodal feature space modulation module structure.

[0049] Figure 3 Schematic diagram of the coding layer module structure of multi-expert ideas.

[0050] Figure 4 Schematic diagram of the lightweight temporal prompt word module structure. DETAILED DESCRIPTION

[0051] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0052] A lightweight multimodal target tracking method, the steps are as follows:

[0053] Step 1: Select the weights of the lightweight pre-trained algorithm D-MAE-Tiny (A Closer Look at Self-Supervised Lightweight Vision Transformers) as the initial weights. Use the training mode and tracking prediction network of the target tracking algorithm OSTrack, and use RGB modality target tracking data to fine-tune all parameters to obtain the RGB tracking algorithm model weights.

[0054] Step 2: Using the RGB tracking algorithm model weight obtained in step 1 as the initialization weight, select the single-stream tracking model structure in the target tracking algorithm OSTrack and its tracking prediction network to construct a multimodal target tracking model; the multimodal target tracking model includes an image block coding module, a multimodal feature space modulation module and a multi-expert coding layer module, and the multi-expert coding layer module includes a shared self-attention module and a non-shared multi-layer perceptron; in the multimodal target tracking model, image block coding modules with the same structure but non-shared weights are designed for different modalities, and after the image block coding module, a multimodal feature space modulation module is inserted to perform adaptive feature fusion on the RGB modality and the auxiliary modality; then, the feature fusion information of different auxiliary modalities and RGB modalities is processed separately through the multi-expert coding layer module; in the multi-expert coding layer module, different auxiliary modalities share the self-attention module weights and different auxiliary modalities do not share the multi-layer perceptron weights; the auxiliary modalities include infrared modalities, depth modalities and event modalities, and the auxiliary modalities and RGB modalities constitute different modalities;

[0055] (1) Multimodal feature space modulation module:

[0056] In the single-stream tracking model structure of the conventional target tracking algorithm OSTrack, the features of the template area image and the search area image are obtained after being divided and encoded. After concatenation, the encoded features are obtained , C is the number of channels, is the number of image blocks in the search area, is the number of image blocks in the template area, the right subscript x refers to the search area, and the right subscript z refers to the template area. The encoded features will be fed into multiple Transformer encoding layers with self-attention modules and multi-layer perceptron modules as the core:

[0057] ,

[0058]

[0059] in, is the number of coding layers, Represents the Transformer encoding result after layer l, Represents the self-attention module, MLP represents the multi-layer perceptron module, Representation layer normalization.

[0060] In the multimodal target tracking model of the present invention, image block encoding modules with the same structure but not sharing weights are designed for different modalities to obtain image block encoding features respectively:

[0061]

[0062]

[0063] Among them, the upper right script X represents a certain auxiliary mode, Represents the image block encoding features of RGB modality, Image block encoding features representing the auxiliary modality, The image patch encoding features representing the search area in RGB mode, Image patch encoding features representing the template region in RGB mode; The image block encoding features representing the search area of ​​the auxiliary modality, Image patches encoding features representing the template region of the auxiliary modality.

[0064] The multimodal feature space modulation module is used to explore the discrimination advantages of different modalities in different scenarios and realize effective multimodal adaptive fusion. For the search area features, the multimodal feature space modulation module first encodes the image block features of the search area of ​​the RGB mode. Image block encoding features of the search area and auxiliary modality Splice according to the channel dimension to obtain , and then use two sets of multi-layer perceptron networks and Acting on splicing features , generate corresponding spatial weights for RGB modality and auxiliary modality image block encoding features respectively , to achieve spatial modulation of the coding features:

[0065]

[0066]

[0067] For the template region features, the processing method is the same as that for the search region features, but the two sets of multi-layer perceptron networks responsible for predicting the spatial modulation weights 、 Two sets of multilayer perceptrons with search area features 、 No shared weights:

[0068]

[0069]

[0070] in, It is the image block encoding feature of the template area in RGB mode Image block encoding features of the template area and auxiliary modality It is obtained by splicing according to the channel dimension;

[0071] The auxiliary modality image block coding feature through adaptive modulation is finally:

[0072]

[0073] The RGB modal image block encoding features through adaptive modulation are finally:

[0074]

[0075] Finally, the adaptive modulation results are added together to obtain the fused image block coding features:

[0076]

[0077] (2) Coding layer module of multi-expert thinking:

[0078] To enable the model to simultaneously handle the fusion tracking of multiple auxiliary modalities and RGB modalities, a multi-expert approach was introduced into the model structure design. Under the premise of a shared self-attention module, a specific multi-layer perceptron is configured for each RGB modality and auxiliary modality task to form a multi-expert approach encoding layer module. The multi-layer perceptrons in the multi-expert approach encoding layer module include infrared multi-layer perceptron, depth multi-layer perceptron, and event multi-layer perceptron. Each RGB-X multi-modal tracking task only activates the corresponding multi-layer perceptron module, achieving simultaneous joint and unified training of multiple multi-modal tracking tasks:

[0079] ,

[0080]

[0081] Among them, T stands for infrared modality, D stands for depth modality, and E stands for event modality.

[0082] During the joint unified training process, the self-attention map distillation method used in the lightweight pre-training algorithm D-MAE-Tiny is adopted. The multimodal tracking algorithm model ViPT (Visual Prompt Multi-Modal Tracking) is used as the teacher model to distill the self-attention map of the last layer of the current lightweight student model. This provides effective information to guide the training of the larger model. The distillation loss is combined with the tracking loss used by the target tracking algorithm OSTrack to jointly guide the model training:

[0083]

[0084] in, is the distillation loss, MSE is the mean square error loss function, is the last layer of self-attention map of the student model, is the last layer of self-attention map of the teacher model. is the mapping layer of a certain auxiliary modality X;

[0085] Step 3: After the structural design and model training in step 2, in order to further optimize the results of multimodal tracking, a lightweight temporal cue word module is used to introduce high-level semantic information from the previous frame into the current frame. Specifically:

[0086] The lightweight temporal cue word module consists of a two-layer multilayer perceptron module. It takes as input the last layer of features expressed by the joint modeling of the search area and template area of ​​the previous frame, and generates a cue word expression that is added to the fused image block encoding features in step 2:

[0087] ,

[0088] Updated to:

[0089]

[0090] The right subscript t-1 represents the previous frame, and TP represents the lightweight temporal prompt word module. represents the fused image block encoding features with added temporal hints, It is a time sequence prompt word expression.

[0091] During this training phase, only the parameters of the lightweight temporal cue word module are trainable, while the parameters of the rest of the multimodal object tracking model are frozen.

[0092] Step 4: Given the initial frame and the initial frame target annotation box of the RGB modality video data and the corresponding infrared modality / depth modality / event modality video data, multimodal target tracking is performed. For the encoding layer module of the multi-expert idea, only the parameters of the multi-layer perceptron of a specific auxiliary modality are activated.

[0093] In a specific embodiment:

[0094] Figure 1 The multimodal target tracking model diagram is shown in Figure 2. The overall lightweight multimodal target tracking method is divided into two training stages. In the first training stage, the multimodal target tracking model is embedded in Figure 2 The multimodal feature space modulation module shown and Figure 3 The encoding layer module, using a multi-expert approach, jointly trains a multimodal object tracking model using three multimodal tracking datasets: RGB-infrared, RGB-depth, and RGB-event. The training datasets include RGB-infrared data (LasHeR), RGB-depth data (DepthTrack and RGBD-1K), and RGB-event data (VisEvent), with a sampling ratio of 2:1:1:2. The optimizer uses ADAMW, and training is performed for 60 epochs, sampling 60,000 sample pairs per epoch. The learning rate is decayed by 0.1 at the 48th epoch. The overall training uses the candidate box elimination mechanism and tracking loss function of the RGB tracking algorithm (OSTrack). The overall loss of the multimodal object tracking model is the sum of the tracking loss and the distillation loss, with both loss weights set to 1. The hidden dimension of the multilayer perceptron network in the multimodal feature spatial modulation module is 192.

[0095] In the second training phase, Figure 4Except for the lightweight temporal cue word module shown, all parameter weights are frozen, the overall loss of the multimodal object tracking model remains unchanged, and the training dataset remains unchanged. Training is conducted for 10 epochs, using triple sampling, with each sample set sampling two search regions and one template region. Other training parameter settings refer to those for the first phase of training. The hidden dimension of the multilayer perceptron module in the lightweight temporal cue word module is 32.

[0096] During training, the search area is 16 times the size of the target area, scaled to 256×256. The template area is 4 times the size of the target area, scaled to 128×128. The batch size per GPU is 64. During inference, the multi-expert approach activates the parameters of a specific auxiliary modality multilayer perceptron in the encoding layer module based on the given RGB video data and the corresponding infrared, depth, and event modal data for multimodal target tracking.

[0097] Table 1: Performance comparison on the RGB-T tracking dataset LasHeR

[0098] Tracker Success rate SR Accuracy CPU speed GPU speed The present invention 53.3 65.9 83.8 29.1 SDSTrack 53.1 66.5 19.4 1.5 ViPT 52.5 65.1 39.5 6.3 Un-Track 51.3 54.6 - - ProTrack 42.0 53.8 - -

[0099] As shown in Table 1, the performance of the present invention on the LasHeR dataset is good on multiple RGB-T, RGB-D, and RGB-E tracking datasets. The GPU is an RTX 3090 GPU and the CPU is an Intel Core i9-10850K CPU.

Claims

1. A lightweight multimodal target tracking method, characterized in that: Here are the steps: Step 1: Select the weights of the lightweight pre-trained algorithm D-MAE-Tiny as the initialization weights, use the training mode and tracking prediction network of the target tracking algorithm OSTrack, and use the RGB modality target tracking data to fine-tune all parameters to obtain the RGB tracking algorithm model weights; Step 2: Using the RGB tracking algorithm model weight obtained in step 1 as the initialization weight, select the single-stream tracking model structure in the target tracking algorithm OSTrack and its tracking prediction network to build a multimodal target tracking model; the multimodal target tracking model includes an image block encoding module, a multimodal feature space modulation module and a multi-expert coding layer module, and the multi-expert coding layer module includes a shared self-attention module and an unshared multi-layer perceptron; in the multimodal target tracking model, image block encoding modules with the same structure but unshared weights are designed for different modalities, and after the image block encoding module, a multimodal feature space modulation module is inserted to perform adaptive feature fusion on the RGB modality and the auxiliary modality; Then, the feature fusion information of different auxiliary modalities and RGB modalities is processed separately through the encoding layer module of multi-expert ideas; In the encoding layer module of the multi-expert approach, different auxiliary modalities share the weights of the self-attention module, and different auxiliary modalities do not share the weights of the multi-layer perceptron; the auxiliary modalities include infrared modalities, depth modalities, and event modalities, and the auxiliary modalities and RGB modalities constitute different modalities; Step 3: After the structural design and model training in step 2, a lightweight temporal cue word module is used to introduce the high-level semantic information of the previous frame into the current frame; Step 4: Given the initial frame and the initial frame target annotation box of the RGB modality video data and the corresponding infrared modality / depth modality / event modality video data, multimodal target tracking is performed, wherein for the encoding layer module of the multi-expert idea, only the parameters of the multi-layer perceptron of a specific auxiliary modality are activated.

2. A lightweight multi-modal target tracking method according to claim 1, characterized in that: The multimodal feature space modulation module in step 2: In the multimodal target tracking model, image block encoding modules with the same structure but not sharing weights are designed for different modalities to obtain image block encoding features respectively: Among them, the superscript X represents a certain auxiliary mode. Represents the image patch encoding features in RGB mode, The image patch encoding features representing the auxiliary modality, The image patch encoding features representing the search area in RGB mode, Image patch encoding features representing the template region in RGB modality; The image patch encoding features representing the search area of ​​the auxiliary modality, An image patch encoding feature representing a template region of an auxiliary modality; For the search area features, the multimodal feature space modulation module first encodes the image block features of the search area of ​​the RGB mode. Image block encoding features of the search area and auxiliary modality Splice according to the channel dimension to obtain , and then use two sets of multi-layer perceptron networks and Acting on the splicing feature , generate corresponding spatial weights for the RGB modality and auxiliary modality image block encoding features respectively , to achieve spatial modulation of the coding features: The template region features are processed in the same way as the search region features, but two sets of multi-layer perceptron networks are responsible for predicting the spatial modulation weights. , Two sets of multi-layer perceptrons with search area features , Weights are not shared: in, It is the image patch encoding feature of the template area in RGB mode Image block encoding features of the template area of ​​the auxiliary modality It is obtained by splicing according to the channel dimension; The auxiliary modality image block coding feature through adaptive modulation is finally: The RGB modal image block encoding features through adaptive modulation are finally: Finally, the adaptive modulation results are added to obtain the fused image block coding features: 。 3. A lightweight multi-modal target tracking method according to claim 2, characterized in that: The coding layer module of the multi-expert approach in step 2: Under the premise of a shared self-attention module, a specific multi-layer perceptron is configured for each RGB modality and auxiliary modality task to form a multi-expert coding layer module. The multi-layer perceptrons in the coding layer module of the multi-expert idea include infrared multi-layer perceptron, depth multi-layer perceptron, and event multi-layer perceptron. Each RGB-X multi-modal tracking task only activates the corresponding multi-layer perceptron module to achieve simultaneous joint and unified training of multiple multi-modal tracking tasks: , Among them, T represents infrared modality, D represents depth modality, and E represents event modality; At the same time, in the joint unified training process, the self-attention map distillation method in the lightweight pre-training algorithm D-MAE-Tiny is used, and the multimodal tracking algorithm model ViPT is used as the teacher model to distill the last layer of the self-attention map of the current lightweight student model, providing effective information to guide the training of the larger model. The distillation loss and the tracking loss used by the target tracking algorithm OSTrack jointly guide the training of the model: in, is the distillation loss, MSE is the mean square error loss function, is the last layer of self-attention map of the student model, It is the last layer of self-attention map of the teacher model; is the mapping layer of a certain auxiliary mode X.

4. A lightweight multi-modal target tracking method according to claim 3, characterized in that: Step 3 is as follows: The lightweight temporal cue word module is composed of a two-layer multi-layer perceptron module. It takes the last layer of features expressed by the joint modeling of the search area and the template area of ​​the previous frame as input, and generates a cue word expression to add to the fused image block encoding features in step 2: ， Updated to: The right subscript t-1 represents the previous frame, and TP represents a lightweight temporal prompt word module. represents the fused image block encoding features with added temporal hints, It is expressed as a time sequence cue word; In this stage of training, only the parameters of the lightweight temporal cue word module can be trained, and the parameters of the rest of the multimodal target tracking model are frozen.

Images