Multi-modal unified target tracking method based on modal unified representation

Abstract

The invention belongs to the field of machine learning, computer vision and target tracking, and discloses a multi-modal unified target tracking method based on modal unified representation. Visible light, depth, infrared, event and natural language modalities are expressed in a unified marking form through a multi-modal embedding layer, so that it is possible to train a Transform model to extract and fuse joint features of multiple modalities, and a target tracking model capable of processing different multi-modal input signals is developed. Besides, a task identification training strategy is introduced in the method to enhance the ability of the model to distinguish different modal tracking tasks, soft tag type embedding is proposed to provide accurate tag type information for the model, and the performance of the multi-modal unified model is further improved. Finally, according to the method, different multi-modal tracking tasks are solved through a unified scheme, architecture unification, model unification and knowledge sharing among different tasks are achieved, and good tracking performance is obtained on the five multi-modal tracking tasks.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning, computer vision, and target tracking, and relates to a multimodal signal representation algorithm, a multi-task joint training algorithm, and a multimodal target tracking algorithm; specifically, it is a multimodal unified target tracking method based on modal unified representation. Background Art

[0002] Visual target tracking aims to continuously track the target of interest specified by the user in a continuous video by using visual features and temporal information. As one of the basic technologies of computer vision systems, target tracking is widely used in defense and military, intelligent transportation, autonomous driving, security monitoring, human-computer interaction, and smart business.

[0003] Currently, single-modal visual target tracking algorithms based on visible light images have achieved good performance in many scenarios. However, in more complex or specific scenarios, visible light images may not provide sufficient information to support accurate tracking. To address this challenge, many tracking methods have begun to use multimodal signals to enhance tracking performance.

[0004] Specifically, depth modal data provides three-dimensional information about the shape and position of scenes and targets, thereby enhancing the robustness of tracking methods to occlusion and complex backgrounds. Representative methods include SPT, DeT, DAL, and CA3DMS. Infrared modal data can provide target and background features that are not affected by ambient lighting conditions, so it is crucial for accurate tracking in low lighting or extreme weather conditions. Representative methods include APFNet, CMPP, JMMAC, and DAFNet. Event modal data is captured at a high frame rate, providing high dynamic range and low latency perception capabilities, and is suitable for fast-moving or high-speed changing scenes. Representative methods include OSTrack_E and SiamRCNN_E. Natural language modal data can describe the attributes and behaviors of the target in natural language, further enhancing the understanding and reasoning capabilities of the tracker. Representative methods include JointNLT, DecoupleTNL, and VLT. TT , and CapsuleTN, etc.

[0005] The above multimodal tracking methods for a single task have achieved good performance, but due to the different characteristics of various modal data, these methods design targeted model architectures for each modality and train independent models. This independence between multimodal tracking methods leads to redundant training processes, complex model structures, and limits the sharing of cross-modal knowledge.

[0006] Therefore, how to design and develop a target tracking method that can simultaneously process multiple modal inputs to improve the performance and application scope of the algorithm is one of the urgent challenges to be solved in the current tracking field. Summary of the invention

[0007] The present invention aims to provide a multimodal unified target tracking method based on unified representation of modalities. Through a multimodal embedding layer, visible light, depth, infrared, event, and natural language modalities are represented as a unified labeling form, making it possible to train a Transformer model to extract and fuse joint features of multiple modalities, thereby developing a target tracking model that can handle different multimodal input signals. In addition, the method introduces a task identification training strategy to enhance the model's ability to distinguish different modal tracking tasks, and proposes soft label type embedding to provide the model with accurate label type information, further improving the performance of the multimodal unified model. Finally, the method solves different multimodal tracking tasks through a unified solution, realizes the unified architecture, unified model, and knowledge sharing among different tasks, and achieves good tracking performance on five multimodal tracking tasks (visible light, visible light-depth, visible light-infrared, visible light-event, and visible light-natural language tracking).

[0008] The technical solution of the present invention:

[0009] A multi-modal unified target tracking method based on modality unified representation, the steps are as follows:

[0010] Step 1: Obtain videos of multimodal target tracking tasks from public datasets for multimodal target tracking model training. The videos of multimodal target tracking tasks include five modalities: visible light, visible light-depth tracking, visible light-infrared tracking, visible light-event tracking, and visible light-natural language. The depth, infrared, event, and natural language modalities are paired with the visible light modality, and the depth, infrared, and event modalities are stored in the form of three-channel images. Randomly collect image frames from the videos of multimodal target tracking tasks, expand the bounding boxes of the targets of interest marked on the image frames by 2 times and 4 times respectively to generate sample pairs of template images and search area images, and use brightness transformation and inversion for augmentation.

[0011] Step 2: Convert the input signals of different modalities of each multimodal target tracking task into a unified label embedding form, i.e., a unified representation of the modality;

[0012] (1) For the visible light-depth target tracking task, the visible light-infrared target tracking task, and the visible light-event target tracking task, the depth, infrared, and event modalities are collectively referred to as auxiliary modal data and marked as DTE; the visible light modal data is marked as RGB; the visible light modal data and the auxiliary modal data are paired, and after the visible light modal data and the auxiliary modal data are spliced ​​in the channel direction, a multimodal embedding layer is used to jointly perform the label embedding conversion, so that the visible light modal data and the auxiliary modal data are converted into a unified label embedding representation;

[0013] Image and images Stitching is performed along the channel direction to obtain a stitched image , see the following formula:

[0014]

[0015] Next, stitch the images together It is divided into fixed-size image blocks, each of which has a size of ; Then, each The image block is flattened into a length of A one-dimensional vector of; Finally, a linear transformation is applied to map the flattened image block vector to the embedding space, as shown below:

[0016]

[0017] in, Indicates The embedding vector of the image block has the dimension ; Indicates The flattened vector of the image patch, The dimension is The weight matrix of The dimension is The bias term of

[0018] (2) For the target tracking task that does not contain DTE data, a six-channel input is created by copying the three channels of the RGB data, and then processed using the multimodal embedding layer to obtain a multimodal embedding;

[0019] In the visible light-natural language object tracking task, for the natural language modality, a language model is used as a text encoder to extract a language feature embedding; the language model is a CLIP-L model, on which a linear layer is added to adjust the dimension; the language feature embedding is then concatenated with the multimodal embedding and input into the Transformer encoder;

[0020] (3) For target tracking tasks that do not contain language modalities, a fixed meaningless sentence is used to fill in the gaps;

[0021] Through the above methods, unified tag embeddings of multimodal search areas, templates, and text descriptions are obtained; these unified tag embeddings are directly sent to a Transforme encoder after being spliced ​​in the spatial dimension; the Transformer encoder completes the joint feature extraction and fusion of these unified tag embeddings through the attention mechanism;

[0022] Step 3: Add soft marker type encoding to the obtained unified marker embedding to enhance the multimodal tracking model to accurately distinguish between the foreground marker of the template image, the background marker of the template image, and the marker of the search area image; the foreground marker of the template image and the background marker of the template image are separated by a given bounding box To distinguish, as follows:

[0023] Given a template image containing the target and its bounding box , the template image containing the target is the result of splicing the template image obtained in step 1 in the channel direction of step 2; first create a mask with the same size as the template image , in this mask, pixels inside the bounding box are assigned a value of 1, and pixels outside the bounding box are assigned a value of 0:

[0024]

[0025] Next, the mask Divide into sizes non-overlapping image blocks; The image blocks are denoted as ; Then, for each value in the image block, calculate the average value:

[0026]

[0027] in, For the The average value of the image blocks represents the degree to which the label embedding corresponding to the image block is regarded as the foreground;

[0028] For each marker type, there is a learnable marker type embedding, including the foreground marker type embedding of the template image. , Template background tag type embed and search region image tag type embedding , these tag type embeddings are learned in the training phase and fixed in the inference phase; the tag type embeddings corresponding to different types of tags are weighted based on the average value of the image patch, thereby enhancing the multimodal image patch embedding of the template image and the search area image; for the kth image patch embedding of the template image, the following adjustments are made:

[0029]

[0030] in, Indicates The adjusted embedding of image patches, represents the original multimodal image patch embedding, is the foreground tag type embedding, is the background marker type embedded; for the search area image, only the search area marker type is embedded Added to each image patch embedding, no longer distinguishing between foreground and background:

[0031]

[0032] Step 4: Use the above multimodal tag embedding to train a multimodal target tracking model; the multimodal target tracking model consists of a Transformer encoder and a tracking head prediction network; the Transformer encoder is used to extract and fuse the features of the input multimodal search area image and template image, using the HiViT structure; the tracking head prediction network is used to predict the tracking results based on the features output by the Transformer encoder, using the OSTrack head network structure; in order to train the multimodal target tracking model, the following training and optimization strategies are used:

[0033] A multi-task data mixing training method is adopted, that is, data from five multimodal target tracking tasks are mixed in each training batch; for the multimodal target tracking task, a weighted focal loss is used for the supervision of foreground and background classification; for the supervision of bounding box regression, a weighted focal loss is used. The multimodal target tracking model is supervised by a combination of the loss and the generalized intersection-over-union loss. In addition, a task identification training strategy is designed to enhance the ability of the multimodal target tracking model to distinguish different modal target tracking tasks and corresponding modalities. The task identification training strategy uses an explicit task identification mechanism to enable the multimodal target tracking model to clearly identify the target tracking task currently being performed during the training process, thereby optimizing its performance under different target tracking tasks.

[0034] First, all feature embeddings output by the Transformer encoder are averaged to generate a single feature vector ; The calculation formula of the eigenvector is:

[0035]

[0036] in, represents the number of output feature embeddings, Indicates The output feature embedding; next, the tracking model inputs the feature vector into a multi-layer perceptron for task classification; the classified tasks include five types of visible light target tracking tasks, visible light-depth target tracking tasks, visible light-infrared target tracking tasks, visible light-event target tracking tasks, and visible light-natural language target tracking tasks; the task classification formula is:

[0037]

[0038] in, Represents a multi-layer perceptron, through which the multimodal target tracking model predicts which of the five tasks the current task belongs to; the output is a task probability distribution, which indicates the predicted probability of each task by the multimodal target tracking model. Then, the cross entropy loss function is used to calculate the gap between the predicted result of the multimodal target tracking model and the true task label, so as to guide the learning of the multimodal target tracking model. The expression of the cross entropy loss function is:

[0039]

[0040] in, Indicates the number of tasks, ; It's a task The real label, The multimodal target tracking model predicts that the current data belongs to the task By minimizing the cross entropy loss function, the multimodal target tracking model can better distinguish which task and modality the current data belongs to, thereby improving the understanding of the data and promoting the overall tracking performance.

[0041] Finally, the loss function of the multimodal target tracking model training is:

[0042]

[0043] in, represents the weighted focal loss for foreground / background classification, represents the generalized intersection-over-union loss, yes norm loss, is the cross entropy loss used for task identification; and They are the generalized intersection-over-union loss and Regularization coefficient of the loss;

[0044] Step 5: In the inference phase, a multi-template strategy is adopted; two templates are used: one is a static initial template, and the other is a template that is dynamically updated during the tracking process; the template update mechanism decides when to update based on a fixed time interval and confidence threshold.

[0045] The above public data sets include VastTrack, LaSOT, GOT-10k, TrackingNet, COCO, DepthTrack, Lasher, VisEvent, and TNL2K.

[0046] Beneficial effects of the present invention:

[0047] (1) The multimodal unified representation method proposed in this invention makes it possible to train a target tracking model that can uniformly process multiple modal signals. The developed multimodal unified tracking model can handle tracking tasks of five different modalities through a shared architecture and model, simplifying the development process and model architecture, promoting knowledge sharing, and thus improving the flexibility and accuracy of the tracking model.

[0048] (2) The task identification auxiliary training strategy proposed in the present invention promotes the effect of unified training of multimodal tasks. The proposed soft label type encoding provides the model with clearer label type information to avoid confusion in label embedding, and ultimately further improves the accuracy of the multimodal unified tracking model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a framework diagram of the multimodal tracking method based on unified modal representation. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0051] A multi-modal unified target tracking method based on modality unified representation, the steps are as follows:

[0052] Step 1: Data Preparation

[0053] Training data is collected from five multimodal tracking tasks, including visible light, visible light-depth, visible light-infrared, visible light-event, and visible light-natural language datasets. Samples are generated by pairing data from different modalities, and data diversity is enhanced by random frame sampling and data augmentation. In addition, the target bounding box is appropriately expanded to generate template and search area image pairs.

[0054] Step 2: Unified representation of modalities

[0055] Using the multimodal embedding layer, the input data of different modalities are mapped into a unified labeled embedding representation. The specific operations include:

[0056] (1) Channel splicing of visible light and other modal data;

[0057] (2) Fix the image block size, split and flatten the spliced ​​data;

[0058] (3) Use linear transformation to embed the flattened data into a unified representation space;

[0059] (4) For natural language modality, the CLIP-L model is used to extract language features and then concatenated with other modal features to form a unified representation.

[0060] Step 3: Soft Tag Type Embedding

[0061] Pixels inside and outside the target bounding box are classified, and soft label weights are introduced to adjust the feature embedding. This adjustment mechanism is applied to both template and search area data to ensure the accuracy of label embedding, especially at the junction of foreground and background areas.

[0062] Step 4: Model training

[0063] A hybrid training strategy is adopted to uniformly train the five modal task data. The Transformer encoder is combined for feature extraction and fusion, and the task identification strategy is used to enhance the model's ability to distinguish different tasks. At the same time, the model performance is improved through the joint optimization of foreground / background classification loss, bounding box regression loss, and task identification loss.

[0064] Step 5: Multi-template update strategy in the inference phase

[0065] During the reasoning process, the static initial template and the dynamically updated template are combined to continuously adjust the target appearance information and adapt to the changes in the target's appearance, thereby achieving more stable and accurate target tracking.

Claims

1. A multi-modal unified target tracking method based on modality unified representation, characterized in that: Here are the steps: Step 1: Obtain videos of multimodal target tracking tasks from public datasets for multimodal target tracking model training. The videos of multimodal target tracking tasks include five modalities: visible light, visible light-depth tracking, visible light-infrared tracking, visible light-event tracking, and visible light-natural language. The depth, infrared, event, and natural language modalities are paired with the visible light modality, and the depth, infrared, and event modalities are stored in the form of three-channel images. Randomly collect image frames from the videos of multimodal target tracking tasks, expand the bounding boxes of the targets of interest marked on the image frames by 2 times and 4 times respectively to generate sample pairs of template images and search area images, and use brightness transformation and inversion for augmentation. Step 2: Convert the input signals of different modalities of each multimodal target tracking task into a unified label embedding form, i.e., a unified representation of the modality; (1) For the visible light-depth target tracking task, the visible light-infrared target tracking task, and the visible light-event target tracking task, the depth, infrared, and event modalities are collectively referred to as auxiliary modality data and marked as DTE; the visible light modality data is marked as RGB; The visible light modality data and the auxiliary modality data are paired. After the visible light modality data and the auxiliary modality data are spliced ​​in the channel direction, a multimodal embedding layer is used to jointly perform a label embedding conversion, so that the visible light modality data and the auxiliary modality data are converted into a unified label embedding representation; Image and images Stitching is performed along the channel direction to obtain a stitched image , see the following formula: Next, stitch the images together It is divided into fixed-size image blocks, each of which has a size of ; Then, each The image block is flattened into a length of A one-dimensional vector of; Finally, a linear transformation is applied to map the flattened image block vector to the embedding space, as shown below: in, Indicates The embedding vector of the image block has the dimension ; Indicates The flattened vector of the image patch, The dimension is The weight matrix of The dimension is The bias term of (2) For the target tracking task that does not contain DTE data, a six-channel input is created by copying the three channels of the RGB data, and then processed using the multimodal embedding layer to obtain a multimodal embedding; In the visible light-natural language object tracking task, for the natural language modality, a language model is used as a text encoder to extract a language feature embedding; the language model is a CLIP-L model, on which a linear layer is added to adjust the dimension; the language feature embedding is then concatenated with the multimodal embedding and input into the Transformer encoder; (3) For target tracking tasks that do not contain language modalities, a fixed meaningless sentence is used to fill in the gaps; Through the above methods, unified tag embeddings of multimodal search areas, templates, and text descriptions are obtained; these unified tag embeddings are directly sent to a Transforme encoder after being spliced ​​in the spatial dimension; the Transformer encoder completes the joint feature extraction and fusion of these unified tag embeddings through the attention mechanism; Step 3: Add soft marker type encoding to the obtained unified marker embedding to enhance the multimodal tracking model to accurately distinguish between the foreground marker of the template image, the background marker of the template image, and the marker of the search area image; the foreground marker of the template image and the background marker of the template image are separated by a given bounding box To distinguish, as follows: Given a template image containing the target and its bounding box , the template image containing the target is the result of splicing the template image obtained in step 1 in the channel direction of step 2; first create a mask with the same size as the template image , in this mask, pixels inside the bounding box are assigned a value of 1, and pixels outside the bounding box are assigned a value of 0: Next, the mask Divide into sizes non-overlapping image blocks; The image blocks are denoted as ; Then, for each value in the image block, calculate the average value: in, For the The average value of the image blocks represents the degree to which the label embedding corresponding to the image block is regarded as the foreground; For each marker type, there is a learnable marker type embedding, including the foreground marker type embedding of the template image. , Template background tag type embed and search region image tag type embedding , these tag type embeddings are learned in the training phase and fixed in the inference phase; the tag type embeddings corresponding to different types of tags are weighted based on the average value of the image patch, thereby enhancing the multimodal image patch embedding of the template image and the search area image; for the kth image patch embedding of the template image, the following adjustments are made: in, Indicates The adjusted embedding of image patches, represents the original multimodal image patch embedding, is the foreground tag type embedding, is the background marker type embedded; for the search area image, only the search area marker type is embedded Added to each image patch embedding, no longer distinguishing between foreground and background: Step 4: Use the above multimodal tag embedding to train a multimodal target tracking model; the multimodal target tracking model consists of a Transformer encoder and a tracking head prediction network; the Transformer encoder is used to extract and fuse the features of the input multimodal search area image and template image, using the HiViT structure; the tracking head prediction network is used to predict the tracking results based on the features output by the Transformer encoder, using the OSTrack head network structure; in order to train the multimodal target tracking model, the following training and optimization strategies are used: A multi-task data mixing training method is adopted, that is, data from five multimodal target tracking tasks are mixed in each training batch; for the multimodal target tracking task, a weighted focal loss is used for the supervision of foreground and background classification; for the supervision of bounding box regression, a weighted focal loss is used. The multimodal target tracking model is supervised by a combination of the loss and the generalized intersection-over-union loss. In addition, a task identification training strategy is designed to enhance the ability of the multimodal target tracking model to distinguish different modal target tracking tasks and corresponding modalities. The task identification training strategy uses an explicit task identification mechanism to enable the multimodal target tracking model to clearly identify the target tracking task currently being performed during the training process, thereby optimizing its performance under different target tracking tasks. First, all feature embeddings output by the Transformer encoder are averaged to generate a single feature vector ; The calculation formula of the eigenvector is: in, represents the number of output feature embeddings, Indicates The output feature embedding; next, the tracking model inputs the feature vector into a multi-layer perceptron for task classification; the classified tasks include five types of visible light target tracking tasks, visible light-depth target tracking tasks, visible light-infrared target tracking tasks, visible light-event target tracking tasks, and visible light-natural language target tracking tasks; the task classification formula is: in, Represents a multi-layer perceptron, through which the multimodal target tracking model predicts which of the five tasks the current task belongs to; the output is a task probability distribution, which indicates the predicted probability of each task by the multimodal target tracking model. Then, the cross entropy loss function is used to calculate the gap between the predicted result of the multimodal target tracking model and the true task label, so as to guide the learning of the multimodal target tracking model. The expression of the cross entropy loss function is: in, Indicates the number of tasks, ; It's a task The real label, The multimodal target tracking model predicts that the current data belongs to the task By minimizing the cross entropy loss function, the multimodal target tracking model can better distinguish which task and modality the current data belongs to, thereby improving the understanding of the data and promoting the overall tracking performance. Finally, the loss function of the multimodal target tracking model training is: in, represents the weighted focal loss for foreground / background classification, represents the generalized intersection-over-union loss, yes norm loss, is the cross entropy loss used for task identification; and They are the generalized intersection-over-union loss and Regularization coefficient of the loss; Step 5: In the inference phase, a multi-template strategy is adopted; two templates are used: one is a static initial template, and the other is a template that is dynamically updated during the tracking process; the template update mechanism decides when to update based on a fixed time interval and confidence threshold.

2. The multi-modal unified target tracking method based on modality unified representation according to claim 1 is characterized in that: Public datasets include VastTrack, LaSOT, GOT-10k, TrackingNet, COCO, DepthTrack, Lasher, VisEvent, and TNL2K.

Images