A multi-modal target tracking method based on state space model

Abstract

The present application belongs to the technical field of computer vision, and relates to a multi-modal target tracking method based on a state space model. The present application proposes a multi-modal visual tracking method based on a state space model for the visual tracking task of a moving target under the conditions of fast motion and poor illumination, so as to realize accurate tracking of a visual target. The method fully combines the advantages of RGB images and event data: the RGB images provide rich texture information, and the event camera can still capture the edge and motion information of an object in a complex scene, thereby improving the tracking performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and relates to a method for tracking a moving target based on deep learning, combining event camera and RGB camera images. BACKGROUND

[0002] Visual object tracking is a basic task in computer vision, and the main goal is to estimate the position and shape of the target in the video sequence according to the initial state of the target. In recent years, the combination of event cameras and traditional RGB cameras in the multi-modal method has attracted more and more attention in the visual tracking task due to the advantages of high temporal resolution and high dynamic range of event cameras. These RGB-event multi-modal methods significantly improve the tracking performance of the model under challenging conditions such as fast motion and low light.

[0003] Although the above methods perform well in improving tracking effect, they often need to invest a lot of effort to develop overly complex fusion algorithms. In addition, existing RGB image-event image tracking algorithms usually ignore the mining and application of time information, which is crucial for target tracking because it can capture the appearance changes and motion trends of the target. Existing RGB visual trackers update appearance information by replacing or fusing templates, but these simple methods sometimes fail to effectively filter out low-quality images, thereby affecting performance. To solve this problem, some attention mechanism-based methods introduce an autoregressive mechanism for passing time information between multiple search frames. However, the attention mechanism itself lacks a design specifically for modeling time series, especially when dealing with long sequences, the above problems become a bottleneck for model performance:

[0004] (1) State Space Model (SSM)

[0005] The general form of the state space model can be expressed as: mapping a one-dimensional input sequence to a one-dimensional output sequence through an implicit high-dimensional state sequence. Compared with traditional convolutional neural networks and attention mechanisms, state space models exhibit unique advantages in modeling global information. Recent research such as the Mamba model innovatively introduces data-dependent state space parameters and parallel scanning selection mechanisms, significantly improving inference speed and overall performance, surpassing attention mechanism models of the same size.

[0006] Recently, state space models have been widely applied in the field of computer vision and have attracted widespread attention. Their characteristics naturally fit the time series information transmission in target tracking, however, there is currently no effective research applying state space models to the field of visual object tracking.

[0007] (2) Tracking based on historical time series information

[0008] The historical information can provide additional prior information for the motion, appearance and change trend of the target, thereby improving the robustness of the tracker. The tracking method using historical time sequence information can be roughly divided into two categories. The first category of method processes a single frame search image, relies on dynamic filtering or additional update branch to capture the change of target appearance over time, and dynamically updates the template image during tracking. However, this kind of method usually needs to be carefully designed, and manually adjusted according to different application scenarios. The second category of method inputs multiple search sequences, decodes the whole trajectory sequence step by step, thereby modeling the motion trajectory of the target in time sequence. However, this kind of method is limited by the limitation of attention mechanism architecture in long time sequence modeling.

[0009] (3) RGB-event information multimodal tracking

[0010] In recent years, due to the unreliability of RGB images in extreme scenes, more and more researches introduce event cameras to improve the performance of target tracking. These multimodal trackers can be mainly divided into two categories according to the visual backbone network adopted. The first category of method relies on traditional convolutional neural network (CNN) to extract features from RGB and event modalities, and fuses them through carefully designed cross-modal attention mechanism. The second category of method adopts attention mechanism as the backbone network, and models the relationship between RGB image information and event information by using multi-head attention mechanism. However, these methods often overemphasize the architecture design of multimodal fusion, and ignore the mining of historical time sequence information, and perform poorly in scenes where the target appearance changes dramatically or the field of view is lost. SUMMARY

[0011] The present application proposes a multimodal visual tracking method based on state space model for the task of visual tracking of moving targets in fast motion and poor lighting conditions, to realize accurate tracking of visual targets. This method fully combines the advantages of RGB images and event data: RGB images provide rich texture information, while event cameras can still capture the edge and motion information of objects in complex scenes, thereby improving the tracking performance.

[0012] The technical scheme of the present application is as follows:

[0013] A multimodal target tracking method based on state space model, the steps are as follows:

[0014] Step 1, RGB-event image data processing

[0015] Firstly, image data in various challenging scenes are collected by the calibrated event camera and RGB camera, the event stream generated by the event camera is stacked into event image frames according to a certain time step, and the image alignment and screening are performed according to the frequency of the RGB camera, to obtain the event image. Then, the RGB image and the event image are preprocessed and sent into the model.

[0016] Step 2, construction of multi-modal encoder

[0017] The multi-modal encoder is used to extract the information of RGB image and event image, and the attention network HiViT with vertical architecture is used as the multi-modal encoder to extract features layer by layer through token encoding, token merging and self-attention mechanism. Between the attention layers in the middle, the modal fusion module FM based on state space model is inserted to perform cross-modal fusion. Finally, the final layer features extracted are sent to the historical information decoder and the prediction head for the next step of prediction.

[0018] Step 2.1, construction of modal fusion module based on state space model

[0019] The modal fusion module FM based on state space model is constructed to fuse the semantic information of multi-modal, and the target perception scanning module and the cross-modal scanning module are introduced into the FM module respectively.

[0020] Further:

[0021] (1) Target perception scanning module

[0022] The target perception scanning module utilizes the global modeling capability of the state space model to enhance the interaction between the template region and the search region within the modal, and the target perception scanning module adopts a bidirectional state space model scanning mechanism to calculate the correlation between the template features and the search features.

[0023] Definition of target perception scanning module:

[0024]

[0025] wherein, represents a linear projection layer; represents a layer normalization operation; σ is a SiLU activation function; ψ is a 1-dimensional convolution operation; and ⊙ represents element-wise multiplication. represents the features of a certain layer in the multi-modal encoder, respectively represent the RGB image features and the event image features, is the projection of is a gating signal; represents a forward scanning state space model, represents a backward scanning state space model, Y f and Y b are the outputs through the forward and backward directions (i.e. and ) of the SSM. ​is the output of the target-aware scan module in the nth layer encoder after jointly modeling the template region features and the search region features, denotes the template region features in the nth layer encoder, denotes the search region features in the nth layer encoder.

[0026] (2) Cross-modal scan module

[0027] The cross-modal scan module is established based on the gating mechanism of the state space model to cross-fuse two modalities, and the definition of the cross-modal scan module is:

[0028]

[0029] Taking the RGB modality image as an example, wherein, denotes the features of the RGB image after being encoded by the target-aware scan module in the nth layer, denotes the search region features of the RGB image in the nth layer, Y ′ denotes intermediate features after being scanned by the state space model, denotes the fusion features of the RGB image in the nth layer extracted after being scanned by the cross-modal scan module, and the modality-specific gating signal and are used to guide the interaction between modalities, thereby enhancing the main modality while suppressing the degenerative modality.

[0030] Through the target-aware scan module and the cross-modal scan module, the multi-modal encoder can promote the interaction within and across modalities. The RGB and event features of the last layer encoder are added to obtain the final fusion features final fusion features will be used to predict the target position after the history information decoder.

[0031] Step 3, history information decoder construction

[0032] The history information decoder perceives the motion trend and appearance change of the target by decoding the historical information, thereby assisting the model in making accurate predictions. Each layer of the history information decoder includes a historical state perception module and a feature sequence attention module:

[0033] Further, (1) the historical state perception module

[0034] The historical state perception module is used to model the historical information in the video sequence, and a sequence modeling module based on the state space model is adopted to model the target state of N frames in the sequence as N one-dimensional learnable vectors Q(q1,…, q N )∈R N×DWhere D is the encoding dimension. The sequence is modeled using a state-space model, and Q is output. ′ =(q ′ ,…,q ′ N )∈R N×D The formula is as follows:

[0035]

[0036] Among them, X Q Z represents the input features after Q-projection. Q This indicates a query for the gated feature after Q-projection. This represents a state-space model. For linear projection layers, For normalization layer,

[0037] Output Q ′ This will then be used as input to the feature sequence attention module. Target query vector Q ′ Subsequently, the feature sequence attention module is used in conjunction with the multimodal features from step 2. By merging the data, the target within the current search area can be accurately located.

[0038] (2) Definition of the feature sequence attention module:

[0039]

[0040] Among them, Q ″ This represents the characteristics of historical information scanned by the historical state perception module. H represents the vector concatenation operation. i This represents the intermediate features after self-attention. Let q represent the attention mechanism, k represent the attention query, and v represent the attention value. as well as All are learnable parameters, d m Indicates the input dimension d q d k , and d v Let n represent the vector dimensions of q, k, and v, respectively. h This is represented as the token length. It is the Softmax function, Q ″ After processing by the feedforward network (FFN), the data is input into the subsequent decoding layer to generate the query for the next time step. Finally, spatial and temporal information are combined using the following matrix multiplication method:

[0041]

[0042] where F out represent the multi-modal feature fused with historical temporal information, and highlight the possible location of the target.

[0043] Step 4, input the multi-modal feature F The output F of the historical information decoder out These features will be input into the tracking head for the prediction of the target bounding box.

[0044] The beneficial effects of the present application are:

[0045] (1) Historical temporal information perception

[0046] The existing RGB-event multi-modal tracking method usually ignores the mining of historical temporal information, which is crucial for the multi-modal tracking task. This information can capture the changes in the state of the target and the related motion trend. Unlike the past methods that design a template branch or correlation filtering to update the template image information, the present patent first captures the temporal information between multiple search frames based on a linear complexity state space model. By regressing the historical information and associating it with the current frame information, the position of the next frame is predicted in an autoregressive manner, solving the problem of target loss and complex background in long-distance tracking.

[0047] (2) Efficient fusion of RGB and event modalities

[0048] Due to the asynchrony of event data, unlike the past methods of fusing RGB-event information based on convolutional networks or attention mechanisms, the present patent first explores the way to fuse RGB data and event data based on a state space model scanning mechanism. The FM module proposed in the present patent effectively extracts features from the RGB and event modalities through a gating mechanism, and fuses them according to the gating signal significance of the state space model. By dynamically fusing the information of the two modalities in different scenarios, the present patent can effectively complement the advantages of the two sensors to solve the target tracking problem under the degeneration of a single modality. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the overall target tracking flowchart of the present application.

[0050] Figure 2 is the structure diagram of the modality fusion module (FM) based on the state space model of the present application, which includes (a) the overall architecture of the model, (b) the target awareness scanning module (TAS), and (c) the cross-modality scanning module (CMS).

[0051] Figure 3 is the structure diagram of the historical information decoder of the present application.

[0052] Figure 4 is the comparative experimental result of the present application on the VisEvent dataset.

[0053] Figure 5 is the comparative experimental result of the present application on the FELT dataset.

[0054] Figure 6 is the comparative experimental result of the present application on the FE108 dataset. DETAILED DESCRIPTION

[0055] The present application will be further described in conjunction with the specific embodiments, but the present application is not limited to the specific embodiments.

[0056] A multi-modal single target visual tracking method based on state space model, including training and testing of network model.

[0057] (1) Multi-modal encoder

[0058] The multi-modal encoder is used to extract the information of RGB and event images. This method adopts a vertical architecture self-attention network HiViT as a multi-modal encoder, and extracts features layer by layer through token encoding, token merging and self-attention mechanism. Between the attention layers in the middle, a modal fusion module based on state space model is inserted for cross-modal fusion. Finally, the final layer features extracted are sent to the decoder and prediction head for the next step of prediction.

[0059] (2) Modal fusion module (FM) based on state space model

[0060] The present patent proposes a modal fusion module (FM) based on state space model, which is used to fuse the semantic information of multi-modal. The design of FM module is based on the following two principles: (i) the interaction between template image features and search image features is crucial, which can help the tracker to better focus on the search area related to the target; (ii) direct fusion of RGB and event modalities may ignore the problem that one of the modalities cannot provide valuable semantic information in some cases. In the FM module, target-aware scanning module (TAS) and cross-modal scanning module (CMS) are introduced respectively to solve the above problems.

[0061] Target-aware scanning module (TAS)

[0062] The target-aware scanning module enhances the interaction between the template region and the search region within the modality through the global modeling capability of the state space model. This module adopts a bidirectional scanning mechanism to calculate the correlation between the template features and the search features. After obtaining the features of a certain layer in the multi-modal encoder wherein denote the RGB image feature and event image feature, respectively, and the formula of the TAS module is shown as follows:

[0063]

[0064] wherein, denotes the linear projection layer; denotes the layer normalization operation; σ is the SiLU activation function; ψ is the 1-dimensional convolution operation; and ⊙ denotes the element-wise multiplication; is the projection of ; is the gating signal; denotes the forward scanning state space model, denotes the backward scanning state space model; Y f and Y b are the outputs through the bidirectional scanning (i.e. and ); is the output of the target-aware scanning module after jointly modeling the template region feature and the search region feature in the nth layer encoder, denotes the RGB image feature in the nth layer encoder, denotes the event image feature in the nth layer encoder, and denotes the forward.

[0065] Cross-modal scanning module (CMS)

[0066] The model performs inter-modal interaction in the CMS module, and proposes a simple and effective cross-fusion scheme based on a gating mechanism to fuse semantic information and inter-modal complementarity. Unlike the target-aware scanning module, the cross-modal scanning module only takes the search feature as the input to prevent interference of the template feature information. Taking the RGB image modality as an example, given the RGB image search region feature in the nth layer The cross-modal scanning module is defined as follows:

[0067]

[0068] wherein the modality-specific gating signal and are used to guide the interaction between modalities, thereby enhancing the main modality and suppressing the degenerate modality. Through the target-aware scanning and cross-modal scanning modules proposed in the present patent, the multi-modal encoder can effectively promote the intra-modal and cross-modal interaction. By adding the RGB and event features of the last layer encoder, the final fusion feature is obtained.

[0069] (3) History information decoder construction

[0070] The history information decoder perceives the motion trend and appearance change of the target by decoding the history information, thereby assisting the model to make accurate prediction. Each layer of the history information decoder proposed in the model is composed of two key components: a history state perception module and a feature sequence attention module.

[0071] History state perception module (HSA)

[0072] The history state perception module is used for modeling the history information in a video sequence, and a sequence modeling module based on a state space model architecture is proposed, which models the target state of N frames in a sequence as N one-dimensional learnable vectors Q(q1,…,qN)∈RN. N N×D where D is the encoding dimension. The sequence is modeled by the state space model and Q is output. ′ ′ ′ N N×D , the formula is as follows:

[0073]

[0074] is a linear projection layer, is a normalization layer, and Q is output. ′ Subsequently, it is input as a feature sequence attention module. Thanks to the efficient time series modeling mechanism of the state space model, the hidden state generated by the query vector Q at each time is passed to the next time and participates in the prediction of the subsequent time. In this way, the long-term effectiveness of the history clue modeling is ensured. Compared with the autoregressive model based on the attention mechanism, the inference and training speed is significantly improved.

[0075] Feature sequence attention module (FSA)

[0076] The feature sequence attention module aims to combine the multi-modal spatial features and the historical time features Q ′ to improve the confidence of the prediction head. The FSA module of the project adopts a multi-head attention mechanism, in which the history information sequence is used as the query and the multi-modal features are used as the key and value. The formula is as follows:

[0077]

[0078] is the Q ″ processed by a feedforward network (FFN) and input into the subsequent decoding layer to generate the final time query Finally, we combine the spatial and temporal information through the following matrix multiplication method, thereby avoiding the introduction of additional parameters:

[0079] ​​​​

[0080] where F out represents the multi-modal feature fused with historical temporal information and highlights the possible location of the target. Subsequently F out These features are input into the tracking head for the prediction of the target bounding box.

[0081] Network training

[0082] For the MamTrack model proposed in the present application, a pre-trained HiViT model on the ImageNet dataset is used to initialize its parameters. The batch size of the model is set to 32. Adam is used as the optimizer to update the model parameters during training, the number of iterations is set to 50 rounds, the learning rate decay factor is set to 0.2 and decays once every 15 iterations. The learning rates of the classification head, the back head and the backbone network are set to 0.001, 0.001 and 0.0001, respectively. The success rate (SR) and the precision rate (PR) commonly used in target tracking are used as the validation indicators of the present model, where the success rate focuses on the proportion of the regression boxes predicted by the model and the real regression boxes of the target greater than a given threshold, and the PR focuses on the proportion of the center positions of the target predicted by the model and the real center positions of the target less than a given threshold. Comparative experiments are carried out on the VisEvent dataset, the FE108 dataset and the FELT dataset, which are all multi-modal single target tracking datasets based on RGB-event images. The images are captured by DAVIS346 time cameras with an image resolution of 346x260, a dynamic range of 120dB and a camera delay of 20μs. The VisEvent dataset is the largest RGB-event multi-modal target tracking dataset at present, containing 820 video sequences, of which 500 sequences are used for training and 320 sequences are used for testing. The VisEvent dataset is used to verify the performance of the present model in outdoor scenes. The FE108 dataset contains 108 video sequences, of which 76 sequences are used for training and 32 sequences are used for testing. The FE108 dataset is used to verify the tracking performance of the present model in indoor scenes. The FELT is a RGB-event target tracking dataset for long-time tracking, containing 742 video sequences, of which 1,594,474 RGB-event images contain 45 types of indoor and outdoor targets. 520 sequences are used for training and 222 video sequences are used for testing. The FELT dataset is used to verify the performance of the present model in long-time tracking. The experimental results of the present model are as follows:

[0083] As shown in the accompanying Figure 4 Fig. 2, the present model achieves a success rate of 61.6% and a precision rate of 79.2% on the VisEvent dataset, exceeding the current SOTA models AQATrack and SDSTrack, proving the robustness of the model in the present application. As shown in the accompanyingFigure 5 As shown in the table, the success rate of the model on the FELT dataset is 48.9%, and the accuracy is 60.8%, which is higher than the current SOTA model AiATrack and BAT, verifying the effectiveness of the model in long-time tracking; the details are shown in the following table. Figure 6 As shown in the table, the success rate of the model on the FE108 dataset is 66.4%, and the accuracy is 94.2%, reaching a new SOTA effect, further verifying the generalization ability of the model in different scenarios.

Claims

1. A multimodal target tracking method based on a state-space model, characterized in that, The steps are as follows: Step 1: RGB-Event Image Data Processing First, image data under various challenging scenarios are collected by calibrated event cameras and RGB cameras. Event streams generated by the event cameras are superimposed into event image frames according to a certain time step. Image alignment and filtering are performed according to the frequency of the RGB cameras to obtain event images. Then, the RGB images and event images are fed into the model after preprocessing. Step 2: Construct a multimodal encoder The multimodal encoder is used to extract information from RGB images and event images. It adopts the HiViT attention network with a vertical architecture as the multimodal encoder, and extracts features layer by layer through token encoding, token merging and self-attention mechanism. Between the intermediate attention layers, the modality fusion module FM based on the state space model is inserted to perform cross-modal fusion. Finally, the extracted final layer features are sent to the historical information decoder and prediction head for the next prediction. Step 2.1: Construction of the modal fusion module based on the state-space model A modal fusion module FM based on a state-space model is constructed to fuse semantic information from multiple modalities. The FM module introduces a target perception scanning module and a cross-modal scanning module respectively. (1) Target perception scanning module The target perception scanning module utilizes the global modeling capability of the state space model to enhance the interaction between the template region and the search region within the modality. The target perception scanning module adopts a bidirectional state space model scanning mechanism to calculate the correlation between template features and search features. (2) Cross-modal scanning module A cross-modal scanning module is established based on a state-space model-based gating mechanism to cross-fuse two modes; Through the target-aware scanning module and the cross-modal scanning module, the multimodal encoder can facilitate intramodal and cross-modal interactions; the RGB and event features of the last layer encoder are added together to obtain the final fused features. Final fusion features It will be used to predict the target location after decoding historical information; Step 3: Construction of Historical Information Decoder The historical information decoder perceives the target's movement trend and appearance changes by decoding historical information, thereby assisting the model in making accurate predictions; each layer of the historical information decoder includes a historical state perception module and a feature sequence attention module. The historical state awareness module is used to model historical information in the video sequence, employing a sequence modeling module based on a state-space model. The output of the historical state awareness module will then be used as input to the feature sequence attention module along with the multimodal features in step 2. By merging data, the target within the current search area can be accurately located; Step 4: Combine multimodal features The output F of the historical information decoder out The data is input into the tracking head for prediction of the target bounding box.

2. The multimodal target tracking method based on a state-space model according to claim 1, characterized in that, Definition of the target-aware scanning module: in, Indicates a linear projection layer; The denotes layer normalization operation; σ is the SiLU activation function; ψ is a 1D convolution operation; ⊙ represents element-wise multiplication; This represents the features of a certain layer in a multimodal encoder. These represent RGB image features and event image features, respectively. yes The projection; It is a gating signal; This represents the forward scan state space model. Represents the backward scan state space model, Y f and Y b It is through the forward and backward directions of SSM (i.e. and The output of ); It is the output of the target-aware scanning module in the nth layer encoder after jointly modeling the template region features and the search region features. This represents the template region features in the nth layer encoder. This represents the search region features in the nth layer encoder.

3. The multimodal target tracking method based on a state-space model according to claim 1, characterized in that, Definition of cross-modal scan module: Taking RGB modal images as an example, where, This represents the features of the RGB image after being encoded by the target-aware scanning module at layer n. Y represents the RGB image search region features in the nth layer. ′ express Intermediate features after scanning using the state-space model. This represents the fusion features extracted from the RGB image in layer n after passing through the cross-modal scanning module, and the modality-specific gating signal. and It is used to guide the interaction between modalities, thereby enhancing the dominant modality while suppressing the degenerate modality.

4. The multimodal target tracking method based on a state-space model according to claim 1, characterized in that, The historical information decoder is specifically as follows: (1) Historical Status Awareness Module The target state of N frames in the sequence is modeled as N one-dimensional learnable vectors Q(q1,…,q). N )∈R N×D Where D is the encoding dimension; the sequence is modeled using a state-space model and Q is output. ′ =(q ′ ,…,q ′ N )∈R N×D The formula is as follows: Among them, X Q Z represents the input features after Q-projection. Q This indicates a query for the gated feature after Q-projection. Representing a state-space model; For linear projection layers, For normalization layer, (2) Definition of the feature sequence attention module: Among them, Q ″ This represents the characteristics of historical information scanned by the historical state perception module. H represents the vector concatenation operation. i This represents the intermediate features after self-attention. This represents the attention mechanism, where q represents the attention query, k represents the attention key, and v represents the attention value. as well as All are learnable parameters, d m Indicates the input dimension d q d k and d v Let n represent the vector dimensions of q, k, and v, respectively. h The length of the token. It is the Softmax function, Q ″ After processing by the feedforward network (FFN), the data is input into the subsequent decoding layer to generate the query for the next time step. Finally, spatial and temporal information are combined using the following matrix multiplication method: Among them, F out It represents multimodal features that incorporate historical time-series information and highlights the possible location of the target.

Images