An RGBT target tracking method and system based on interactive hidden state space-time information

Abstract

The application discloses an RGBT target tracking method and system based on interactive hidden state space-time information, and the method comprises the following steps: combining an RGB mode image and a TIR mode image to construct an RGBT image sample dataset and acquire an RGBT image coordinate sequence pair; introducing a shared vocabulary table unit, a feature extraction module with an interactive bridge and an interactive mode fusion module to construct an RGBT target tracking model; performing target tracking prediction on the RGBT target tracking model based on the RGBT image sample dataset and the RGBT image coordinate sequence pair to obtain a target tracking prediction result. The application can perform channel feature interaction and fusion between two independent modes, more effectively utilize space-time information of a tracking sequence and improve the accuracy of target tracking in a complex scene. The application can be widely applied to the technical field of computer vision target tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision target tracking, and in particular to an RGBT target tracking method and system based on interactive hidden state space-time information. BACKGROUND

[0002] RGBT target tracking provides a new possibility for realizing target tracking in all-weather and all-environment by combining the advantages of visible light (RGB) and thermal infrared (TIR) images. In RGBT target tracking, the feature interaction mode of the two modalities is very important to the performance of the tracker. A target tracking sequence has rich temporal information. When target tracking is performed, the target state and the background change over time. Making full use of the temporal information of the sequence can improve the performance of the tracker. With the development of technology, TCTrack efficiently introduces temporal information in the feature dimension by using an improved OnlineTAdaConv, and corrects the feature map through the accumulated temporal information. ARTrack models the target trajectory in a time autoregressive manner by regarding tracking as a coordinate sequence interpretation task, thereby realizing cross-frame tracking of the target. STMT introduces a dynamic template, realizes the interaction of the search area and the template image through an attention mechanism, and embeds time information to solve target appearance changes. HIPTrack encodes high-quality historical target features and generates effective historical cues for tracking. TATrack obtains time information by referencing an online updated template, and performs multi-modal feature extraction and cross-modal interaction in the initial template branch and the online template branch. Although the above-mentioned trackers have achieved certain improvement in performance, they do not make full use of the context information between consecutive frames, resulting in unsatisfactory tracking results in complex scenes. SUMMARY

[0003] To solve the above technical problems, the purpose of the present application is to provide an RGBT target tracking method and system based on interactive hidden state space-time information, which can perform channel feature interaction and fusion between two independent modalities and more effectively utilize the space-time information of the tracking sequence, thereby improving the accuracy of target tracking in complex scenes.

[0004] The first technical solution adopted by the present application is: an RGBT target tracking method based on interactive hidden state space-time information, comprising the following steps:

[0005] An RGBT image sample data set is constructed by combining RGB modality images and TIR modality images, and an RGBT image coordinate sequence pair is obtained, which includes an RGB modality image coordinate sequence and a TIR modality image coordinate sequence;

[0006] A shared vocabulary unit, a feature extraction module with an interaction bridge, and an interactive modality fusion module are introduced to construct an RGBT target tracking model.

[0007] training the RGBT target tracking model based on the RGBT image sample dataset and the RGBT image coordinate sequence pair to obtain optimal model parameters;

[0008] embedding the optimal model parameters into the RGBT target tracking model and performing target tracking prediction on the RGBT image sample dataset and the RGBT image coordinate sequence pair to obtain a target tracking prediction result.

[0009] Further, the RGBT target tracking model specifically includes a patch embedding layer, a shared vocabulary, a feature extraction module, and an interactive modal fusion module, wherein:

[0010] The shared vocabulary has a plurality of coordinate tokens, the plurality of coordinate tokens are marked as trajectory cues, the tokens connecting past trajectory coordinates are connected to construct a shared vocabulary with autoregressive target past position information;

[0011] The feature extraction module includes a plurality of encoders and an interactive bridge;

[0012] The interactive modal fusion module includes a channel feature extraction module, a spatial feature extraction module, a first SoftMax function, and a second SoftMax function.

[0013] Further, the interactive bridge specifically includes a first Mamba component and a second Mamba component, the first Mamba component and the second Mamba component each include a first linear layer, a second linear layer, a one-dimensional convolution layer, an activation function, a state space model, and a third linear layer, the output end of the first linear layer is connected to the input end of the one-dimensional convolution layer, the output end of the second linear layer is connected to the input end of the activation function, the output end of the one-dimensional convolution layer is connected to the input end of the state space model, the output end of the state space model, the output end of the activation function, and the input end of the third linear layer are connected.

[0014] Further, the step of training the RGBT target tracking model based on the RGBT image sample dataset and the RGBT image coordinate sequence pair to obtain optimal model parameters specifically includes:

[0015] cropping the RGBT image sample dataset to obtain an initial template image and a search region image;

[0016] performing coordinate mapping processing on the RGBT image coordinate sequence pair to obtain mapped RGBT image coordinates;

[0017] inputting the initial template image, the search region image, and the mapped RGBT image coordinates into the RGBT target tracking model for target tracking to obtain a target bounding box;

[0018] track the loss value between the prediction loss of the bounding box of the target and the real label, and perform target tracking training on the RGBT target tracking model based on the RGBT image sample data set and the RGBT image coordinate sequence until the tracking loss value reaches a minimum value, and output optimal model parameters.

[0019] Further, the step of inputting the initial template image, the search region image, and the mapped RGBT image coordinates into the RGBT target tracking model for target tracking to obtain the bounding box of the target specifically includes:

[0020] inputting the initial template image, the search region image, and the mapped RGBT image coordinates into the RGBT target tracking model;

[0021] based on the patch embedding layer of the RGBT target tracking model, performing segmentation processing on the initial template image, the search region image, and the mapped RGBT image coordinates to obtain a plurality of patch sequences;

[0022] based on the shared vocabulary table of the RGBT target tracking model, performing coordinate discretization and mapping processing on the plurality of patch sequences to obtain RGB modality features and TIR modality features;

[0023] based on the feature extraction module of the RGBT target tracking model, performing spatiotemporal information interaction processing on the RGB modality features and the TIR modality features to obtain interactive historical hidden state information of the RGB modality and interactive historical hidden state information of the TIR modality;

[0024] based on the interactive modality fusion module of the RGBT target tracking model, performing modality feature fusion processing on the interactive historical hidden state information of the RGB modality and the interactive historical hidden state information of the TIR modality to obtain the bounding box of the target.

[0025] Further, the step of based on the feature extraction module of the RGBT target tracking model, performing spatiotemporal information interaction processing on the RGB modality features and the TIR modality features to obtain interactive historical hidden state information of the RGB modality and interactive historical hidden state information of the TIR modality specifically includes:

[0026] inputting the RGB modality features and the TIR modality features into the feature extraction module of the RGBT target tracking model;

[0027] based on the encoder of the feature extraction module, performing encoding processing on the RGB modality features and the TIR modality features to obtain encoded RGB modality features and encoded TIR modality features;

[0028] The interaction bridge based on the feature extraction module interacts the encoded RGB modality feature and the encoded TIR modality feature in time context information to obtain the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality.

[0029] Further, the step of the interaction bridge based on the feature extraction module interacting the encoded RGB modality feature and the encoded TIR modality feature in time context information to obtain the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality specifically includes:

[0030] The encoded RGB modality feature and the encoded TIR modality feature are input to the interaction bridge of the feature extraction module.

[0031] The first linear layer and the one-dimensional convolution layer of the interaction bridge are used to respectively map and convolve the encoded RGB modality feature and the encoded TIR modality feature to obtain the preprocessed RGB modality feature and the preprocessed TIR modality feature.

[0032] The state space model based on the interaction bridge is used to respectively perform dimension lifting on the preprocessed RGB modality feature and the preprocessed TIR modality feature to obtain time scale parameters and continuous parameters.

[0033] The continuous parameters are discretized to obtain the hidden state of the previous frame of the RGB modality feature and the TIR modality feature, and the hidden state of the current frame of the RGB modality feature and the TIR modality feature is updated in combination with the time scale parameters to obtain the RGB modality feature with time context information and the TIR modality feature with time context information.

[0034] The second linear layer and the activation function of the interaction bridge are used to perform sequence space-time information calculation on the encoded RGB modality feature through the RGB modality feature with time context information and on the encoded TIR modality feature through the TIR modality feature with time context information, and dimension reduction processing is performed through the third linear layer of the interaction bridge to obtain the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality.

[0035] Further, the step of the interaction modality fusion module based on the RGBT target tracking model performing modality feature fusion processing on the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality to obtain the bounding box of the target specifically includes:

[0036] The interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality are input to the interaction modality fusion module of the RGBT target tracking model.

[0037] The channel feature extraction module based on the interactive modality fusion module performs averaging, max pooling, and concatenation on the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality respectively to obtain aggregated channel features.

[0038] Based on the spatial feature extraction module of the interaction modality fusion module, the interaction history hidden state information of RGB modality and the interaction history hidden state information of TIR modality are averaged, max-pooled and concatenated to obtain aggregated spatial features.

[0039] Based on the first SoftMax function of the interactive modal fusion module, modal weights are calculated on the aggregated channel features to obtain the corresponding modal channel weights;

[0040] Based on the second SoftMax function of the interactive modality fusion module, modal weights are calculated on the aggregated space features to obtain the corresponding modal space weights;

[0041] The corresponding modal channel weights and the corresponding modal space weights are multiplied and added to the corresponding modal features respectively to obtain the bounding box of the target.

[0042] Furthermore, the step of embedding the optimal model parameters into the RGBT target tracking model and performing target tracking prediction on the RGBT image sample dataset and RGBT image coordinate sequence pairs to obtain the target tracking prediction result specifically includes:

[0043] The optimal model parameters are embedded into the RGBT target tracking model to obtain the optimal RGBT target tracking model;

[0044] The initial template image, the search region image, and the mapped RGBT image coordinates are concatenated to construct the original feature information;

[0045] The original feature information is input into the optimal RGBT target tracking model for feature interaction. If the current frame is the first frame, the hidden state information of the current frame is directly retained. Otherwise, the hidden information of the current frame is interacted with the hidden information of the previous frame, and then the hidden state information of the current frame is updated to obtain the interactive feature information.

[0046] The interactive feature information is processed with hybrid channel attention weights and spatial attention weights to obtain the target coordinate information of the current frame;

[0047] The target coordinates of the current frame, the hidden state of the current frame, and the search image of the next frame are input into the optimal RGBT target tracking model for iterative tracking and prediction until the iteration conditions are met, and the target tracking prediction result is output.

[0048] The second technical solution of the present application is: an RGBT target tracking system based on interactive hidden state space-time information, comprising:

[0049] A first module is configured to combine RGB modal images and TIR modal images to construct an RGBT image sample dataset, and obtain an RGBT image coordinate sequence pair, wherein the RGBT image coordinate sequence pair comprises an RGB modal image coordinate sequence and a TIR modal image coordinate sequence.

[0050] A second module is configured to introduce a shared vocabulary unit, a feature extraction module with an interactive bridge, and an interactive modal fusion module, and construct an RGBT target tracking model.

[0051] A third module is configured to perform target tracking training on the RGBT target tracking model based on the RGBT image sample dataset and the RGBT image coordinate sequence pair, and obtain optimal model parameters.

[0052] A fourth module is configured to embed the optimal model parameters into the RGBT target tracking model, and perform target tracking prediction on the RGBT image sample dataset and the RGBT image coordinate sequence pair, to obtain a target tracking prediction result.

[0053] The present application has the following advantages: the present application combines RGB modal images and TIR modal images to construct an RGBT image sample dataset, and then introduces a shared vocabulary unit, a feature extraction module with an interactive bridge, and an interactive modal fusion module to construct an RGBT target tracking model. The interactive bridge uses a Mamba component to perform channel feature interaction and fusion between two independent modalities, and performs interaction and updates based on the information of the hidden state of the previous frame, so that the model can more effectively utilize the space-time information of the tracking sequence. The interactive modal fusion module further refines the channel information importance and spatial importance of each feature under the guidance of the input feature, and fully mixes the channel attention weight and the spatial attention weight to ensure effective fusion of key information and strengthen the model's representation ability for the target, thereby improving the performance of the tracker. Finally, the RGBT target tracking model is trained and predicted based on the RGBT image sample dataset and the RGBT image coordinate sequence pair, thereby improving the accuracy of target tracking prediction. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a step flowchart of an RGBT target tracking method based on interactive hidden state space-time information according to the present application;

[0055] Figure 2 is a structural block diagram of an RGBT target tracking system based on interactive hidden state space-time information according to the present application;

[0056] Figure 3is a schematic diagram of an RGBT tracker network structure provided by the embodiment of the present application;

[0057] Figure 4 is a schematic diagram of an interaction bridge structure provided by the embodiment of the present application;

[0058] Figure 5 is a schematic diagram of the structure of the interaction modality fusion module provided by the embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] First of all, it needs to be pointed out that RGBT target tracking is one of the important research directions in the field of computer vision, which combines the advantages of visible light (RGB) and thermal infrared (TIR) images to provide new possibilities for achieving all-weather and all-environment target tracking. RGB images perform well in daily lighting conditions with their rich color and detail information, while TIR images exhibit their unique penetrating power and stability at night or in adverse weather conditions. This modality fusion not only makes up for the shortcomings of a single modality under certain conditions, but also provides more accurate target positioning and motion estimation in complex scenes. RGBT tracking has wide application potential in the fields of security monitoring, autonomous driving, etc.

[0061] In related technologies, in RGBT target tracking, the interaction mode of features of two modalities is important for the performance of the tracker. HATFNet designs a multi-modal feature fusion, fuses the features of two modalities according to the reliability of the modalities, and fully utilizes the positive role of the high-quality modality. The DFAT network proposes an online adaptive decision-level fusion method to extract complementary information conveyed by two modalities. The DuSiamRT network enhances the recognition ability of the target appearance by fusing the feature information of two modalities. The CBPNet network realizes bidirectional feature selection by enhancing the interaction between multi-modal features, thereby improving the robustness of the tracker. The APFNet network uses a progressive fusion module to gradually interact and fuse the shallow and deep features of two modalities, promoting the efficiency of modality fusion. The ADRNet network transfers discriminative features from one modality to another, which helps to enhance the discriminative ability of the weak modality. ViPT adopts the prompt-tuning method, effectively fusing different perception data sources. TBSI bridges the cross-modal interaction between the RGB and TIR search areas by collecting and distributing target-related objects and environmental context. The BAT network effectively realizes the cross-modal prompt between the modalities through the Bi-directionalAdapter structure.

[0062] However, the related art cannot fully utilize the spatiotemporal information of the tracking sequence, resulting in performance degradation in complex scenes.

[0063] Therefore, the embodiment of the present application utilizes the autoregressive target past coordinate position information to construct an RGBT tracking model; based on the Mamba component, the channel feature interaction and fusion between two independent modalities are performed, and the interaction and update are performed based on the information of the hidden state of the previous frame, so that the model can more effectively utilize the spatiotemporal information of the tracking sequence; a modality fusion module is designed to further refine the importance mapping of the channel information and spatial information of the input modality, enhance the important feature information, so that the model can better effectively fuse the key information between modalities, and strengthen the representation ability of the model to the target, thereby improving the performance of the tracker.

[0064] With reference to Figure 1 The present application provides an RGBT target tracking method based on interactive hidden state spatiotemporal information, which comprises the following steps:

[0065] S100, combining an RGB modality image and a TIR modality image to construct an RGBT image sample data set, and obtaining an RGBT image coordinate sequence pair, wherein the RGBT image coordinate sequence pair comprises an RGB modality image coordinate sequence and a TIR modality image coordinate sequence;

[0066] In this embodiment, a publicly available visible light-thermal infrared RGBT data sample set is obtained and divided to obtain a training set and a test set.

[0067] S200, introduces a shared vocabulary unit, a feature extraction module with interactive bridges, and an interactive modality fusion module to construct an RGBT target tracking model;

[0068] Specifically, such as Figure 3 As shown, the RGBT target tracking model specifically includes a patch embedding layer, a shared vocabulary, a feature extraction module, and an interactive modality fusion module. The shared vocabulary contains several coordinate terms, which are marked as trajectory cues. Terms connecting past trajectory coordinates are used to construct a shared vocabulary with autoregressive target past position information. The feature extraction module includes several encoders and interactive bridges. Figure 5 As shown, the interactive modal fusion module includes a channel feature extraction module, a spatial feature extraction module, a first SoftMax function, and a second SoftMax function.

[0069] Furthermore, such as Figure 4 As shown, the interactive bridge specifically includes a first Mamba component and a second Mamba component. Both the first Mamba component and the second Mamba component include a first linear layer, a second linear layer, a one-dimensional convolutional layer, an activation function, a state space model, and a third linear layer. The output of the first linear layer is connected to the input of the one-dimensional convolutional layer, the output of the second linear layer is connected to the input of the activation function, the output of the one-dimensional convolutional layer is connected to the input of the state space model, and the outputs of the state space model and the activation function are connected to the input of the third linear layer.

[0070] In this embodiment, a pre-trained RGB tracker is used as a baseline. An RGBT target tracking model is constructed using autoregressive target past position information. First, the pre-trained RGB tracker is extended to support a bimodal RGBT tracker, specifically a visible light RGB mode and a thermal infrared (TIR) ​​mode. The RGB tracker is extended into a weight-shared bimodal tracker, and features are extracted for both the visible light RGB mode and the thermal infrared (TIR) ​​mode. Coordinate tokens are introduced into the RGBT tracker, and a shared vocabulary is used to mark these tokens as trajectory cues. By connecting the tokens with past trajectory coordinates, an RGBT target tracking model with autoregressive target past position information is obtained. This modeling is expressed as conditional probability, and the specific calculation process is described below:

[0071] p(Y t |Y t-N:t-1 ,(C,Z,Xt ))

[0072] where Z and X t denote the given template and search image at time step t, C as command tokens, Y denotes the target sequence associated with X, Y t-N:t-1 is the coordinate token Token of the past frame, to obtain a more accurate coordinate prediction Yt. First, the coordinate token Token is obtained from the estimated probability distribution. Since the sampling is not differentiable, the coordinate is represented by the expectation of the distribution.

[0073] S300, based on the RGBT image sample data set and the RGBT image coordinate sequence pair, the target tracking model is trained for target tracking, and the optimal model parameter is obtained;

[0074] S310, the RGBT image sample data set is cropped to obtain an initial template image and a search area image;

[0075] S320, the RGBT image coordinate sequence pair is processed for coordinate mapping to obtain the mapped RGBT image coordinates;

[0076] S330, the initial template image, the search area image and the mapped RGBT image coordinates are input into the RGBT target tracking model for target tracking to obtain the target bounding box;

[0077] Specifically, the initial template image, the search area image and the mapped RGBT image coordinates are input into the RGBT target tracking model; based on the patch embedding layer of the RGBT target tracking model, the initial template image, the search area image and the mapped RGBT image coordinates are segmented to obtain a plurality of patch sequences; based on the shared vocabulary of the RGBT target tracking model, the plurality of patch sequences are processed for coordinate discretization and mapping to obtain the RGB modal feature and the TIR modal feature; based on the feature extraction module of the RGBT target tracking model, the RGB modal feature and the TIR modal feature are processed for spatiotemporal information interaction to obtain the interactive history hidden state information of the RGB modal and the interactive history hidden state information of the TIR modal; based on the interactive modal fusion module of the RGBT target tracking model, the interactive history hidden state information of the RGB modal and the interactive history hidden state information of the TIR modal are processed for modal feature fusion to obtain the target bounding box.

[0078] In the embodiment, the initial template image, the search region image and the initial coordinate information are taken as the input of the final RGBT target tracking model to predict the motion state of the target in the subsequent frame. First, the input image is segmented into P*P size blocks through patch embedding, and then the blocks are flattened into 6 patch sequences. The continuous coordinates are discretized using a shared vocabulary, and the quantized token index is a learnable vocabulary to obtain the token corresponding to the coordinates. Then, the features of the initial template, the search image and the target coordinate information of the two modalities are obtained through a linear projection layer respectively. Finally, the token of the RGB and TIR modalities is spliced along the channel dimension, and then sent to the encoder for processing.

[0079] Further, for the feature extraction module of the RGBT target tracking model, the RGB modality feature and the TIR modality feature are input into the feature extraction module of the RGBT target tracking model; based on the encoder of the feature extraction module, the RGB modality feature and the TIR modality feature are encoded to obtain the encoded RGB modality feature and the encoded TIR modality feature; based on the interaction bridge of the feature extraction module, the encoded RGB modality feature and the encoded TIR modality feature are interacted with the time context information to obtain the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality.

[0080] Further, for the interaction bridge of the feature extraction module, the encoded RGB modality feature and the encoded TIR modality feature are input into the interaction bridge of the feature extraction module; based on the first linear layer and the one-dimensional convolution layer of the interaction bridge, the encoded RGB modality feature and the encoded TIR modality feature are respectively mapped and convolved to obtain the preprocessed RGB modality feature and the preprocessed TIR modality feature; based on the state space model of the interaction bridge, the preprocessed RGB modality feature and the preprocessed TIR modality feature are respectively processed in the dimension to obtain the time scale parameter and the continuous parameter; the continuous parameter is discretized to obtain the hidden state of the previous frame of the RGB modality feature and the TIR modality feature, and the hidden state of the current frame of the RGB modality feature and the TIR modality feature is updated by combining the time scale parameter to obtain the RGB modality feature with time context information and the TIR modality feature with time context information; based on the second linear layer and the activation function of the interaction bridge, the encoded RGB modality feature is calculated through the RGB modality feature with time context information, and the encoded TIR modality feature is calculated through the TIR modality feature with time context information, and the dimension is reduced through the third linear layer of the interaction bridge to obtain the interactive history hidden state information of the RGB modality and the interactive history hidden state information of the TIR modality.

[0081] It should be noted that, since the tracking sequence possesses rich spatiotemporal information, the proper utilization of this information can improve the tracker's performance. The interactive bridge uses the Mamba component to perform feature interaction and fusion between two independent modalities, and interacts and updates based on the hidden state information of the previous frame, enabling the model to more effectively utilize the spatiotemporal information of the tracking sequence. During tracking, the interactive bridge retains the hidden state information of the previous frame, then interacts with the current frame to update the hidden state information of the current frame. This embodiment of the invention utilizes the Mamba component to perform feature interaction and fusion between two independent modalities, and interacts and updates based on the hidden state information of the previous frame. This allows the model to more effectively utilize the spatiotemporal information of the tracking sequence. This feature interaction method maximizes the utilization of the sequence's spatiotemporal information.

[0082] In this embodiment, the interaction bridge consists of two independent Mamba components. Each Mamba component processes the input of one modality. The modal input to the Mamba component is then increased in dimension through a linear layer and further split into a gated branch z and a main branch x. The main branch x is then subjected to 1D convolution and SiLU activation. The main branch x is then fed into the state-space model SSM. The main branch x is further increased in dimension to obtain the time-scale parameter Δ and continuous parameters A and B. The continuous parameters A and B are then discretized. Finally, the hidden state H from the previous frame is used... t-1 The hidden state H of the current frame is updated based on the input x' of the current frame. t Perform the above operations on both modalities simultaneously to obtain the time context information output for both modalities. and For the gated branch z, we perform SiLU activation on it. Then we use... and The gated branches z for each modality are calculated. Then, linear layers are used to reduce dimensionality. The hidden states of the two modalities are updated, and the gated branches z are used to complement features and suppress redundant features, effectively utilizing the spatiotemporal information of the sequence. The specific calculation process is shown below:

[0083] (z RGB ,x RGB =Chunk(Liner(Norm(X)) RGB )))

[0084] (z TIR ,x TIR =Chunk(Liner(Norm(X)) TIR )))

[0085] x = SiLU(Conv1d(x))

[0086] Δ,A,B=Split(Liner(x′))

[0087] Delta = Softplus(Liner(Delta))

[0088]

[0089]

[0090]

[0091]

[0092] where the modal inputs are and where Norm is a root mean square normalization layer, Liner is a linear layer that boosts the dimension, Chunk is an operation that splits the dimension after boosting the dimension, (z I ,x I ), I e (RGB, TIR) are the two branches after splitting of each modality, Conv1d is a 1D convolution, discrete parameters and are obtained by discretizing the continuous parameters A and B, the hidden state of the previous frame is H t-1 , based on the current frame input x', the hidden state of the current frame H t , is the output of obtaining the temporal context information, the temporal context information outputs of the two modalities are and and are respectively the outputs of the historical hidden state information between the two modalities.

[0093] For the interactive modal fusion module of the RGBT target tracking model, the interactive history hidden state information of the RGB modal and the interactive history hidden state information of the TIR modal are input into the interactive modal fusion module of the RGBT target tracking model; based on the channel feature extraction module of the interactive modal fusion module, the interactive history hidden state information of the RGB modal and the interactive history hidden state information of the TIR modal are respectively subjected to average and maximum pooling and splicing processing to obtain aggregated channel features; based on the spatial feature extraction module of the interactive modal fusion module, the interactive history hidden state information of the RGB modal and the interactive history hidden state information of the TIR modal are respectively subjected to average and maximum pooling and splicing processing to obtain aggregated spatial features; based on the first SoftMax function of the interactive modal fusion module, the modal weight value of the aggregated channel features is calculated to obtain the corresponding modal channel weight value; based on the second SoftMax function of the interactive modal fusion module, the modal weight value of the aggregated spatial features is calculated to obtain the corresponding modal spatial weight value; the corresponding modal channel weight value and the corresponding modal spatial weight value are multiplied and added with the corresponding modal features to obtain the bounding box of the target.

[0094] It should be noted that the interactive modal fusion module further refines the channel information importance and spatial importance of each feature under the guidance of the input features, and fully mixes the channel attention weight and the spatial attention weight, so as to ensure effective fusion of key information, strengthen the representation ability of the model to the target, and thus improve the performance of the tracker. The embodiment of the present application designs a modal fusion module, which respectively performs average pooling and maximum pooling on the channel dimension and the spatial dimension of the two modalities, and then connects them, obtains the features of the two modalities through linear layers, obtains the channel weight and the spatial weight of the two modalities through the normalized exponential function, and then multiplies the weight of the two modalities with the features of the two modalities and adds them, so as to effectively fuse the features of the two modalities.

[0095] In the present embodiment, the interactive modal fusion module is used to enhance the feature information of the tracked target and strengthen the representation ability of the model to the target, so that the RGB modal and the TIR modal are represented as X RGB and X TIR First, the features of the RGB modal and the features of the TIR modal are respectively subjected to average and maximum pooling in the channel and then splicing to obtain aggregated channel features W channel , which are respectively passed through two linear layers to obtain the channel features of the two modalities, and then the SoftMax is used to make their sum equal to 1. In the spatial dimension, the features of the RGB modal and the features of the TIR modal are respectively subjected to average and maximum pooling in the spatial dimension and then splicing to obtain aggregated spatial features W spatialThe spatial features of the two modalities are obtained through two linear layers respectively, similar to the process of obtaining channel features, and then their sum is equal to 1 using SoftMax, that is, by comparing the values between them to determine the higher part between them, so as to determine the important part between the features, finally, the weights of the two modalities are multiplied with the two modal features and added, effectively fusing the features of the two modalities, and the specific calculation process is as follows:

[0096] W channel =Concat(Avg(X RGB ,X TIR ),Max(X RGB ,X TIR ))

[0097]

[0098]

[0099]

[0100] W spatial =Concat(Avg(X RGB ,X TIR ),Max(X RGB ,X TIR ))

[0101]

[0102]

[0103]

[0104]

[0105] wherein W channel is the aggregated channel feature, and are channel weights, W spatial is the aggregated spatial feature, and are spatial weights, and F fusion represents the fused feature, since the sum of the modal weights is equal to 1, the important feature information is enhanced, thereby realizing effective feature fusion.

[0106] S340, calculate the tracking loss value between the prediction loss of the bounding box of the target and the real label, loop the target tracking training of the RGBT target tracking model based on the RGBT image sample data set and the RGBT image coordinate sequence, until the tracking loss value reaches the minimum value, output the optimal model parameter.

[0107] In summary, the implementation steps of the RGBT target tracking model training are as follows:

[0108] 1) input the training sequence including RGB and TIR two modal images and their corresponding target coordinates;

[0109] 2) respectively crop the training sequence of two modal images to obtain the corresponding initial template image and search image, and map the coordinates of the original image as the input of the model;

[0110] 3) connect the initial frame template image, search image and four command words representing the target boundary together, and input them into the RGBT target tracking model for joint feature extraction and prediction to obtain the target boundary box;

[0111] 4) calculate the tracking loss between the tracking result and the real label using SIoU loss, L1 loss and cross entropy loss;

[0112] 5) the model parameters corresponding to the minimum tracking loss are used as the optimal parameters to obtain the trained RGBT target tracking model with the optimal matching ability.

[0113] 6) load the optimal parameters of the last step, and input the entire training sequence into the RGBT target tracking model;

[0114] 7) keep the predicted target position information and hidden state information of each frame in the sequence, and the input of the next frame model includes the predicted target position information and hidden state information of the last frame;

[0115] 8) update the model parameters by back propagation for each sequence, and the model parameters corresponding to the minimum tracking loss are used as the optimal parameters to obtain the final RGBT target tracking model after training.

[0116] S400, embed the optimal model parameters into the RGBT target tracking model and perform target tracking prediction on the RGBT image sample data set and the RGBT image coordinate sequence pair to obtain the target tracking prediction result.

[0117] Specifically, the optimal model parameters are embedded into the RGBT target tracking model to obtain an optimal RGBT target tracking model; the initial template image, the search region image and the mapped RGBT image coordinates are spliced to construct original feature information; the original feature information is input into the optimal RGBT target tracking model for feature interaction; if the current frame is the first frame, the hidden state information of the current frame is directly retained, otherwise the hidden information of the current frame and the previous frame is interacted, and then the hidden state information of the current frame is updated to obtain the interacted feature information; the interacted feature information is processed by mixed channel attention weight and spatial attention weight to obtain predicted current frame target coordinate information; the predicted current frame target coordinate information, the current frame hidden state information and the next frame search image are input into the optimal RGBT target tracking model for iterative tracking prediction until the iteration condition is met, and the target tracking prediction result is output.

[0118] In the embodiment, the final RGBT target tracking model is used to predict the motion state of the target in the subsequent frame, and the implementation steps are as follows:

[0119] 1) load the optimal parameters of the RGBT target tracking model trained;

[0120] 2) input the test frame image including two modalities of RGB and TIR, and target coordinate information;

[0121] 3) respectively crop the test frame image of two modalities to obtain the corresponding initial template image and search image, map the original image target coordinate information as the input of the model;

[0122] 4) connect the initial template image, the search image and the four command tokens representing the target boundary together to obtain the connected input original feature information;

[0123] 5) send the connected original feature information into the RGBT target tracking model encoder for feature interaction;

[0124] 6) send the feature into the interaction bridge, if the current frame is the first frame, the hidden state information of the current frame is directly retained, otherwise the hidden information of the current frame and the previous frame is interacted, and then the hidden state information of the current frame is updated;

[0125] 7) the two modal features passing through the interaction bridge are further mixed with channel attention weight and spatial attention weight under the guidance of the input feature through the interaction modal fusion module to enhance important feature information.

[0126] 8) predict the current frame position coordinates, take the next frame search image, the current frame target coordinate information, the four command tokens representing the target boundary and the current frame hidden state information as the original feature information, execute step 5), and when the sequence tracking ends, execute step 9).

[0127] 9) sequence prediction process ends, retaining and outputting the target tracking result.

[0128] To sum up, the embodiment of the application acquires an RGBT data sample set and divides a training set and a test set; an RGBT tracking model is constructed using autoregressive target past coordinate position information and is trained; an interactive bridge for channel feature interaction and fusion between two independent modalities is constructed based on Mamba, which is used for interaction and updating of past hidden state information; a cross-modal fusion module is constructed to further refine the importance mapping of channel information and spatial information of the modalities after interaction and enhance important feature information; optimal parameters are loaded, hidden state information is interacted and updated using the model in the process of tracking the target, and a target tracking result is acquired and output. The historical state information of the target is fully utilized, thereby improving the performance of the tracker.

[0129] Therefore, the embodiment of the application has the following advantages compared with the prior art:

[0130] 1) The Mamba component is used to perform channel feature interaction and fusion between two independent modalities, and the information of the hidden state of the previous frame is used for interaction and updating. This enables the model to more effectively utilize the spatiotemporal information of the tracking sequence.

[0131] 2) A modal fusion module is designed to further refine the importance mapping of channel information and spatial information of the input modalities and enhance important feature information. This enables the model to more effectively fuse key information between modalities and strengthen the representation ability of the model for the target, thereby improving the performance of the tracker.

[0132] Reference Figure 2 An RGBT target tracking system based on interactive hidden state spatiotemporal information, comprising:

[0133] A first module 201 is configured to construct an RGBT image sample data set by combining an RGB modality image and a TIR modality image, acquire an RGBT image coordinate sequence pair, and the RGBT image coordinate sequence pair comprises an RGB modality image coordinate sequence and a TIR modality image coordinate sequence;

[0134] A second module 202 is configured to introduce a shared vocabulary unit, a feature extraction module with an interactive bridge, and an interactive modal fusion module, and construct an RGBT target tracking model;

[0135] A third module 203 is configured to perform target tracking training on the RGBT target tracking model based on the RGBT image sample data set and the RGBT image coordinate sequence pair, and acquire optimal model parameters;

[0136] The fourth module 204 is configured to embed the optimal model parameters into the RGBT target tracking model, and perform target tracking prediction on the RGBT image sample data set and the RGBT image coordinate sequence pair to obtain a target tracking prediction result.

[0137] The content in the method embodiments is applicable to the system embodiments. The system embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0138] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application. These equivalent modifications or replacements are all included in the scope defined by the claims of the application.

Claims

1. A method for RGBT target tracking based on interactive hidden state spatio-temporal information, characterized in that, The method comprises the following steps: An RGBT image sample dataset is constructed by combining RGB modality images and TIR modality images, and an RGBT image coordinate sequence pair is obtained, which comprises an RGB modality image coordinate sequence and a TIR modality image coordinate sequence; An RGBT target tracking model is constructed by introducing a shared vocabulary unit, a feature extraction module with an interaction bridge, and an interactive modality fusion module; The RGBT target tracking model specifically comprises a patch embedding layer, a shared vocabulary, a feature extraction module, and an interactive modality fusion module, wherein: The shared vocabulary has a plurality of coordinate word units, the plurality of coordinate word units are marked as trajectory prompts, the word units connecting past trajectory coordinates are connected to construct a shared vocabulary with autoregressive target past position information; The feature extraction module comprises a plurality of encoders and an interaction bridge; The interaction bridge specifically comprises a first Mamba component and a second Mamba component, the first Mamba component and the second Mamba component each comprise a first linear layer, a second linear layer, a one-dimensional convolution layer, an activation function, a state space model, and a third linear layer, the output end of the first linear layer is connected to the input end of the one-dimensional convolution layer, the output end of the second linear layer is connected to the input end of the activation function, the output end of the one-dimensional convolution layer is connected to the input end of the state space model, and the output end of the state space model and the output end of the activation function are connected to the input end of the third linear layer; The interactive modality fusion module comprises a channel feature extraction module, a spatial feature extraction module, a first SoftMax function, and a second SoftMax function; An RGBT target tracking model is trained based on an RGBT image sample dataset and an RGBT image coordinate sequence pair to obtain optimal model parameters; The optimal model parameters are embedded into the RGBT target tracking model, and the RGBT image sample dataset and the RGBT image coordinate sequence pair are subjected to target tracking prediction to obtain a target tracking prediction result. 2.The RGBT object tracking method based on interactive hidden state spatio-temporal information according to claim 1, wherein, The step of training the RGBT target tracking model based on the RGBT image sample dataset and the RGBT image coordinate sequence pair to obtain the optimal model parameters specifically comprises: The RGBT image sample dataset is subjected to cropping processing to obtain an initial template image and a search region image; The RGBT image coordinate sequence pair is subjected to coordinate mapping processing to obtain mapped RGBT image coordinates; The initial template image, the search region image, and the mapped RGBT image coordinates are input into the RGBT target tracking model for target tracking to obtain a target bounding box; A tracking loss value between a prediction loss of the target bounding box and a real label is calculated, the RGBT target tracking model is trained based on the RGBT image sample dataset and the RGBT image coordinate sequence pair in a loop until the tracking loss value reaches a minimum value, and the optimal model parameters are output.

3. The RGBT target tracking method based on interactive hidden state spatio-temporal information according to claim 2, characterized in that, The step of inputting the initial template image, the search region image and the mapped RGBT image coordinates into the RGBT target tracking model for target tracking to obtain the bounding box of the target specifically includes: Inputting the initial template image, the search region image and the mapped RGBT image coordinates into the RGBT target tracking model; Based on the patch embedding layer of the RGBT target tracking model, the initial template image, the search region image and the mapped RGBT image coordinates are segmented to obtain a plurality of patch sequences; Based on the shared vocabulary table of the RGBT target tracking model, the plurality of patch sequences are coordinate-discretized and mapped to obtain RGB modality features and TIR modality features; Based on the feature extraction module of the RGBT target tracking model, the RGB modality features and the TIR modality features are interactively processed in space-time information to obtain interactive historical hidden state information of the RGB modality and interactive historical hidden state information of the TIR modality; Based on the interactive modality fusion module of the RGBT target tracking model, the interactive historical hidden state information of the RGB modality and the interactive historical hidden state information of the TIR modality are modality feature fused to obtain the bounding box of the target.

4. The RGBT target tracking method based on interactive hidden state spatio-temporal information according to claim 3, characterized in that, The step of interactively processing the RGB modality features and the TIR modality features in space-time information based on the feature extraction module of the RGBT target tracking model to obtain interactive historical hidden state information of the RGB modality and interactive historical hidden state information of the TIR modality specifically includes: Inputting the RGB modality features and the TIR modality features into the feature extraction module of the RGBT target tracking model; Based on the encoder of the feature extraction module, the RGB modality features and the TIR modality features are encoded to obtain encoded RGB modality features and encoded TIR modality features; Based on the interaction bridge of the feature extraction module, the encoded RGB modality features and the encoded TIR modality features are interactively processed in time context information to obtain interactive historical hidden state information of the RGB modality and interactive historical hidden state information of the TIR modality.

5. The RGBT target tracking method based on interactive hidden state spatio-temporal information according to claim 4, characterized in that, The step of interactively processing the encoded RGB modality features and the encoded TIR modality features in time context information based on the interaction bridge of the feature extraction module to obtain interactive historical hidden state information of the RGB modality and interactive historical hidden state information of the TIR modality specifically includes: Inputting the encoded RGB modality features and the encoded TIR modality features into the interaction bridge of the feature extraction module; Based on the first linear layer and the one-dimensional convolution layer of the interaction bridge, the encoded RGB modality features and the encoded TIR modality features are respectively mapped and convoluted to obtain preprocessed RGB modality features and preprocessed TIR modality features; Based on the state space model of the interaction bridge, the preprocessed RGB modality features and the preprocessed TIR modality features are respectively processed in dimension to obtain time scale parameters and continuous parameters; Discretize the continuous parameters to obtain the hidden state of the previous frame of the RGB modality feature and the TIR modality feature, and update the hidden state of the current frame of the RGB modality feature and the TIR modality feature by combining the time scale parameter, to obtain the RGB modality feature with time context information and the TIR modality feature with time context information; Based on the second linear layer of the interaction bridge and the activation function, the sequence space-time information of the encoded RGB modality feature through the RGB modality feature with time context information and the sequence space-time information of the encoded TIR modality feature through the TIR modality feature with time context information are calculated, and the dimension reduction processing is performed through the third linear layer of the interaction bridge, to obtain the interaction history hidden state information of the RGB modality and the interaction history hidden state information of the TIR modality.

6. The RGBT target tracking method based on interactive hidden state spatio-temporal information according to claim 5, characterized in that, The interaction modality fusion module based on the RGBT target tracking model, the specific steps of which for performing modality feature fusion processing on the interaction history hidden state information of the RGB modality and the interaction history hidden state information of the TIR modality to obtain the bounding box of the target include: Inputting the interaction history hidden state information of the RGB modality and the interaction history hidden state information of the TIR modality into the interaction modality fusion module of the RGBT target tracking model; Based on the channel feature extraction module of the interaction modality fusion module, performing average and maximum pooling and splicing processing on the interaction history hidden state information of the RGB modality and the interaction history hidden state information of the TIR modality respectively to obtain the aggregated channel feature; Based on the spatial feature extraction module of the interaction modality fusion module, performing average and maximum pooling and splicing processing on the interaction history hidden state information of the RGB modality and the interaction history hidden state information of the TIR modality respectively to obtain the aggregated spatial feature; Based on the first SoftMax function of the interaction modality fusion module, performing modality weight calculation on the aggregated channel feature to obtain the corresponding modality channel weight; Based on the second SoftMax function of the interaction modality fusion module, performing modality weight calculation on the aggregated spatial feature to obtain the corresponding modality spatial weight; Multiplying and adding the corresponding modality channel weight and the corresponding modality spatial weight with the corresponding modality feature respectively to obtain the bounding box of the target.

7. The RGBT target tracking method based on interactive hidden state spatio-temporal information according to claim 6, characterized in that, The specific steps of embedding the optimal model parameters into the RGBT target tracking model and performing target tracking prediction on the RGBT image sample data set and the RGBT image coordinate sequence pair to obtain the target tracking prediction result include: Embedding the optimal model parameters into the RGBT target tracking model to obtain the optimal RGBT target tracking model; Splicing the initial template image, the search area image, and the mapped RGBT image coordinates to construct the original feature information; Inputting the original feature information into the optimal RGBT target tracking model for feature interaction, if the current frame is the first frame, directly retaining the hidden state information of the current frame, otherwise, interacting the hidden information of the current frame and the previous frame, and then updating the hidden state information of the current frame to obtain the interactive feature information; The mixed channel attention weight and the spatial attention weight are processed on the interactive feature information, to obtain predicted current frame target coordinate information; The predicted current frame target coordinate information, the current frame hidden state information and the next frame search image are input into the optimal RGBT target tracking model for iterative tracking prediction until the iteration condition is met, and a target tracking prediction result is output.

8. An RGBT target tracking system based on interactive hidden state spatio-temporal information, characterized in that, The method comprises the following modules: A first module is configured to combine an RGB modality image and a TIR modality image to construct an RGBT image sample dataset, and obtain an RGBT image coordinate sequence pair, wherein the RGBT image coordinate sequence pair comprises an RGB modality image coordinate sequence and a TIR modality image coordinate sequence; A second module is configured to introduce a shared vocabulary unit, a feature extraction module with an interaction bridge and an interactive modality fusion module, and construct an RGBT target tracking model; The RGBT target tracking model comprises a patch embedding layer, a shared vocabulary, a feature extraction module and an interactive modality fusion module, wherein: The shared vocabulary comprises a plurality of coordinate vocabulary units, the plurality of coordinate vocabulary units are marked as trajectory prompts, the vocabulary units connected with past trajectory coordinates are used to construct a shared vocabulary with self-recurrent target past position information; The feature extraction module comprises a plurality of encoders and an interaction bridge; The interaction bridge comprises a first Mamba component and a second Mamba component, the first Mamba component and the second Mamba component each comprise a first linear layer, a second linear layer, a one-dimensional convolution layer, an activation function, a state space model and a third linear layer, an output end of the first linear layer is connected with an input end of the one-dimensional convolution layer, an output end of the second linear layer is connected with an input end of the activation function, an output end of the one-dimensional convolution layer is connected with an input end of the state space model, an output end of the state space model and an output end of the activation function are connected with an input end of the third linear layer; The interactive modality fusion module comprises a channel feature extraction module, a spatial feature extraction module, a first SoftMax function and a second SoftMax function; A third module is configured to perform target tracking training on the RGBT target tracking model based on the RGBT image sample dataset and the RGBT image coordinate sequence pair, and obtain optimal model parameters; A fourth module is configured to embed the optimal model parameters into the RGBT target tracking model, and perform target tracking prediction on the RGBT image sample dataset and the RGBT image coordinate sequence pair, to obtain a target tracking prediction result.

Images