A multi-modal target tracking method based on sparse space-time marker guidance

Abstract

The application discloses a kind of sparse space-time mark guided multimodal target tracking method, joint feature extraction, modal fusion and template search matching are carried out by network model, and sparse space-time mark is introduced in multiple levels to embed target appearance change information to complete accurate target tracking.First stage, according to the target frame of initial frame, the search area image and template image of two modalities are cut respectively, then the template search image is encoded and projected to sequence feature space, the template and search sequence features of two modalities are spliced together, and sent into stacked sequence feature coding layer;Second stage, sparse space-time mark from historical frame is introduced in preset coding layer, and cross attention is carried out with the search area of respective modal respectively to embed target change information, third stage, the search mark features of two modalities extracted by multi-layer coding are spliced together and sent into tracking prediction head to obtain the coordinate of the response point with the highest score and the value of regression frame, finally, the tracking result coordinate and boundary frame of actual target are obtained by relative transformation.

Description

Technical Field

[0001] This invention relates to multimodal single-target tracking technology, specifically to a multimodal target tracking method guided by sparse spatiotemporal labels. Background Technology

[0002] With the rapid iteration of deep neural network technology and its deep integration with computer vision technology, visual target tracking, one of the many fundamental research topics in the field of computer vision, has achieved significant breakthroughs in both theoretical and methodological innovation and practical application expansion. As a core technology of intelligent perception systems, VOT (Virtual Object Tracking) provides crucial data support for subsequent intelligent decision-making by establishing a spatiotemporal correlation model of target motion behavior. From an application perspective, this technology has formed a multi-dimensional application ecosystem: in the public safety field, its deep integration with intelligent transportation systems has effectively improved traffic monitoring efficiency and pedestrian detection accuracy; in the national defense and military field, it has constructed a key technology closed loop for battlefield situational awareness, providing technical support for tasks such as strategic target identification and ballistic trajectory prediction. Furthermore, in human-computer interaction scenarios, the integration of behavior recognition and motion capture technologies has driven paradigm shifts in intelligent interaction systems. The core task of visual target tracking is to locate and scale a specified target in continuous video frames, establishing data support for subsequent research on the dynamic characteristics and motion patterns of the target.

[0003] Traditional single-modal tracking is limited by the specific limitations of each mode, such as the light sensitivity of visible light imaging and the thermal sensitivity of thermal infrared imaging. This has led to increased attention being paid to multimodal target tracking due to its modal complementarity. However, most multimodal tracking research focuses on modal fusion design, neglecting effective handling of changes in target appearance. While some methods incorporate historical frames or fuse and replace initial templates to include temporal information, these methods risk destroying the original target appearance information and accumulating errors over time. Furthermore, current multimodal tracking work employs a three-stage paradigm, a discrete process that leads to breaks in cross-stage information flow, making end-to-end modal complementarity mining difficult. Simultaneously, most multimodal temporal methods attempting to incorporate complete historical features ignore the potential for introducing more background noise and increasing computational cost, both of which significantly impact tracking efficiency.

[0004] In summary, existing multimodal target tracking methods have the following problems:

[0005] 1) Most existing studies use a multi-stage tracking framework, which affects tracking efficiency. The discrete stages also prevent the full fusion of multimodal information.

[0006] 2) Multimodal scenarios with multiple challenges require more consideration of factors that change in the target, and existing methods ignore the importance of temporal information.

[0007] 3) Most video frames carry a lot of repetitive information. The existing time-series methods will bring a lot of computational burden to the network by inputting complete historical frames, which will affect the tracking efficiency. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a multimodal target tracking method based on sparse spatiotemporal markers, which can track targets more accurately in complex scenarios where targets are changing.

[0009] Technical Solution: This invention provides a multimodal target tracking method based on sparse spatiotemporal labeling, which inputs a prepared multimodal image into a backbone network (for feature extraction, modality fusion, and target search and matching). The modal image includes an RGB template image. RGB search image TIR template image and TIR search images H and W represent the height and width of the image, respectively, and z and x represent the template image and the search image, respectively. Upon receiving the multimodal image, the following steps are performed:

[0010] Step (1): For the input multimodal image, map its two-dimensional data into one-dimensional multi-channel feature sequence labels, as follows:

[0011] First, the corresponding image is completely split into patch sequences at a resolution of P×P=16×16. in P represents the length of the split template and the searched patch sequence. 2 Indicates the pixel area of ​​the patch at its resolution;

[0012] Then, using a parameter The projection layer projects the patch sequence into a D-dimensional latent space, while embedding two learnable locations. and The patch sequence is added separately to the template and the search region embeddings, and finally the final feature label sequence is obtained.

[0013] Step (2): The feature label sequence The sequence is then assembled into a complete label sequence H1 (which is then fed into the backbone network).

[0014] Step (3): Input the complete labeled sequence H1 into the encoding layer (i.e., the Transformer encoding layer). First, it undergoes layer normalization, so that the labeled sequences in step (2) have a similar distribution before processing. Then, it completes joint feature extraction, modality fusion, and target matching through multi-head self-attention. The specific method is as follows:

[0015] First, the labeled sequence H1 is subjected to multiple linear projections to generate Q, K, V matrices, and then self-attention calculation is performed. The self-attention result is then added to the original labeled sequence H1 to obtain an intermediate result G′, which is fed into the layer for normalization and then passed through an MLP (Multi-Layer Perceptron). Finally, it is added to G′ to obtain the output H of the current coding layer. i+1 :

[0016] Next, the sparse spatiotemporal marker M containing historical frames is... rgb M rgb The data are fed into the sparse spatiotemporal labeling module for temporal embedding, and then cross-attention processing is performed between the data and the search features.

[0017] Finally, for the tracking process that does not provide sparse spatiotemporal labels for historical frames, sparse labels are filtered through attention results to serve as candidates for future tracking frames;

[0018] Step (4): After processing by the remaining coding layers in step (3), the search feature labels of the two modalities are separated separately, then concatenated together and sent to the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, the sparse spatiotemporal label extracted in the current frame is saved and sent to the next tracking frame for processing. The output of the tracking head is transformed to obtain the target bounding box.

[0019] Furthermore, the backbone network is based on the Transformer network and includes 12 coding layers. The 12 coding layers are stacked together, and sparse spatiotemporal labeling modules are embedded after the 4th, 7th and 10th coding layers, respectively, to completely concatenate the sequence labels of the two modalities and input them into a single-stream network.

[0020] Furthermore, the formula for mapping the original multimodal image to a one-dimensional multichannel feature sequence label in step (1) is as follows:

[0021]

[0022] Furthermore, the detailed method of step (2) is as follows:

[0023] The four patch sequences are embedded and concatenated to construct a complete label sequence H1, as shown in the following formula, where Concat is the concatenation operation:

[0024]

[0025] Furthermore, the detailed process of step (3) is as follows:

[0026] Step (3.1): First, perform multiple linear projections on the input label sequence H1 to generate Q, K, V matrices, and then perform self-attention calculation, as shown in the following formula:

[0027] H′ i =LN(H i ),

[0028]

[0029] In the above formula, LN(·) represents the layer normalization operation, which controls the range of feature distribution through normalization; H′ represents the intermediate result of sequence features after normalization; W Q W K W V For a linear projection layer, the subscript i of H indicates which encoder level the output comes from, where A represents the attention weight matrix, G represents the output of multi-head self-attention, and C represents the channel dimension of K;

[0030] The self-attention calculation result is then added to the original label sequence to obtain the intermediate result G′, which is further fed into the layer for normalization and then passed through an MLP (Multi-Layer Perceptron). Finally, it is added to G′ to obtain the output H of the current coding layer. i+1 It is also the input to the next coding layer:

[0031] G′=G+H i ,

[0032] H i+1 =MLP(LN(G′))+G′,

[0033] Step (3.2): The sparse spatiotemporal marker M containing historical frames... rgb M rgb The data is fed into a sparse spatiotemporal labeling module for temporal embedding, where it undergoes cross-attention processing with the search features. Here, taking visible light modalities as an example, modal labeling is ignored, and the current time frame search feature label is used as the query Q. x The subscript indicates that the search region is from which the sparse spatiotemporal marker is used as the key value K. M V M The subscript indicates that it comes from a historical frame, and C represents K. M The channel dimension, and its process is represented as follows:

[0034]

[0035] In the above formula, H represents the output result of the coding layer. i+1 The search feature labels extracted from the data are represented by CA(Input1,Input2), which signifies the cross-attention process. Input1 is used to form the query matrix, while Input2 is used to obtain the key-value matrix. The processing of the two modalities and their concatenation and merging can be simplified as follows:

[0036]

[0037] In the above formula, Concat(...) represents the process of concatenating the templates and search tags of two modalities;

[0038] Step (3.3): For tracking processes that do not provide sparse spatiotemporal markers for historical frames, sparse markers are selected as candidates for future tracking frames based on the attention results. The attention matrix is ​​represented as follows, where the left part of the right side of the equals sign represents the visible light modal template z. rgb Attention matrix obtained from query:

[0039]

[0040] In the above formula, A represents the attention weight matrix. The attention result of the visible light modal templates to the search region is represented by the following: The search attention of the templates of the two modalities is extracted, and the mean of their channel dimension and multi-head dimension is taken. Then, the Top K labels are selected as sparse spatiotemporal labels. The extraction process can be represented as follows:

[0041]

[0042] In the above formula, mean(...) represents taking the average value, TopK(...) represents selecting the top K labels after sorting by attention weights, and M... rgb M rgb The superscript of the sparse spatiotemporal marker indicates the current embedding level.

[0043] The sparse spatiotemporal label M of the historical frames in step (3.2) above rgb M rgb There are multiple pairs, and they are applied to different layers of the network. The sparse spatiotemporal label comes from the previous tracking frame, and the sparse label is filtered and sent to the next frame every 50 frames.

[0044] The sparse label selection process in step (3.3) is repeated at multiple levels, with the value of K controlled by a retention ratio parameter of 0.7. The specific process is as follows:

[0045] In step (3.3), the length of the initial search marker is the complete number, and the size of the marker sequence is B×L×C=12×256×768; B represents the batch size, C represents the channel, and L represents the length of the marker sequence; the initial screening K value is 256*0.7≈179, and the number of root screening modules follows the same pattern.

[0046] Beneficial Effects: This invention proposes an end-to-end multimodal temporal tracking network that unifies tracking from three stages to one stage, achieving a significant improvement in tracking efficiency. Simultaneously, this network introduces sparse spatiotemporal markers to handle target appearance changes. It adaptively selects advantageous feature markers by filtering sparse spatiotemporal markers based on global attention weights, and then embeds temporal information hierarchically through multi-level sparse filtering. This achieves the effect of integrating temporal features into multiple stages of the network and ensuring full interaction while maintaining tracking efficiency. Compared with existing technologies, this invention has the following advantages:

[0047] (1) Based on the discrete disadvantage of the multi-stage framework, this invention unifies the multimodal input into a single-stage network. The joint modal interaction, feature extraction and template matching process are completed through a unified-stage Transformer temporal tracking network. Furthermore, the sparse selection strategy is guided by joint attention weights, which effectively improves the tracking efficiency while maintaining the tracking accuracy.

[0048] (2) This invention proposes a sparse spatiotemporal labeling method, which introduces multi-level sparse spatiotemporal labels to embed temporal information in a hierarchical manner to focus on changes in the appearance of the target. It completes the accurate matching of target features in the search features through a small number of key feature labels, and tracks the target more accurately in scenarios with complex target changes.

[0049] (3) This invention adopts a unified end-to-end training strategy, using temporal sampling to train the temporal module, in order to optimize the binding relationship between target changes and tracking results. This effectively adapts to the challenges of target appearance changes and improves tracking performance. Attached Figure Description

[0050] Figure 1 This is an overall flowchart of the present invention;

[0051] Figure 2 This is a schematic diagram of the network model in an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram of the sparse spatiotemporal labeling module according to an embodiment of the present invention.

[0053] Figure 4 This is a schematic diagram of the sparse label extraction process in an embodiment of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0055] The present invention provides a sparse spatiotemporal marker-guided multimodal target tracking method, which uses multimodal visible light and thermal infrared images as input, performs joint feature extraction, modality fusion and template search matching through a network model, and introduces sparse spatiotemporal markers at multiple levels to embed target appearance change information to achieve accurate target tracking.

[0056] In the first stage, the search region images and template images of the two modalities are cropped according to the target bounding boxes of the initial frame. Then, the template search image is encoded and projected into the sequence feature space. The template sequence features and search sequence features of the two modalities are concatenated together and fed into a stacked sequence feature encoding layer. The encoding layer is normalized and then fed into a multi-head self-attention layer to complete feature extraction, modality fusion and target search matching.

[0057] In the second stage, sparse spatiotemporal markers from historical frames are introduced into several preset coding layers and cross-attention is performed with the search regions of their respective modalities to embed information about target changes.

[0058] In the third stage, the search marker features of the two modalities extracted through multi-layer encoding are concatenated together and fed into the tracking prediction head to obtain the coordinates of the highest-scoring response point and the value of the regression box. Finally, the tracking result coordinates and bounding box of the actual target are obtained through relative transformation.

[0059] like Figures 1 to 4 As shown, in this embodiment, the prepared multimodal images are input into the backbone network, and the modal images include RGB template images. RGB search image TIR template image and TIR search images H and W represent the height and width of the image, respectively, and z and x represent the template image and the search image, respectively. The backbone network is based on the Transformer network and consists of 12 coding layers stacked together. Sparse spatiotemporal labeling modules are embedded after the 4th, 7th, and 10th coding layers, respectively, to completely concatenate the sequence labels of the two modalities and input them into a single-stream network.

[0060] After receiving the multimodal image, perform the following steps:

[0061] Step (1): For the input multimodal image, map its two-dimensional data into one-dimensional multi-channel feature sequence labels, as follows:

[0062] First, the corresponding image is completely split into patch sequences at a resolution of P×P=16×16. in P represents the length of the split template and the searched patch sequence. 2 Indicates the pixel area of ​​the patch at its resolution;

[0063] Then, using a parameter The projection layer projects the patch sequence into a D-dimensional latent space, while embedding two learnable locations. and The patch sequence is added separately to the template and the search region embeddings, and finally the final feature label sequence is obtained.

[0064] Step (2): For the feature marker sequence obtained in step (1), the patch sequence is concatenated into a complete marker sequence H1;

[0065] Step (3): Input the complete labeled sequence H1 into the encoding layer. First, it undergoes layer normalization, so that the labeled sequences in step (2) have similar distributions before processing. Then, joint feature extraction, modality fusion, and target matching are completed through multi-head self-attention. The data flow of the backbone network is described here. The final output is the feature sequence encoded by the repeated encoding layers, which is used to input the tracking head to obtain the final coordinates and target boxes. The specific method is as follows:

[0066] First, the labeled sequence H1 is linearly projected multiple times to generate Q, K, V matrices, and then self-attention calculation is performed. Next, the self-attention result is added to the original labeled sequence H1 to obtain an intermediate result G′, which is then fed into the layer for normalization and passed through an MLP. Finally, it is added to G′ to obtain the output H of the current coding layer. i+1 :

[0067] Next, the sparse spatiotemporal marker M containing historical frames is... rgb M rgb The data are fed into the sparse spatiotemporal labeling module for temporal embedding, and then cross-attention processing is performed between the data and the search features.

[0068] Finally, for the tracking process that does not provide sparse spatiotemporal labels for historical frames, sparse labels are filtered through attention results to serve as candidates for future tracking frames;

[0069] Step (4): After processing by the remaining coding layers in step (3), the search feature labels of the two modalities are separated separately, then concatenated together and sent to the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, the sparse spatiotemporal label extracted in the current frame is saved and sent to the next tracking frame for processing. The output of the tracking head is transformed to obtain the target bounding box.

[0070] The formula for mapping the original multimodal image to a one-dimensional multichannel feature sequence label in step (1) is as follows:

[0071]

[0072] Step (2) embeds the four patch sequences and concatenates them to construct a complete label sequence H1, as shown in the following formula, where Concat is the concatenation operation:

[0073]

[0074] The detailed process of step (3) is as follows:

[0075] Step (3.1): First, perform multiple linear projections on the input label sequence H1 to generate Q, K, V matrices, and then perform self-attention calculation, as shown in the following formula:

[0076] H′ i =LN(H i ),

[0077]

[0078] In the above formula, LN(·) represents the layer normalization operation, H′ represents the intermediate result of the sequence features after normalization; W Q W K W V For a linear projection layer, the subscript i of H indicates which encoder level the output comes from, A represents the attention weight matrix, G represents the output of multi-head self-attention, and C represents the channel dimension of K.

[0079] Then, the self-attention calculation result is added to the original label sequence to obtain the intermediate result G′, which is further fed into the layer for normalization and then passed through the MLP. Finally, it is added to G′ to obtain the output H of the current coding layer. i+1 It is also the input to the next coding layer:

[0080] G′=G+H i ,

[0081] H i+1 =MLP(LN(G′))+G′,

[0082] Step (3.2): The sparse spatiotemporal marker M containing historical frames... rgb M rgn The data are fed into the sparse spatiotemporal labeling module for temporal embedding, and then cross-attention processing is performed between the data and the search features.

[0083] For visible light modal images, the current time frame is used as the search feature marker for query Q. x The subscript indicates that the search region is from which the sparse spatiotemporal marker is used as the key value K. M V MThe subscript indicates that it comes from a historical frame, and C represents K. M The channel dimension, and its process is represented as follows:

[0084]

[0085] In the above formula, H represents the output result of the coding layer. i+1 The search feature labels extracted from the input are represented by CA(Input1,Input2), which indicates the cross-attention process. Input1 forms the query matrix, and Input2 yields the key-value matrix. The processing of the two modalities and their concatenation and merging are shown below:

[0086]

[0087] In the above formula, Concat(...) represents the process of concatenating the templates and search tags of two modalities;

[0088] Step (3.3): For tracking processes that do not provide sparse spatiotemporal labels for historical frames, sparse labels are filtered through the attention results to serve as candidates for future tracking frames. The attention matrix is ​​represented as follows:

[0089]

[0090] In the above formula, A represents the attention weight matrix; The attention result of the visible light modal templates to the search region is represented by the following: The search attention of the templates of the two modalities is extracted, and the mean of their channel dimension and multi-head dimension is taken. Then, the Top K labels are selected as sparse spatiotemporal labels. The extraction process can be represented as follows:

[0091]

[0092] In the above formula, mean(...) represents taking the average value, TopK(...) represents selecting the top K labels after sorting by attention weights, and M... rgb M rgb The superscript of the sparse spatiotemporal marker indicates the current embedding level.

[0093] Step (1): Use the prepared multimodal image data as input to the backbone network. There are a total of four images, which are RGB template images. RGB search image TIR template image and TIR search images

[0094] First, the image is completely divided into patch sequences at a resolution of P×P = 16×16. in P represents the length of the split template and the searched patch sequence. 2 This indicates the pixel area of ​​the patch at its resolution. Then, a parameter is used... The projection layer projects the patch sequence into a D-dimensional latent space. It also includes two learnable positional embeddings. and The patch sequence is added separately to the template and the search region embeddings, and finally the final feature label sequence is obtained. The above process maps two-dimensional image data into one-dimensional multi-channel feature sequence labels, which can be described by the following formula:

[0095]

[0096] Step (2): For the feature label sequence obtained in step (1), the patch sequence embedding is prepared to be sent into the backbone network after the following preliminary processing. The four patch sequence embeddings are concatenated to construct a complete label sequence H1, as shown in the following formula, where Concat is the concatenation operation:

[0097]

[0098] Step (3): Input the labeled sequence into the encoding layer. First, it undergoes layer normalization, so that the labeled sequences in step (2) have a similar distribution before processing. Then, multi-head self-attention is used to complete joint feature extraction, modality fusion, and target matching. The specific method is as follows:

[0099] Step (3.1): First, perform multiple linear projections on the input sequence to generate Q, K, V matrices, and then perform self-attention calculation, as shown in the following formula:

[0100] H′ i =LN(H i ),

[0101]

[0102] In the above formula, LN(·) represents the layer normalization operation, which controls the range of feature distribution through normalization, and H′ represents the intermediate result of sequence features after normalization. W Q W K W VFor a linear projection layer, the subscript i in H indicates which encoder layer's output it comes from, where A represents the attention weight matrix, G represents the output of multi-head self-attention, and C represents the channel dimension of K. The output is then added to the original label sequence to obtain the intermediate result G′, which is further fed into the layer for normalization and then passed through an MLP (Multi-Layer Perceptron). Finally, it is added to G′ to obtain the output H of the current encoding layer. i+1 It is also the input to the next coding layer:

[0103] G′=G+H i ,

[0104] H i+1 =MLP(LN(G′))+G′,

[0105] Step (3.2): The sparse spatiotemporal marker M containing historical frames... rgb M rgn Input the sparse spatiotemporal labeling module and perform cross-attention processing with the search features. Here, taking visible light modalities as an example, modal labeling is ignored, and the search feature label of the current time frame is used as the query Q. x The subscript indicates that the search region is from which the sparse spatiotemporal marker is used as the key value K. M V M The subscript indicates that it comes from a historical frame, and C represents K. M The channel dimension, and its process can be represented as follows:

[0106]

[0107] In the above formula, H represents the output result of the coding layer. i+1 The search feature labels extracted from the data are represented by CA(Input1,Input2), which indicates the cross-attention process. Input1 is used to form the query matrix, while Input2 is used to obtain the key-value matrix. The processing of the two modalities and their concatenation and merging can be simplified as follows:

[0108]

[0109] In the above formula, Concat(...) represents the process of concatenating the templates and search tags of two modalities;

[0110] Step (3.3): For tracking processes that do not provide sparse spatiotemporal markers for historical frames, we filter sparse markers as candidates for future tracking frames using the attention results. The attention matrix can be represented as follows, where the left part of the right side of the equals sign represents the visible light modal template z. rgb Attention matrix obtained from query:

[0111]

[0112] In the above formula, A represents the attention weight matrix, where... For example, this represents the attention result of the visible light modal template on the search region. The search attention of the template pairs for the two modalities is extracted separately, and the average of their channel dimension and multi-head dimension is taken. Then, the Top K labels are selected as sparse spatiotemporal labels. The extraction process can be represented as follows:

[0113]

[0114] In the above formula, mean(...) represents taking the average value, TopK(...) represents selecting the top K labels after sorting by attention weights, and M... rgb M rgb The superscript of the sparse spatiotemporal marker indicates the current embedding level;

[0115] Step (4): After processing by the remaining coding layers in step (3), the search feature labels of the two modalities are separated separately, then concatenated together and sent to the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, the sparse spatiotemporal label extracted in the current frame is saved and sent to the next tracking frame for processing. The output of the tracking head is transformed to obtain the target bounding box;

[0116] The backbone network mentioned in step (2) of this embodiment is a Transformer network constructed by stacking twelve coding layers and embedding sparse spatiotemporal labeling modules after layers 4, 7, and 10. The sequence labels of the two modalities are completely concatenated and input into a single-stream network. Among these 12 coding layers, layers 1, 2, 3, 5, 6, 8, 9, 11, and 12 are regular coding layers, layers 4, 7, and 10 are coding layers with embedded sparse spatiotemporal labeling modules, and the remaining coding layers refer to the layers 11 and 12 after the last embedded module coding layer (layer 10) to obtain the output of the backbone network. The twelve coding layers and the final tracking head form a complete single-stream network model.

[0117] In step (3.2) of this embodiment, the sparse spatiotemporal marker M of the historical frame rgb M rgb There are multiple pairs, and they are applied to different layers of the network. The sparse spatiotemporal label comes from the previous tracking frame, and the sparse label is filtered and sent to the next frame every 50 frames.

[0118] In step (3.3) of this embodiment, the process of filtering sparse markers is repeated at multiple levels, and the value of K is controlled by a retention ratio parameter of 0.7. The specific working process is as follows:

[0119] Step (3.3): For the initial search marker, its length is the complete number, and the size of the marker sequence is B×L×C=12×256×768; B represents the batch size, C represents the channel, and L represents the length of the marker sequence; the initial screening K value is 256*0.7≈179, and the number of root screening modules follows the same pattern.

[0120] To verify the rationality and effectiveness of the technical solution of this invention, this embodiment selects the RGBT234 dataset and the LasHeR dataset for experiments, and uses Precision Rate (PR) and Success Rate (SR) as evaluation metrics for quantitative performance assessment, and Frames Per Second (FPS) as an efficiency metric. PR measures the percentage of all frames where the distance between the tracking result's center point and the ground reality is less than a threshold; a representative PR score is calculated by setting the threshold to 20 pixels across the three datasets. SR measures the percentage of successfully tracked frames with overlap greater than a threshold; a representative SR score is calculated using the area under the curve.

[0121] This embodiment is implemented based on the deep learning framework PyTorch and utilizes a graphics processing unit (GPU) to accelerate computation, with 20GB of memory and an Nvidia GeForce GTX 2080Ti graphics card.

[0122] The RGBT234 dataset contains 234 RGBT-high aligned video pairs, totaling approximately 234K frames, and is annotated with 12 attributes to facilitate analysis of the effectiveness of different tracking algorithms for various challenges. LasHeR is currently the largest RGBT tracking dataset, containing 1224 aligned video sequences, including more diverse attribute annotations. 245 sequences are separately allocated to the test set, with the remainder used for the training set.

[0123] Table 1. Detection results for the RGBT234 and LasHeR datasets.

[0124]

[0125] The experimental results in Table 1 show that the technical solution of the present invention performs well on both datasets, and the target tracking method of the present invention is significantly better than existing similar methods in multimodal target tracking tasks.

Claims

1. A multimodal target tracking method based on sparse spatiotemporal labeling guidance, characterized in that, The prepared multimodal images are input into the backbone network, including RGB template images. RGB search image TIR template image and TIR search images H and W represent the height and width of the image, respectively, and z and x represent the template image and the search image, respectively; after receiving the multimodal image, the following steps are performed: Step (1): For the input multimodal image, map its two-dimensional data into one-dimensional multi-channel feature sequence labels, as follows: First, the corresponding image is split into a patch sequence. in P represents the length of the split template and the searched patch sequence. 2 Indicates the pixel area of ​​the patch at its resolution; Then, using a parameter The projection layer projects the patch sequence into a D-dimensional latent space, while embedding two learnable locations. and The patch sequence is added separately to the template and the search region to obtain the final feature label sequence. Step (2): The feature label sequence Concatenate them into a complete label sequence H1; Step (3): Input the complete labeled sequence H1 into the encoding layer, first perform layer normalization, and then complete joint feature extraction, modality fusion, and search target matching through multi-head self-attention; the specific method is as follows: First, the labeled sequence H1 is linearly projected multiple times to generate Q, K, V matrices, and then self-attention calculation is performed. Next, the self-attention result is added to the original labeled sequence H1 to obtain an intermediate result G′, which is then fed into the layer for normalization and passed through an MLP. Finally, it is added to G′ to obtain the output H of the current coding layer. i+1 : Next, the sparse spatiotemporal marker M containing historical frames is... rgb M rgb The data are fed into the sparse spatiotemporal labeling module for temporal embedding, and then cross-attention processing is performed between the data and the search features. Finally, for tracking processes that do not provide sparse spatiotemporal labels for historical frames, sparse labels are filtered through attention results to serve as candidates for future tracking frames; Step (4): After processing by the remaining coding layers in step (3), the search feature labels of the two modalities are separated separately, then concatenated together and sent to the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, the sparse spatiotemporal label extracted in the current frame is saved and sent to the next tracking frame for processing. The output of the tracking head is transformed to obtain the target bounding box.

2. The multimodal target tracking method based on sparse spatiotemporal marker guidance according to claim 1, characterized in that, The backbone network is based on the Transformer network and includes 12 coding layers. The 12 coding layers are stacked together, and sparse spatiotemporal labeling modules are embedded after the 4th, 7th and 10th coding layers, respectively, to completely concatenate the sequence labels of the two modalities and input them into a single-stream network.

3. The multimodal target tracking method based on sparse spatiotemporal marker guidance according to claim 1, characterized in that, The formula for mapping the original multimodal image to a one-dimensional multichannel feature sequence label in step (1) is as follows:

4. The multimodal target tracking method based on sparse spatiotemporal marker guidance according to claim 1, characterized in that, Step (2) embeds the four patch sequences and concatenates them to construct a complete label sequence H1, as shown in the following formula, where Concat is the concatenation operation:

5. The multimodal target tracking method based on sparse spatiotemporal marker guidance according to claim 1, characterized in that, The detailed process of step (3) is as follows: Step (3.1): First, perform multiple linear projections on the input label sequence H1 to generate Q, K, V matrices, and then perform self-attention calculation, as shown in the following formula: H′ i =LN(H i ), In the above formula, LN(·) represents the layer normalization operation, H′ represents the intermediate result of the sequence features after normalization; W Q W K W V For a linear projection layer, the subscript i of H indicates which encoder level the output comes from, A represents the attention weight matrix, G represents the output of multi-head self-attention, and C represents the channel dimension of K. Then, the self-attention calculation result is added to the original label sequence H′ to obtain the intermediate result G′, which is fed into the layer for normalization and then passed through the MLP. Finally, it is added to G′ to obtain the output H of the current coding layer. i+1 ; G′=G+H i , H i+1 =MLP(LN(G′))+G′, Step (3.2): The sparse spatiotemporal marker M containing historical frames... rgb M rgb The data are fed into the sparse spatiotemporal labeling module for temporal embedding, and then cross-attention processing is performed between the data and the search features. For visible light modal images, the current time frame is used as the search feature marker for query Q. x sparse spatiotemporal labels are used as key values ​​K M V M C represents K M The channel dimension, and its process is represented as follows: In the above formula, H represents the output result of the coding layer. i+1 The search feature labels extracted from the input are represented by CA(Input1,Input2), which indicates the cross-attention process. Input1 forms the query matrix, and Input2 yields the key-value matrix. The processing of the two modalities and their concatenation and merging are shown below: In the above formula, Concat(...) represents the process of concatenating the template tags and search tags of two modalities; Step (3.3): For tracking processes that do not provide sparse spatiotemporal labels for historical frames, sparse labels are selected as candidates for future tracking frames based on the attention results. The attention matrix is ​​represented as follows: In the above formula, A represents the attention weight matrix; The attention result of the visible light modal templates to the search region is represented by the following: The search attention of the templates of the two modalities is extracted, and the mean of their channel dimension and multi-head dimension is taken. Then, the Top K labels are selected as sparse spatiotemporal labels. The extraction process is as follows: In the above formula, mean(...) represents taking the average value, TopK(...) represents selecting the top K labels after sorting by attention weights, and M... rgb M rgb The superscript of the sparse spatiotemporal marker indicates the current embedding level.

Images