Multi-modal target tracking method guided by sparse space-time mark

Abstract

The invention discloses a multi-modal target tracking method guided by sparse space-time markers, which performs joint feature extraction, modal fusion and template search matching through a network model, and introduces sparse space-time markers at multiple levels to embed target appearance change information to complete accurate target tracking. In the first stage, search area images and template images of two modalities are cut according to a target frame of an initial frame, then the template search images are coded and projected to a sequence feature space, templates and search sequence features of the two modalities are spliced together, and the spliced templates and search sequence features are sent to a stacked sequence feature coding layer; in the second stage, sparse space-time marks from historical frames are introduced into a preset coding layer, and cross attention is carried out on the sparse space-time marks and search areas of the modes of the sparse space-time marks so as to embed target change information; search mark features of two modals extracted through multilayer coding are spliced together and sent to a tracking prediction head to obtain a response point coordinate with the highest score and a value of a regression frame, and finally a tracking result coordinate and a bounding box of an actual target are obtained through relative transformation.

Description

Technical Field

[0001] The present invention relates to multi-modal single-object tracking technology, and particularly to a multi-modal object tracking method guided by sparse spatio-temporal markers. Background Art

[0002] With the rapid iteration of deep neural network technology and the deep integration of computer vision technology, visual object tracking, as one of many basic research topics in the field of computer vision, has made remarkable breakthroughs in both theoretical method innovation and practical application expansion. As one of the core technologies of intelligent perception systems, VOT provides key data support for subsequent intelligent decision-making by establishing spatio-temporal correlation models of target motion behaviors. Analyzed from the application dimension, this technology has formed a multi-dimensional application ecosystem: in the field of public security, it is deeply integrated with intelligent transportation systems, effectively improving traffic monitoring efficiency and pedestrian detection accuracy; in the field of national defense and military, it constructs a key technology closed-loop for battlefield situation awareness, providing technical support for tasks such as strategic target recognition and ballistic trajectory prediction. In addition, in the human-computer interaction scenario, by integrating behavior recognition and motion capture technologies, it has promoted the paradigm innovation of intelligent interaction systems. The core task of visual object tracking is to locate and scale the specified target in consecutive video frames, establishing data support for subsequent research on the dynamic characteristics and motion laws of the target.

[0003] Since traditional single-modal tracking is affected by the limitations of their respective modalities, such as the defect of light sensitivity in visible light imaging and the problem of thermal sensitivity in thermal infrared imaging, multi-modal object tracking has attracted the attention of many researchers due to its advantage of modality complementarity. However, most multi-modal tracking research mainly focuses on modality fusion design, while ignoring the effective processing of target appearance changes. Although some methods introduce historical frames or fuse and replace the initial template to include temporal information, they have the risk of destroying the original target appearance information and accumulating errors over time. In addition, current multi-modal tracking work all adopts a three-stage paradigm processing process, and this discrete process leads to the break of cross-stage information flow, making it difficult to achieve end-to-end modality complementarity mining. At the same time, most multi-modal temporal methods that attempt to integrate complete historical features ignore the impact of their easy introduction of more background noise and increased computational complexity, which have a great impact on tracking efficiency.

[0004] In summary, the existing multi-modal object tracking has the following problems:

[0005] 1), Most of the existing research uses a multi-stage tracking framework, which affects tracking efficiency, and the discrete stages also make it impossible to fully integrate multi-modal information.

[0006] 2), Multi-modal scenarios with various challenges need to consider more target change factors, and the existing methods ignore the importance of temporal information.

[0007] 3) Most video frames carry a large amount of repetitive information. The existing temporal methods that input the complete historical frames into the network will bring a large computational burden and affect the tracking efficiency. Summary of the Invention

[0008] Object of the Invention: The object of the present invention is to solve the deficiencies existing in the prior art and provide a multi-modal object tracking method guided by sparse spatio-temporal markers, which can track the object more accurately in complex scenes where the object changes.

[0009] Technical Solution: A multi-modal object tracking method guided by sparse spatio-temporal markers according to the present invention inputs the prepared multi-modal images into a backbone network (for feature extraction, modal fusion, and target search and matching). The modal images include an RGB template image RGB search image TIR template image and a TIR search image Let H and W represent the height and width of the image respectively, and z and x represent the template image and the search image respectively. After receiving the multi-modal images, the following steps are executed:

[0010] Step (1): For the input multi-modal images, map their two-dimensional data into a one-dimensional multi-channel feature sequence marker. The method is as follows:

[0011] First, split all the corresponding images into patch sequences according to a resolution of P×P = 16×16 where is the length of the patch sequences of the template and the search after splitting, and P 2 represents the pixel area of the resolution of the patch;

[0012] Subsequently, use a projection layer with parameters to project all the patch sequences into a D-dimensional latent space, and at the same time add two learnable position embeddings and to the patch sequence embeddings of the template and the search regions respectively. Finally, obtain the final feature marker sequence

[0013] Step (2): Concatenate the feature marker sequence into a complete marker sequence H1 (and send it into the backbone network);

[0014] Step (3): Input the complete marker sequence H1 into the encoding layer (i.e., the Transformer encoding layer). First, perform layer normalization (Layer Normalization), that is, make the marker sequence in step (2) have a similar distribution before processing. Immediately afterwards, complete joint feature extraction, modal fusion, and search target matching through multi-head self-attention. The specific method is as follows:

[0015] First, perform multiple linear projections on the token sequence H1 to generate the Q, K, and V matrices, and then perform self-attention calculations; then add the self-attention calculation results to the original token sequence H1 to obtain an intermediate result G′, send it to layer normalization and then pass through an MLP (Multi-Layer Perceptron), and finally add it to G′ to obtain the output H of the current encoding layer. i+1 :

[0016] Next, the sparse spatio-temporal tokens M rgb , M rgb are sent into the sparse spatio-temporal token module for temporal embedding, and cross-attention processing is performed with the search features respectively;

[0017] Finally, for the tracking process without providing sparse spatio-temporal tokens of historical frames, sparse tokens are screened through the attention results as candidates for future tracking frames;

[0018] In step (4), after being processed by the remaining encoding layers in step (3), the search feature tokens of the two modalities are separately split out, and then concatenated and sent into the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, the sparse spatio-temporal tokens extracted from the current frame are saved and sent into the next tracking frame for processing, and the output of the tracking head is transformed to obtain the target bounding box.

[0019] Furthermore, the backbone network is based on the Transformer network and includes 12 encoding layers. The 12 encoding layers are stacked together, and sparse spatio-temporal token modules are embedded after the 4th, 7th, and 10th encoding layers respectively. The sequence tokens of the two modalities are fully concatenated and input into a single-stream network.

[0020] Furthermore, the formula for mapping the original multi-modal image to a one-dimensional multi-channel feature sequence token in step (1) is as follows:

[0021]

[0022] Furthermore, the detailed method of step (2):

[0023] The four patch sequence embeddings are concatenated to construct a complete token sequence H1, and the process is shown in the following formula, where Concat is the concatenation operation:

[0024]

[0025] Furthermore, the detailed process of step (3) is:

[0026] Step (3.1): First, perform multiple linear projections on the input token sequence H1 to generate Q, K, and V matrices, and then perform self-attention calculation. The process is shown in the following formula:

[0027] H′ i = LN(H i ),

[0028]

[0029] In the above formula, LN(·) represents the layer normalization operation, which controls the range of feature distribution through normalization. H′ represents the intermediate result after the sequence features are normalized; W Q , W K , W V are linear projection layers. The subscript i of H indicates which encoder layer its output comes from. Among them, A represents the attention weight matrix, G represents the output of multi-head self-attention, and C represents the channel dimension of K;

[0030] Then, add the self-attention calculation result to the original token sequence to obtain the intermediate result G′, further send it to layer normalization and then through MLP (Multi-Layer Perceptron), and finally add it to G′ to obtain the output H i+1 of the current encoding layer, which is also the input of the next encoding layer:

[0031] G′ = G + H i ,

[0032] H i+1 = MLP(LN(G′)) + G′,

[0033] Step (3.2): Send the sparse spatio-temporal tokens M rgb , M rgb with historical frames into the sparse spatio-temporal token module for temporal embedding, and perform cross-attention processing with the search features respectively. Here, taking the visible light modality as an example, the modality annotation is ignored. The search feature token of the current time frame is used as the query Q x , and the subscript indicates from the search area. The sparse spatio-temporal tokens are used as the key-value K M , V M , and the subscript indicates from the historical frame. C represents the channel dimension of K M , and the process is shown as follows:

[0034]

[0035] In the above formula, represents the output result H i+1The search feature markers split from it, CA(Input1, Input2) represents the cross-attention process, where Input1 is used to form the query matrix, and Input2 is used to obtain the key-value matrix. The processing and concatenation of the two modalities can be simplified as follows:

[0036]

[0037] In the above formula, Concat(…) represents the process of concatenating the templates and search markers of the two modalities;

[0038] Step (3.3), for the tracking process without historical frame sparse spatio-temporal markers, sparse markers are screened through the attention results as candidates for future tracking frames. The attention matrix is represented as follows, where the left part on the right side of the equal sign represents the visible light modality template z rgb The attention matrix obtained by using it as a query:

[0039]

[0040] In the above formula, A represents the attention weight matrix, represents the attention result of the visible light modality template to the search area. The attention of the templates of the two modalities to the search is extracted respectively, the mean value is taken for its channel dimension and multi-head dimension, and then the top K markers are selected as sparse spatio-temporal markers. The extraction process can be represented as follows:

[0041]

[0042] In the above formula, mean(…) represents taking the mean value, TopK(…) represents selecting the top K markers after sorting by attention weights, M rgb ,M rgb represents the sparse spatio-temporal marker, and its superscript represents the current embedding level.

[0043] The sparse spatio-temporal markers M rgb ,M rgb of the historical frame in the above step (3.2) exist in multiple pairs and are applied to different levels of the network. The sparse spatio-temporal markers come from the previous tracking frame, and sparse marker screening is performed every 50 frames and sent to the next frame;

[0044] The process of screening sparse markers in step (3.3) is repeated at multiple levels, and the value of K is controlled by a retention ratio parameter of 0.7. The specific working process is as follows:

[0045] In step (3.3), for the initial search, the length of the tag is a complete number, and the size of the tag sequence is B×L×C = 12×256×768; B represents the Batchsize batch size, C represents the channel, and L represents the length of the tag sequence; the first screening K value is 256*0.7≈179, and the number of root screening modules can be deduced by analogy.

[0046] Beneficial effects: The present invention proposes an end-to-end multi-modal temporal tracking network that unifies the three stages of tracking into one stage, achieving a great improvement in tracking efficiency; at the same time, the network introduces sparse spatio-temporal tags to cope with target appearance change information, screens the sparse spatio-temporal tags according to the global attention weight to adaptively select favorable feature tags, and then hierarchically embeds the temporal information through multi-level sparse screening, so as to achieve the effect of integrating temporal features in multiple stages of the network and fully interacting while taking into account the tracking efficiency. Compared with the prior art, the present invention has the following advantages:

[0047] (1) According to the discrete disadvantage of the multi-stage framework, the present invention unifies the multi-modal input into a single-stage network, completes the joint modal interaction, feature extraction, and template matching processes through a Transformer temporal tracking network in a unified stage, and guides the sparse screening strategy through the joint attention weight, effectively improving the tracking efficiency while maintaining the tracking accuracy.

[0048] (2) The present invention proposes a sparse spatio-temporal tag method, introducing multi-level sparse spatio-temporal tags to hierarchically embed temporal information to pay attention to target appearance changes, and accurately matching the target features in the search features through a small number of key feature tags, so as to track the target more accurately in the scenario of complex target changes.

[0049] (3) The present invention adopts a unified end-to-end training strategy, trains the temporal module in the way of temporal sampling to optimize the binding relationship between target changes and tracking results. It effectively adapts to the challenges of target appearance changes and improves the tracking performance. Description of the Drawings

[0050] Figure 1 It is the overall flowchart of the present invention;

[0051] Figure 2 It is the schematic diagram of the network model of the embodiment of the present invention.

[0052] Figure 3 It is the schematic diagram of the sparse spatio-temporal tag module of the embodiment of the present invention.

[0053] Figure 4 It is the schematic diagram of the sparse tag extraction process of the embodiment of the present invention. Detailed Embodiment

[0054] The technical solution of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the described embodiments.

[0055] A multi-modal object tracking method guided by sparse spatio-temporal markers according to the present invention uses the multi-modal of visible light and thermal infrared images as inputs, performs joint feature extraction, modal fusion, and template search and matching through a network model, and introduces sparse spatio-temporal markers at multiple levels to embed target appearance change information to complete accurate object tracking.

[0056] In the first stage, according to the target box of the initial frame, the search area images and template images of the two modalities are respectively cropped, and then the template search images are encoded and projected into the sequence feature space. The template sequence features and search sequence features of the two modalities are spliced together and sent into the stacked sequence feature encoding layer. In the encoding layer, after normalization operations, they are sent into the multi-head self-attention to complete feature extraction, modal fusion, and target search and matching at the same time.

[0057] In the second stage, sparse spatio-temporal markers from historical frames are introduced in several preset encoding layers to perform cross-attention with the search areas of their respective modalities to embed the information of target changes.

[0058] In the third stage, the search marker features of the two modalities extracted through multi-layer encoding are spliced together and sent into the tracking prediction head to obtain the coordinates of the response point with the highest score and the value of the regression box. Finally, the actual target tracking result coordinates and the bounding box are obtained through relative transformation.

[0059] As Figures 1 to 4 shown, in this embodiment, the prepared multi-modal images are input into the backbone network. The modal images include the RGB template image RGB search image TIR template image and the TIR search image H and W respectively represent the height and width of the image, and z and x respectively represent the template image and the search image. The backbone network is based on the Transformer network and includes 12 encoding layers. The 12 encoding layers are stacked together, and sparse spatio-temporal marker modules are embedded after the 4th, 7th, and 10th encoding layers respectively. The sequence markers of the two modalities are completely spliced and input into a single-stream network.

[0060] After receiving the multi-modal images, the following steps are executed:

[0061] Step (1): For the input multi-modal images, map their two-dimensional data into a one-dimensional multi-channel feature sequence marker. The method is as follows:

[0062] First, all the corresponding images are split into patch sequences according to the resolution of P×P = 16×16 where The length of the patch sequence for the split template and search, P 2 represents the resolution pixel area of the patch;

[0063] Subsequently, using a parameter The projection layer projects the patch sequences into a D-dimensional latent space, while adding two learnable positional embeddings and to the patch sequence embeddings of the template and search regions respectively, and finally obtaining the final feature token sequence

[0064] Step (2): For the feature token sequence obtained in step (1), concatenate the patch sequences into a complete token sequence H1;

[0065] Step (3): Input the complete token sequence H1 into the encoding layer. First, perform layer normalization, that is, make the token sequence in step (2) have a similar distribution before processing, and then complete joint feature extraction, modality fusion, and search target matching through multi-head self-attention. The data flow of the backbone network is described here, and the finally output is the feature sequence encoded by the encoding layer repeatedly, which is used to input the tracking head to obtain the final coordinates and target box. The specific method is as follows:

[0066] First, perform multiple linear projections on the token sequence H1 to generate Q, K, and V matrices, and then perform self-attention calculation; then add the self-attention calculation result to the original token sequence H1 to obtain an intermediate result G′, send it to layer normalization and then through MLP, and finally add it to G′ to obtain the output H of the current encoding layer i+1 :

[0067] Next, send the sparse spatio-temporal tokens M rgb , M rgb existing in the historical frame into the sparse spatio-temporal token module for temporal embedding, and perform cross-attention processing with the search features respectively;

[0068] Finally, for the tracking process without providing sparse spatio-temporal tokens in the historical frame, screen the sparse tokens through the attention result as candidates for future tracking frames;

[0069] Step (4): After being processed by the remaining encoding layers in step (3), split the search feature tokens of the two modalities separately, and then concatenate them and send them into the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, save the sparse spatio-temporal tokens extracted from the current frame and send them to the next tracking frame for processing, and perform transformation operations on the tracking head output to obtain the target bounding box.

[0070] The formula for step (1) to map the original multi-modal image into a one-dimensional multi-channel feature sequence token is as follows:

[0071]

[0072] In step (2), the four patch sequences are embedded and concatenated to construct a complete token sequence H1, as shown in the following formula, where Concat is the concatenation operation:

[0073]

[0074] The detailed process of step (3) is as follows:

[0075] In step (3.1), first, the input token sequence H1 is linearly projected multiple times to generate the Q, K, and V matrices, and then the self-attention calculation is performed, as shown in the following formula:

[0076] H′ i = LN(H i ),

[0077]

[0078] In the above formula, LN(·) represents the layer normalization operation, and H′ represents the intermediate result after the sequence features are normalized; W Q , W K , W V are the linear projection layers. The subscript i of H indicates the output from which encoder layer. A represents the attention weight matrix, G represents the output of the multi-head self-attention, and C represents the channel dimension of K;

[0079] Then, the self-attention calculation result is added to the original token sequence to obtain the intermediate result G′, which is further fed into layer normalization and then through the MLP, and finally added to G′ to obtain the output H i+1 of the current encoding layer, which is also the input to the next encoding layer:

[0080] G′ = G + H i ,

[0081] H i+1 = MLP(LN(G′)) + G′,

[0082] In step (3.2), the sparse spatio-temporal tokens M rgb , M rgb with historical frames are fed into the sparse spatio-temporal token module for temporal embedding, and cross-attention processing is performed with the search features respectively;

[0083] For visible light modality images, the search feature tokens of the current time frame are used as the query Q x , and the subscript indicates from the search area. The sparse spatio-temporal tokens are used as the key-value K M , V M, the subscript indicates from the historical frame, and C represents the channel dimension of K M The process is as follows:

[0084]

[0085] In the above formula, represents the search feature signature split from the output result H of the encoding layer. CA(Input1, Input2) represents the cross-attention process. Input1 forms the query matrix, and Input2 obtains the key-value matrix. The processing processes and splicing combinations of the two modalities are as follows: i+1 The processing processes and splicing combinations of the two modalities are as follows:

[0086]

[0087] In the above formula, Concat(…) represents the process of splicing the templates and search tags of the two modalities;

[0088] Step (3.3): For the tracking process without historical frame sparse spatio-temporal tags, sparse tags are screened through the attention result as candidates for future tracking frames. The attention matrix is represented as follows:

[0089]

[0090] In the above formula, A represents the attention weight matrix; represents the attention result of the visible light modality template to the search area. The attention of the templates of the two modalities to the search is extracted respectively, the mean value is taken for its channel dimension and multi-head dimension, and then the top K tags are selected as sparse spatio-temporal tags. The extraction process can be represented as follows:

[0091]

[0092] In the above formula, mean(…) represents taking the mean value, TopK(…) represents selecting the top K tags after sorting by attention weights, M rgb , M rgb represents the sparse spatio-temporal tag, and its superscript represents the current embedding level.

[0093] Step (1): Use the prepared multi-modal image data as the input of the backbone network. There are a total of four images, namely the RGB template image RGB search image TIR template image and the TIR search image

[0094] First, all the images are split into patch sequences z according to the resolution of P×P = 16×16 rgb , where is the length of the patch sequence for the split template and search, and P 2 represents the resolution pixel area of the patch. Subsequently, a projection layer with parameter projects the patch sequences into a D-dimensional latent space. At the same time, there are two learnable position embeddings and which are added to the patch sequence embeddings of the template and search regions respectively, and finally the final feature token sequence The above process maps the two-dimensional image data into a one-dimensional multi-channel feature sequence token, which can be described by the following formula:

[0095]

[0096] Step (2): For the feature token sequence obtained in step (1), the patch sequence embeddings can be prepared to be fed into the backbone network after the following preliminary processing. The four patch sequence embeddings are concatenated to construct a complete token sequence H1, as shown in the following formula, where Concat is the concatenation operation:

[0097]

[0098] Step (3): Input the token sequence into the encoding layer. First, perform layer normalization (Layer Normalization), that is, make the token sequence in step (2) have a similar distribution before processing, and then complete joint feature extraction, modality fusion, and search target matching through multi-head self-attention; the specific method is as follows:

[0099] Step (3.1): First, perform multiple linear projections on the input sequence to generate Q, K, and V matrices, and then perform self-attention calculation, as shown in the following formula:

[0100] H′ i =LN(H i ),

[0101]

[0102] In the above formula, LN(·) represents the layer normalization operation, which controls the range of feature distribution through normalization, and H′ represents the intermediate result after the sequence features are normalized. W Q ,W K ,W Vis a linear projection layer. The subscript \(i\) of \(H\) indicates which encoder layer its output comes from. Here, \(A\) represents the attention weight matrix, \(G\) represents the output of the multi-head self-attention, and \(C\) represents the channel dimension of \(K\). Then, the output is added to the original token sequence to obtain an intermediate result \(G'\), which is further fed into layer normalization and then passed through an MLP (Multi-Layer Perceptron), and finally added to \(G'\) to obtain the output \(H\) of the current encoding layer. i+1 , which is also the input to the next encoding layer:

[0103] G′ = G + H i ,

[0104] H i+1 = MLP(LN(G′)) + G′,

[0105] Step (3.2): The sparse spatio-temporal tokens \(M\) rgb , \(M\) rgb with historical frames are input into the sparse spatio-temporal token module, and cross-attention processing is performed with the search features respectively. Taking the visible light modality as an example, the modality annotation is ignored here. The search feature tokens of the current time frame are used as the query \(Q\) x , where the subscript indicates the search area, and the sparse spatio-temporal tokens are used as the key-value \(K\) M , \(V\) M , where the subscript indicates the historical frame, and \(C\) represents the channel dimension of \(K\) M . The process can be expressed as follows:

[0106]

[0107] In the above formula, represents the search feature tokens split from the output result \(H\) of the encoding layer. \(CA(Input1, Input2)\) represents the cross-attention process, where \(Input1\) is used to form the query matrix, and \(Input2\) is used to obtain the key-value matrix. The processing and concatenation of the two modalities can be simplified as follows: i+1 In the above formula, \(Concat(…)\) represents the process of concatenating the templates and search tokens of the two modalities;

[0108]

[0109] In the above formula, \(Concat(…)\) represents the process of concatenating the templates and search tokens of the two modalities;

[0110] Step (3.3): For the tracking process without providing sparse spatio-temporal tokens of historical frames, we screen the sparse tokens through the attention results as candidates for future tracking frames. The attention matrix can be expressed as follows, where the left part on the right side of the equal sign represents the template \(z\) of the visible light modality rgb as the query to obtain the attention matrix:

[0111]

[0112] In the above formula, A represents the attention weight matrix, and taking it as an example, it represents the attention result of the visible light modality template on the search area. The templates of the two modalities are respectively extracted for search attention, the mean value is taken for their channel dimension and multi-head dimension, and then the top K markers are selected as sparse spatio-temporal markers. The extraction process can be expressed as follows:

[0113]

[0114] In the above formula, mean(…) represents taking the mean value, TopK(…) represents selecting the top K markers after sorting by attention weights, and M rgb , M rgb represents the sparse spatio-temporal marker, and its superscript represents the current embedding level;

[0115] Step (4), after being processed by the remaining encoding layers in step (3), the search feature markers of the two modalities are separately split out, then spliced together and sent into the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, the sparse spatio-temporal markers extracted from the current frame are saved and sent into the next tracking frame for processing. The output of the tracking head is subjected to a transformation operation to obtain the target bounding box;

[0116] The backbone network mentioned in step (2) of this embodiment is a Transformer network constructed by stacking twelve encoding layers and embedding sparse spatio-temporal marker modules after the 4th, 7th, and 10th layers, and the sequence markers of the two modalities are completely spliced and input into a single-stream network. Among these 12 encoding layers, layers 1, 2, 3, 5, 6, 8, 9, 11, and 12 are conventional encoding layers, the 4th layer, the 7th layer, and the 10th layer are encoding layers with embedded sparse spatio-temporal marker modules, and the remaining encoding layers refer to the encoding layers after the 10th layer, which is the last embedded module, and then through the encoding of layers 11 and 12 to obtain the output of the backbone network. The twelve encoding layers and the final tracking head form a complete single-stream network model.

[0117] In step (3.2) of this embodiment, there are multiple pairs of sparse spatio-temporal markers M rgb , M rgb and they are applied to different levels of the network. The sparse spatio-temporal markers come from the previous tracking frame, and sparse marker screening is performed every 50 frames and sent into the next frame.

[0118] In step (3.3) of this embodiment, the process of screening sparse markers is repeated at multiple levels, and the value of K is controlled by a retention ratio parameter of 0.7. The specific working process is as follows:

[0119] Step (3.3): For the initial search, mark its length as a complete number. The size of the marker sequence is B×L×C = 12×256×768; B represents the Batchsize batch size, C represents the channel, and L represents the marker sequence length. The first screening K value is 256 * 0.7 ≈ 179, and the number of root screening modules can be deduced by analogy.

[0120] To verify the rationality and effectiveness of the technical solution of the present invention, in this embodiment, the RGBT234 dataset and the LasHeR dataset are selected for experiments, and the Precision Rate (PR) and the Success Rate (SR) are used as the evaluation indicators for quantitative performance evaluation, and the Frame Per Second (FPS) is used as the efficiency indicator. Among them, PR measures the percentage of all frames in which the distance between the center point of the tracking result and the ground truth is less than the threshold, and the representative PR score is calculated by setting the threshold to 20 pixels in the three datasets. SR measures the percentage of successfully tracked frames with an overlap greater than the threshold, and the representative SR score is calculated by the area under the curve.

[0121] This embodiment is implemented based on the deep learning framework Pytorch and uses a graphics processing unit (GPU) to accelerate the operation, with 20GB of memory and an Nvidia GeForce GTX 2080Ti graphics card.

[0122] The RGBT234 dataset contains 234 RGBT highly aligned video pairs, with a total of about 234K frames, and 12 attributes are annotated to facilitate the analysis of the effectiveness of different tracking algorithms for different challenges. LasHeR is currently the largest RGBT tracking dataset, containing 1224 aligned video sequences, including more diverse attribute annotations. Among them, 245 sequences are separately divided into the test set, and the rest are used for the training set.

[0123] Table 1 Detection results of the RGBT234 and LasHeR datasets

[0124]

[0125] The experimental results in Table 1 show that the technical solution of the present invention performs excellently on the two datasets, and the object tracking method of the present invention is significantly better than the existing similar methods in the multi-modal object tracking task.

Claims

1. A multi-modal object tracking method based on sparse spatio-temporal marker guidance, characterized in that, Input the prepared multi-modal images into the backbone network. The multi-modal images include RGB template images RGB search images TIR template images and TIR search images Let H and W represent the height and width of the image respectively, and let z and x represent the template image and the search image respectively. After receiving the multi-modal images, perform the following steps: Step (1): For the input multi-modal image, map its two-dimensional data into a one-dimensional multi-channel feature sequence label. The method is as follows: First, all corresponding images are split into patch sequences Among them is the length of the patch sequences of the template and the search after splitting, P 2 represents the resolution pixel area of the patch; Subsequently, using a parameter The projection layer projects the patch sequences into a D-dimensional latent space, while adding two learnable positional embeddings and to the patch sequence embeddings of the template and the search region respectively, to obtain the final feature token sequence Step (2), splice the feature marker sequences into a complete marker sequence H1; Step (3): Input the complete label sequence H1 into the encoding layer. First, perform layer normalization, and then complete joint feature extraction, modality fusion, and search target matching through multi-head self-attention. The specific method is as follows: First, perform multiple linear projections on the token sequence H1 to generate the Q, K, and V matrices, and then perform self-attention calculation; then add the self-attention calculation result to the original token sequence H1 to obtain the intermediate result G′, send it to layer normalization and then pass through the MLP, and finally add it to G′ to obtain the output H of the current encoding layer i+1 : Next, the sparse spatio-temporal tag M with historical frames rgb , M rgb is sent to the sparse spatio-temporal tagging module for temporal embedding, and cross-attention processing is performed with the search features respectively; Finally, for the tracking process without historical frame sparse spatio-temporal labels provided, screen sparse labels through the attention result as candidates for future tracking frames. Step (4): After being processed by the remaining encoding layers in Step (3), separately split the search feature labels of the two modalities, and then splice them together and send them into the tracking head to obtain the confidence score of the target response point. When the confidence score exceeds the threshold, save the sparse spatio-temporal labels extracted from the current frame and send them to the next tracking frame for processing. Perform a transformation operation on the output of the tracking head to obtain the target bounding box.

2. The multimodal object tracking method based on sparse spatio-temporal marker guidance according to claim 1, wherein The backbone network is based on the Transformer network and includes 12 encoding layers. The 12 encoding layers are stacked together, and sparse spatio-temporal label modules are embedded after the 4th, 7th, and 10th encoding layers respectively. The sequence labels of the two modalities are completely spliced and input into a single-stream network.

3. The multi-modal object tracking method based on sparse spatio-temporal marker guidance according to claim 1, characterized in that The formula for Step (1) to map the original multi-modal image into a one-dimensional multi-channel feature sequence label is as follows:

4. The multi-modal object tracking method based on sparse spatio-temporal marker guidance according to claim 1, wherein The process of Step (2) constructing a complete label sequence H1 by splicing the four patch sequences is shown in the following formula, where Concat is the splicing operation:

5. The multi-modal object tracking method based on sparse spatio-temporal marker guidance according to claim 1, wherein, The detailed process of Step (3) is as follows: Step (3.1): First, perform multiple linear projections on the input label sequence H1 to generate Q, K, and V matrices, and then perform self-attention calculation. The process is shown in the following formula: H′ i = LN(H i ), In the above formula, LN(·) represents the layer normalization operation, and H′ represents the intermediate result after the sequence features are normalized; W Q , W K , W V are linear projection layers. The subscript i of H indicates which encoder layer its output comes from. A represents the attention weight matrix, G represents the output of the multi-head self-attention, and C represents the channel dimension of K; Then add the self-attention calculation result to the original token sequence H' to obtain the intermediate result G', send it to layer normalization and then pass through the MLP, and finally add it to G' to obtain the output H of the current encoding layer i+1 ; G′ = G + H i , H i+1 = MLP(LN(G')) + G', Step (3.2): Send the sparse spatio-temporal tag M rgb , M rgb that has historical frames into the sparse spatio-temporal tagging module for temporal embedding, and perform cross-attention processing with the search features respectively; For the visible light modality image, use the current time frame search feature marker as the query Q x , and use the sparse spatio-temporal marker as the key value K M , V M , where C represents the channel dimension of K M , and the process is as follows: In the above formula, represents the output result H of the encoding layer i+1 The search feature flag split from it, CA(Input1, Input2) represents the cross-attention process, Input1 forms the query matrix, Input2 obtains the key-value matrix, and the processing processes and splicing combinations of the two modalities are as follows: In the above formula, Concat(…) represents the process of splicing the template labels and search labels of the two modalities together. Step (3.3): For the tracking process without historical frame sparse spatio-temporal labels provided, screen sparse labels through the attention result as candidates for future tracking frames. The attention matrix is expressed as follows: In the above formula, A represents the attention weight matrix; represents the attention result of the visible light modality template on the search area. The attention of the templates of the two modalities is extracted respectively, the mean value of their channel dimension and multi-head dimension is taken, and then the TopK markers are selected as sparse spatio-temporal markers. The extraction process is shown as follows: In the above formula, mean(…) represents taking the mean value, TopK(…) represents selecting the top K tokens after sorting by attention weights, M rgb , M rgb represents sparse spatio-temporal tokens, and its superscript represents the current embedding level.

Images