Visual target tracking method and device, medium and equipment

Abstract

The invention discloses a visual target tracking method and device, a medium and equipment, and relates to the technical field of visual target tracking. Template bridging interaction updating is used to enrich context information and eliminate background interference. The method comprises the following steps: firstly, performing data enhancement on a video data set to obtain a training set, and then cutting an original template image, a historical template image and a search area image from the training set; then, the group of images are spatially segmented and flattened into three block sequences, the three block sequences are input into a Vit backbone network for feature extraction, and in the extraction process, a template bridging interaction updating module is used for interactively propagating target and context information; through continuous Transform blocks, features of a search area, an original template and a historical template are extracted step by step, a matching relation among the features is captured, and finally the features are sent to a prediction head for a classification task and a regression task; and the target is accurately positioned by combining a classification result and a regression result.

Description

Technical Field

[0001] The present invention relates to the technical field of visual target tracking, and particularly to a visual target tracking method, device, medium and equipment. Background Technique

[0002] Visual target tracking aims to infer the position of a target in the subsequent video frames based on its state (such as position, size, etc.) in a given frame. Generally, the object to be tracked is a specific region or object in the video frame, without considering its semantic information (such as category, etc.). However, there are various challenges in the real world, such as illumination changes, interference from similar objects, low resolution, and complex backgrounds. In addition, the video frames are continuously changing, which brings challenges to solving the changes in the target appearance and the environment.

[0003] Currently, the mainstream methods can be divided into two types: The first type of modeling method, as shown in (a) of Figure 1 , interactively learns template features by two cross-attention operations with the search region. However, the information of the target in the search frame is constantly changing. The template information is based on the local region selected in the first frame, and the lack of information update in subsequent interactions will lead to limited target-background discrimination ability. Another method, as shown in (b) of Figure 1 , can perform both cross-relation modeling and self-relation modeling through self-attention between the concatenated template and the tokens of the search region. However, the search region may contain a large amount of background, and there may be interference objects similar to the target appearance. This method may introduce redundant background noise information and reduce the discrimination ability of the model. At the same time, the lack of highly discriminative representations extracted in some early layers will lead to poor interaction effects between the template and the search region.

[0004] However, different search regions in different contexts may need to consider different relevant reference information, and the above-mentioned solutions do not fully consider the appearance changes of the target and the dynamic changes of the environment, which may reduce the performance of target tracking. Summary of the Invention

[0005] Based on this, in order to solve the technical problems in the prior art, the present invention provides a visual target tracking method, device, medium and equipment.

[0006] The present invention provides a visual target tracking method, including:

[0007] Construct a neural network, which includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module;

[0008] Collect a video sequence to construct a data set, and use the data set to train the constructed neural network to obtain a target tracking model;

[0009] Input the video sequence containing the target to be tracked into the target tracking model. Extract the target historical template, target original template, and target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; perform embedding operations on the target historical template, target original template, and target search area through the embedding module to obtain target historical template embedding features, target original template embedding features, and target search area embedding features; perform feature extraction on the target historical template embedding features, target original template embedding features, and target search area embedding features through the first Transformer module, the second Transformer module, and the third Transformer module to obtain target historical template features, target original template features, and target search area features; map the target historical template features to a first key Key and a first value Value through the first attention mechanism module, map the target original template features to a first query query, and perform attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain target-associated local context aggregation features; map the target-associated local context aggregation features to a second key Key and a second value Value through the second attention mechanism module, map the target search area features to a second query query, and perform attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain target-associated local context allocation features; map the target-associated local context allocation features to a target detection result through the detection module, and output the target to be tracked in each frame image.

[0010] Further, before mapping the target historical template features to a first key Key and a first value Value through the first attention mechanism module, perform spatial feature selection on the target historical template features, specifically including:

[0011] Generate a spatial mask M pre :

[0012]

[0013] Among them, is the attention weight of the i-th head in the n-th self-attention layer of the VIT model, where i ∈ 1, 2, …, N h , N h is the number of heads in the self-attention layer; Attn n is the attention weight of all self-attention layers, where n ∈ 1, 2, …, N, and N is the number of self-attention layers in the VIT model; A score is the attention score; represents extracting the weights associated with the first score score from each head in A ; is the i-th mask corresponding to the i-th head;

[0014] Use the generated spatial mask M pre to perform spatial feature selection on the target historical template features:

[0015] Z pre = f pre ⊙ M pre

[0016] Among them, f pre is the patch feature in the target historical template features; ⊙ represents element-wise multiplication, and Z pre is the target historical template feature after spatial feature selection.

[0017] Furthermore, mapping the target historical template features to the first key Key and the first value Value, mapping the target original template features to the first query query, and performing attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain the target associated local context aggregation features specifically include:

[0018] Perform cross-attention operation on the target original template features and the target historical template features after spatial feature selection:

[0019]

[0020] Among them, Z is the target original template feature; Z pre is the target historical template feature after spatial feature selection; represents the projection matrix; C is the number of feature channels; A t is the first cross-attention matrix;

[0021] After integrating the target original template features and the first cross-attention matrix, generate the target associated local context aggregation features based on the multi-layer perceptron:

[0022]

[0023] Among them, is the integration result of the target original template feature and the first cross-attention matrix, and LN(·) represents the layer normalization operation; is the target-associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

[0024] Furthermore, mapping the target-associated local context aggregation feature to the second key Key and the second value Value, mapping the target search region feature to the second query query, and performing attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain the target-associated local context allocation feature specifically includes:

[0025] Performing a cross-attention operation on the target search region feature and the target-associated local context aggregation feature:

[0026]

[0027] Among them, X is the target search region feature; is the target-associated local context aggregation feature; represents the projection matrix; C is the number of feature channels; A x is the second cross-attention matrix;

[0028] After integrating the target original template feature and the second cross-attention matrix, generating the target-associated local context allocation feature based on the multi-layer perceptron:

[0029]

[0030] Among them, is the integration result of the target original template feature and the cross-attention matrix, and LN(·) represents the layer normalization operation; is the target-associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

[0031] Furthermore, extracting the target historical template, the target original template, and the target search region in each frame image of the video sequence containing the target to be tracked specifically includes:

[0032] Taking the center of the target in the first frame of the video sequence as a benchmark, expanding the region with twice the side length outward, and this region contains the appearance information of the target and its surrounding local scene to obtain the target original template;

[0033] Based on the target center coordinates in the previous frame image, expand the area with four times the side length outward. This area covers the moving range of the target to obtain the target search area;

[0034] Use the position in the previous frame image that has the same size as the target search area and contains the complete context information of the target and its surrounding states as the target historical template.

[0035] Furthermore, the embedding operation on the target historical template, the target original template, and the target search area specifically includes:

[0036] Segment and flatten the target historical template, the target original template, and the target search area respectively to obtain the historical template block sequence P pre 、the initial template block sequence P z and the search area block sequence P X ;

[0037] Perform linear projection on the historical template block sequence P pre 、the initial template block sequence P z and the search area block sequence P X respectively to obtain the initial features of the target historical template, the target original template, and the target search area:

[0038] Z pre =P pre W 0

[0039] Z 0 =P z W 0

[0040] X 0 =P x W 0

[0041] Among them, W 0 is the learnable parameter of the linear projection, and Z pre 、Z 0 and X 0 represent the initial features of the target historical template, the target original template, and the target search area respectively;

[0042] Correspondingly add the position encoding matrices E pre 、E z and E x to the initial features of the target historical template, the target original template, and the target search area to provide position prior information, and obtain the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature embedded with position encoding.

[0043] The present invention provides a visual target tracking device, including:

[0044] A model construction module for constructing a neural network, the neural network including a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module;

[0045] A model training module for collecting video sequences to construct a data set, and using the data set to train the constructed neural network to obtain a target tracking model;

[0046] A target tracking module for inputting a video sequence containing a target to be tracked into the target tracking model, extracting a target historical template, a target original template, and a target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; performing an embedding operation on the target historical template, the target original template, and the target search area through the embedding module to obtain a target historical template embedding feature, a target original template embedding feature, and a target search area embedding feature; performing feature extraction on the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature through the first Transformer module, the second Transformer module, and the third Transformer module to obtain a target historical template feature, a target original template feature, and a target search area feature; mapping the target historical template feature to a first key Key and a first value Value through the first attention mechanism module, mapping the target original template feature to a first query query, and performing an attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain a target-associated local context aggregation feature; mapping the target-associated local context aggregation feature to a second key Key and a second value Value through the second attention mechanism module, mapping the target search area feature to a second query query, and performing an attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain a target-associated local context allocation feature; mapping the target-associated local context allocation feature to a target detection result through the detection module, and outputting the target to be tracked in each frame image.

[0047] The present invention provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned visual target tracking method is implemented.

[0048] The present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned visual target tracking method is implemented.

[0049] The above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects:

[0050] The present invention provides a visual target tracking method. Through a first attention mechanism module, the target historical template features are mapped into a first key (Key) and a first value (Value), and the target original template features are used as a first query (Query). The attention calculation is used to extract local context aggregation features strongly related to the target. This process is guided by the initial information of the original template, screens the core features of the target from the historical template, and suppresses background noise interference at the same time. Through a second attention mechanism module, the target-associated local context aggregation features are mapped into a second key (Key) and a second value (Value), and the target search area features are used as a second query (Query). Further, the context information associated with the target is dynamically allocated to the search area through attention calculation. The second attention mechanism module matches the fused context features with the potential target positions in the search area to enhance the detection of the target. Through the two-stage attention mechanism, the original template features serve as a medium for information transmission during the interaction process, retaining both the target initial appearance information and integrating the temporal state of the historical template and the dynamic context of the search area, achieving the improvement of the accuracy and stability of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0052] Figure 1 are schematic diagrams of the principles of the two-stream modeling method and the single-stream modeling method provided by the present invention. Figure 1 in which (a) is a schematic diagram of the principle of the two-stream architecture. Figure 1 in which (b) is a schematic diagram of the principle of the single-stream architecture. Figure 1 in which (c) is a schematic diagram of the principle of the bridging interactive architecture of the present invention.

[0053] Figure 2 is a schematic diagram of the overall framework provided by the present invention.

[0054] Figure 3 is a schematic diagram of the process of the template bridging interaction update module provided by the present invention.

[0055] Figure 4 is a schematic diagram of the dynamic selection of features based on space provided by the present invention.

[0056] Figure 5 Schematic diagram of the spatial feature selection module provided by the present invention;

[0057] Figure 6 Comparison chart of the tracking effects with other trackers on the OTB100 dataset provided by the present invention; wherein, Figure 6 (a) in is the success rate, Figure 6 (b) in is the precision rate;

[0058] Figure 7 Comparison chart of the tracking effects with other trackers on the OTB100 dataset provided by the present invention. Specific embodiments

[0059] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0060] Although existing methods have achieved success to a certain extent, they do not fully consider the appearance changes of the target and the dynamic changes of the environment. In other words, different relevant reference information may need to be considered for the search areas in different contexts. Although some two-stream modeling methods perform interactive calculations of template memory and search areas, it may be unnecessary to retrieve all information from the memory to the search area, and it may reduce the tracking performance. Although these methods have certain advantages, there are potential limitations in dealing with target and environmental changes, and they do not fully consider the context information.

[0061] In view of the deficiencies of the prior art, the present invention proposes a single-object tracking method based on adaptive context awareness, which uses template bridge interaction update to enhance the information interaction between the historical template and the search area. The present invention uses OSTrack as the benchmark tracking. The present invention proposes a new scheme to bridge the cross-channel interaction between the historical template and the search area through the original template of the target as a medium. At the same time, to prevent the introduction of irrelevant background noise in the historical template, spatial feature selection is gradually used to select the feature markers centered on the target. Specifically, a bridge template interaction update module dedicated to tracking is designed, as Figure 1As shown in (c). The main advantage of this scheme is that the original template contains the original information of the target object, which can guide the extraction of target-related context from the historical template and the search area to adaptively enhance and supplement information. Since the historical template contains a large amount of background information, spatial feature selection is adopted to gradually select the feature markers centered on the target. In spatial feature selection, all spatial indices selected by the multi-head self-attention mechanism within a single channel are combined. After feature selection, the template and the historical template can gradually reduce the interference from background noise during the cross-channel interaction through the original template bridge. Different from Figure 1 the method of directly replacing the old template with the new template in (b) of

[0062] the original template features are interactively updated from the target state and the search area of the previous frame through the bridge template interaction module. The principle of the Template Bridging Interaction Update (TBIU) module is as follows:

[0063] Example 1

[0064] A neural network is constructed, which includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module; a video sequence is collected to construct a data set, and the constructed neural network is trained using the data set to obtain a target tracking model.

[0065] Input a video sequence containing the target to be tracked into the target tracking model. Through the template extraction module, extract the target historical template, target original template, and target search area in each frame image of the video sequence containing the target to be tracked; perform embedding operations on the target historical template, target original template, and target search area through the embedding module to obtain the target historical template embedding feature, target original template embedding feature, and target search area embedding feature; perform feature extraction on the target historical template embedding feature, target original template embedding feature, and target search area embedding feature through the first Transformer module, second Transformer module, and third Transformer module to obtain the target historical template feature, target original template feature, and target search area feature; map the target historical template feature to the first key Key and the first value Value through the first attention mechanism module, map the target original template feature to the first query query, and perform attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain the target-associated local context aggregation feature; map the target-associated local context aggregation feature to the second key Key and the second value Value through the second attention mechanism module, map the target search area feature to the second query query, and perform attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain the target-associated local context allocation feature; map the target-associated local context allocation feature to the target detection result through the detection module, and output the target to be tracked in each frame image.

[0066] The following specifically details the visual target tracking method based on the model as Figure 2 shown, which specifically includes the following steps:

[0067] Step 1: Crop picture triples from the enhanced training set, spatially segment and flatten them, then perform linear projection and add positional encoding on them to respectively obtain the embedding feature X 0 of the search area block and the embedding feature Z 0 of the template token, and the embedding feature Z pre of the historical template block sequence. Through consecutive Transformer blocks, gradually extract the features of the search area and the original template, previous template.

[0068] Specifically as follows:

[0069] 1) Sample images from a video sequence in the video dataset to collect training samples. Use common data augmentations (such as translation and brightness jitter) to expand the training set.

[0070] 2) The original template picture The historical template picture and the search area image are each cropped into a square. Among them, the original template image has a side length that expands twice from the center of the target in the first frame of the video sequence. This side length includes the appearance information of the target and its surrounding local scene. The historical template image contains complete context information, which should include the position of the target and its surrounding state, and needs to maintain the same size as the search area. The search area image expands four times the side length from the center coordinates of the target in the previous frame. The search area typically covers the possible movement range of the target.

[0071] 3) Segment and flatten this set of images to obtain the initial template block sequence P z , the historical template block sequence P pre and the search area block sequence P X .

[0072] 4) Apply the linear projection block embedding layer to these sequences to obtain the initial features of the original template, historical template, and search area as follows:

[0073] Z 0 = P z W 0 ;

[0074] Z pre = P pre ;

[0075] W 0 , X 0 = P x W 0 ;

[0076] where are the learnable parameters of the linear projection, represents the embedded feature of the search area block, represents the embedded feature of the template token, represents the embedded feature of the historical template block sequence, C is the number of feature channels, and N is the number of blocks. Then, a learnable position encoding matrix and are added to provide position prior information.

[0077] 5) The embedded position-encoded X 0 and Z 0 , Z pre are respectively fed into a series of Transformer blocks to extract the features of the search area, original template, and historical template. In each Transformer block, first, the Search feature X is projected three times to obtain query, key, and value. Then, matrix multiplication is performed to extract features, and the generation of the attention weights is as follows:

[0078]

[0079] The feature extraction of the original template and the historical template is also the same operation as above.

[0080] Step 2: Perform spatial feature selection on the features of the historical template, extract features from the historical template through the multi-head attention mechanism, gradually focus the attention on the target area, and finally generate a selection mask M for it pre 。

[0081] 1) The historical template is extracted through VIT for N layers, and there are N h heads in the self-attention layer. The attention weights of the nth layer can be expressed as follows:

[0082]

[0083] where is the attention weight in the ith head of the nth layer. Therefore, the attention weights of all layers are organized as:

[0084]

[0085] 2) Iteratively calculate the attention scores through matrix multiplication:

[0086]

[0087] A score represents the comprehensive relationship between patches. Then, extract the weights associated with the first score score from each head in A .

[0088] 3) For each head, retain the key features and generate a mask

[0089]

[0090] where MASK represents converting the selected feature markers into mask form, and Top s retains the top s important markers (s ∈ N + ). In multi-head attention, different heads focus on different aspects. To capture more details in the channels, "Head-level Concatenation" is adopted to combine the selected feature markers from different heads. The spatial feature selection module is inserted between Transformer blocks to gradually focus on the key area of the target. Finally, as Figure 4 shown, the mask M in the history pre and the historical template features can be expressed as:

[0091]

[0092] Z pre = f pre ⊙M pre ;

[0093] where f pre represents the patch feature of the previoustemplate. ⊙ represents element-wise multiplication, and the process of spatial feature selection is as Figure 5 shown.

[0094] Step 3. Insert the TBIU (Template Bridge Interaction Update) module between the Transformer blocks of the VIT (Vision Transformer) backbone multiple times to bridge the search region and the historical template features. Use cross-attention to aggregate the context of the original template and the historical template, and further adaptively distribute the target prior information contained therein to the search region features. Finally, transfer the features of the original template medium back to the original template for template update to prepare for the interaction of the next layer; the working principle of the template bridge interaction update module is as Figure 3 shown, specifically including:

[0095] 1) Perform cross-attention operation on the original template feature Z pre and the feature Z in the original template:

[0096]

[0097] where represents the parameters of the query, key, and value projection layers. Then, refine the previous template context A t related to the target and integrate it with Z.

[0098]

[0099] where LN and MLP represent LayerNorm and multi-layer perceptron.

[0100] 2) Use the search region features as the query, the original template medium as the key and value, and distribute information through similar cross-channel attention to further refine A x and integrate it with X to enhance the corresponding target-related search region markers:

[0101]

[0102]

[0103] 3) Use to represent the original template medium after historical template interaction, with Z pre as the key and value, and use the original template Z as the query for information transmission. Then, use similar LN (Layer Normalization) and MLP (Multi-Layer Perceptron) layers to refine the output The output search area X of the previous layer Transformer, the updated template marker and the interactive search area marker are used as the input data for the next (i + 1)-th Transformer block in the VIT backbone. Figure 2 The output of the template bridging interaction update in is input into the three transformers of the next layer. The inputs of the three transformers in the next layer include the output x_hat of the template bridging interaction update, the output search area of the previous layer transformer, and the output Zpre of the spatial feature selection.

[0104] Step 4: Use the output of the VIT as the input of the prediction head. The prediction head consists of a classification branch and a regression branch. Each branch is a three-layer perceptron with a hidden dimension d and a ReLU activation function. For the feature map generated by the feature fusion network the prediction head makes predictions for each vector to obtain (n = H z W z ) foreground / background classification results and (n = H z W z ) normalized coordinates of the search area.

[0105] Specifically as follows:

[0106] 1) Locate the approximate position of the target according to the classification result, that is, the position with the largest value in the classification result, and map this position back to the search area to obtain the center position of the target; select the offset values of the corresponding target above, below, left, and right relative to the center position in the regression result according to the position with the largest value in the classification result.

[0107] 2) If there is a tracking failure or the classification score is too low, update the entire model by the gradient descent method.

[0108] 3) Finally, draw the tracking coordinate box according to the center position and offset value of the target to obtain the tracking result.

[0109] The single-object tracking method based on adaptive context awareness proposed by the present invention, in Figure 2The overall framework diagram of the present invention is given. First, a training set is obtained after data augmentation from the video dataset. Subsequently, the original template image, historical template image, and search region image are cropped from the training set. Then, this set of images is input into the Vit network for feature extraction. Secondly, the features extracted from the historical template image are sent to the spatial feature selection module for background screening. In the proposed template bridging interaction update module, the interaction between the historical template and the search region collects and distributes local context related to the target through the original template as a medium to enhance the features of the search region. The original template is also updated with the rich context of the medium. Finally, the updated original template and search region features are concatenated and enhanced and sent to the tracking head for classification tasks and regression tasks; and the target is accurately located by combining the classification result and the regression result. The invention point is to use template bridging interaction update to enrich context information and eliminate background interference: this set of images is spatially segmented and flattened into 3 block sequences, input into the Vit backbone network for feature extraction, and the template bridging interaction update module is used for interactive propagation of target and context information during the extraction process. Through consecutive Transformer blocks, the features of the search region, original template, and historical template are gradually extracted, and the matching relationship between them is captured. Finally, the features are sent to the prediction head for classification tasks and regression tasks; and the target is accurately located by combining the classification result and the regression result.

[0110] Preliminary test experiments were conducted on the fusion feature decoding structure scheme proposed by the present invention. The experiments were carried out on a server with NVIDIA TITAN RTX 4090 using the Pytorch framework. The software platform is Pycharm, and the fusion feature decoding structure scheme was implemented in Python language programming.

[0111] In the experiment, first, the success rate and precision rate on the OTB100 dataset were tested, and then the comparison of the true tracking results in the OTB100 dataset was tested. Figure 6 The two figures in are the precision rate figure and the success rate figure respectively. The precision rate figure represents the Euclidean distance between the center point of the prediction box of the tracking algorithm and the center point of the Ground Truth box. Usually, the threshold is 20 pixels. That is, if their Euclidean distance is within 20 pixels, it is considered a successful tracking. The success rate figure represents the percentage of the overlap rate (overlap score, OS) between the tracking box drawn by the tracking algorithm and the manually annotated tracking box being greater than the given threshold. Figure 6 In, Ours is the name of the tracker proposed by the present invention, and ROMtrack, Mixformer, AiATrack, SparseTT, TransT, and OSTrack are the names of trackers proposed by other scholars in recent years. The following numbers refer to the average precision rate and average accuracy of the trackers. Such as Figure 6As shown in the figure, when compared with advanced trackers in recent years on the general dataset OTB100, the tracking method Ours of the present invention has significantly improved in terms of precision and success rate. Figure 7 This is a comparison of the actual tracking results of the present invention and advanced trackers OSTrack, Mixformer, ROMtrack, and TransT in recent years in the OTB100 dataset. As can be seen from Figure 7 it, when similar targets appear in the search area and during long-term tracking, the bounding boxes generated by the tracking method of the present invention are more accurate than those of other trackers.

[0112] In summary, the present invention proposes a new solution that bridges the cross-channel interaction between the historical template and the search area through the original template of the target as a medium. Specifically, a bridging template interaction update module dedicated to tracking is designed, as shown in Figure 1 (c). The main advantage of this solution is that the original template contains the original information of the target object, which can guide the extraction of target-related context from the historical template and the search area to adaptively enhance and supplement information. The template and the historical template can gradually reduce the interference from background noise during the cross-channel interaction through the bridging of the original template. The original template features are interactively updated from the target state and the search area of the previous frame through the bridging template interaction module. The principle of the template bridging interaction update (TBIU) module is as follows: using the cross-attention mechanism, the original template is used as the query, the target state features of the previous frame are used as the key and value, and the target-related search area context information is collected into the original template medium. Then, the search area features are used as the query, and the fused original template is used as the key and value, and the target-related context is distributed from the medium to the search area. After multiple updates, the most representative target features in the entire video sequence are finally obtained. Finally, the context information of the fused template medium and the search area information are transmitted back to the original template to update the rich context collected between the historical template and the search area, realizing effective single-object tracking.

[0113] The above is the visual target tracking method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding visual target tracking device, including:

[0114] A model construction module for constructing a neural network, where the neural network includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module.

[0115] A model training module for collecting video sequences to construct a data set, and using the data set to train the constructed neural network to obtain a target tracking model.

[0116] A target tracking module for inputting a video sequence containing a target to be tracked into the target tracking model, extracting a target historical template, a target original template, and a target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; performing an embedding operation on the target historical template, the target original template, and the target search area through the embedding module to obtain a target historical template embedding feature, a target original template embedding feature, and a target search area embedding feature; performing feature extraction on the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature through the first Transformer module, the second Transformer module, and the third Transformer module to obtain a target historical template feature, a target original template feature, and a target search area feature; mapping the target historical template feature to a first key Key and a first value Value through the first attention mechanism module, mapping the target original template feature to a first query query, and performing an attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain a target-associated local context aggregation feature; mapping the target-associated local context aggregation feature to a second key Key and a second value Value through the second attention mechanism module, mapping the target search area feature to a second query query, and performing an attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain a target-associated local context allocation feature; mapping the target-associated local context allocation feature to a target detection result through the detection module, and outputting the target to be tracked in each frame image.

[0117] For the specific limitations of the visual target tracking device, reference may be made to the limitations of the visual target tracking method in the foregoing text, which will not be elaborated herein. Each module in the above visual target tracking device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the above modules.

[0118] The present invention also provides a computer-readable storage medium storing a computer program, which can be used to execute the above-provided visual target tracking method.

[0119] The present invention also provides the structure of a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-provided visual target tracking method.

[0120] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memories. The non-volatile memory can include a read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0121] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded by the present invention.

Claims

1. A visual target tracking method, characterized in that: include: Constructing a neural network, the neural network comprising a template extraction module, an embedding module, a context modeling module and a detection module connected in sequence; the context modeling module comprises a first Transformer module, a second Transformer module and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module; Collect video sequences to construct a data set, use the data set to train the constructed neural network, and obtain a target tracking model; The video sequence containing the target to be tracked is input into the target tracking model, and the target history template, the target original template and the target search area in each frame image of the video sequence containing the target to be tracked are extracted by the template extraction module; The target history template, the target original template and the target search area are embedded by the embedding module to obtain the target history template embedding features, the target original template embedding features and the target search area embedding features; the target history template embedding features, the target original template embedding features and the target search area embedding features are extracted by the first Transformer module, the second Transformer module and the third Transformer module to obtain the target history template features, the target original template features and the target search area features; The target history template feature is mapped to the first key Key and the first value Value through the first attention mechanism module, the target original template feature is mapped to the first query query, and the attention mechanism calculation is performed based on the first key Key, the first value Value and the first query query to obtain the target associated local context aggregation feature; The target-associated local context aggregation feature is mapped to a second key Key and a second value Value through a second attention mechanism module, the target search area feature is mapped to a second query query, and an attention mechanism calculation is performed based on the second key Key, the second value Value and the second query query to obtain the target-associated local context allocation feature; The detection module maps the target-associated local context allocation features into target detection results, and outputs the target to be tracked in each frame image.

2. The visual target tracking method according to claim 1, characterized in that: Before mapping the target history template feature to the first key Key and the first value Value through the first attention mechanism module, spatial feature selection is performed on the target history template feature, specifically including: Generate spatial mask M pre : in, is the attention weight of the i-th head in the n-th self-attention layer of the VIT model, i∈1,2,…,N h , N h is the number of heads in the self-attention layer; Attn n A is the attention weight of all self-attention layers, n∈1,2,…,N, N is the number of self-attention layers in the VIT model; score is the attention score; Indicates from A score Each header in the extract is the same as the first The weights associated with the scores; is the i-th mask corresponding to the i-th head; Generate a spatial mask M using pre Perform spatial feature selection on the target history template features: Z pre =f pre ⊙M pre Among them, f pre is the patch feature in the target history template feature; ⊙ represents the element-by-element multiplication, Z pre It is the target history template feature after spatial feature selection.

3. The visual target tracking method according to claim 2, characterized in that: The target historical template feature is mapped to a first key Key and a first value Value, the target original template feature is mapped to a first query query, and an attention mechanism is calculated based on the first key Key, the first value Value and the first query query to obtain the target associated local context aggregation feature, specifically including: Perform cross-attention operation on the target original template features and the target historical template features after spatial feature selection: Among them, Z is the target original template feature; Z pre It is the target history template feature after spatial feature selection; represents the projection matrix; C is the number of feature channels; A t is the first cross attention matrix; After integrating the target original template features and the first cross attention matrix, the target associated local context aggregation features are generated based on the multi-layer perceptron: in, It is the integration result of the target original template feature and the first cross attention matrix, and LN(·) represents the layer normalization operation; is the target associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

4. The visual target tracking method according to claim 3, characterized in that: The target-associated local context aggregation feature is mapped to a second key Key and a second value Value, the target search area feature is mapped to a second query query, and an attention mechanism is calculated based on the second key Key, the second value Value and the second query query to obtain the target-associated local context allocation feature, specifically including: Perform cross-attention operations on the target search area features and the target associated local context aggregation features: Among them, X is the target search area feature; It is the target-associated local context aggregation feature; represents the projection matrix; C is the number of feature channels; A x is the second cross attention matrix; After integrating the target original template features and the second cross attention matrix, the target associated local context allocation features are generated based on the multi-layer perceptron: in, It is the integration result of the target original template features and the cross attention matrix, and LN(·) represents the layer normalization operation; is the target associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

5. The visual target tracking method according to claim 1, characterized in that: The step of extracting the target history template, the target original template and the target search area in each frame image of the video sequence containing the target to be tracked specifically includes: Taking the target center in the first frame of the video sequence as the reference, expand the area with twice the side length outward, which contains the appearance information of the target and its surrounding local scene, to obtain the original template of the target; Based on the target center coordinates in the previous frame image, an area with four times the side length is expanded outward, which covers the moving range of the target, to obtain the target search area; The previous frame image with the same size as the target search area and containing complete context information of the target's location and its surrounding state is used as the target history template.

6. The visual target tracking method according to claim 1, characterized in that: The embedding operation of the target history template, the target original template and the target search area specifically includes: The target history template, target original template and target search area are segmented and flattened respectively to obtain the history template block sequence P pre , initial template block sequence P z and search area block sequence P X ; For the historical template block sequence P pre , initial template block sequence P z and search area block sequence P X Perform linear projection to obtain the target history template, target original template, and initial features of the target search area: WITH pre =P pre W0 Z0=P z W0 X0=P x W0 Among them, W0 is the learnable parameter of linear projection, Z pre , Z0 and X0 represent the initial features of the target history template, the target original template and the target search area respectively; Add the position encoding matrix E to the target history template, the target original template and the initial features of the target search area accordingly pre 、E z and E x By providing position prior information, the target history template embedding features, target original template embedding features and target search area embedding features with embedded position encoding are obtained.

7. A visual target tracking device, characterized in that: include: A model building module, used to build a neural network, the neural network includes a template extraction module, an embedding module, a context modeling module and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module; A model training module is used to collect video sequences to construct a data set, and use the data set to train the constructed neural network to obtain a target tracking model; The target tracking module is used to input the video sequence containing the target to be tracked into the target tracking model, and extract the target history template, target original template and target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; The target history template, the target original template and the target search area are embedded by the embedding module to obtain the target history template embedding features, the target original template embedding features and the target search area embedding features; the target history template embedding features, the target original template embedding features and the target search area embedding features are extracted by the first Transformer module, the second Transformer module and the third Transformer module to obtain the target history template features, the target original template features and the target search area features; The target history template feature is mapped to the first key Key and the first value Value through the first attention mechanism module, the target original template feature is mapped to the first query query, and the attention mechanism calculation is performed based on the first key Key, the first value Value and the first query query to obtain the target associated local context aggregation feature; The target-associated local context aggregation feature is mapped to a second key Key and a second value Value through a second attention mechanism module, the target search area feature is mapped to a second query query, and an attention mechanism calculation is performed based on the second key Key, the second value Value and the second query query to obtain the target-associated local context allocation feature; The detection module maps the target-associated local context allocation features into target detection results, and outputs the target to be tracked in each frame image.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 6 is implemented.

Images