A Visual Target Tracking Method, Device, Medium and Equipment

Abstract

The present invention discloses a visual object tracking method, device, medium and equipment, which relates to the technical field of visual object tracking. The present invention uses template bridging interaction update to enrich context information and exclude background interference. First, after data augmentation on the video dataset, a training set is obtained, and then the original template image, historical template image and search area image are cropped from the training set. Then, this set of images is spatially segmented and flattened into 3 block sequences, which are input into the Vit backbone network for feature extraction, and a template bridging interaction update module is used during the extraction process to interactively propagate object and context information. Through consecutive Transformer blocks, the features of the search area, original template and historical template are gradually extracted, and the matching relationship between them is captured. Finally, the features are sent to the prediction head for classification tasks and regression tasks; and the object is accurately located by combining the classification result and the regression result.

Description

Technical Field

[0001] The present invention relates to the technical field of visual target tracking, and particularly to a visual target tracking method, device, medium and equipment. Background Art

[0002] Visual target tracking aims to infer the position of a target in the subsequent video frames based on its state (such as position, size, etc.) in a given frame. Generally, the object to be tracked is a specific region or object in the video frame, without considering its semantic information (such as category, etc.). However, there are various challenges in the real world, such as illumination changes, interference from similar objects, low resolution, and complex backgrounds. In addition, the video frames are continuously changing, which poses challenges to solving the changes in target appearance and environment.

[0003] Currently, the mainstream methods can be divided into two types: The first type of modeling method, as shown in (a) of Figure 1 , interactively learns template features by two cross-attention operations with the search region. However, the information of the target in the search frame is constantly changing, and the template information is based on the local region selected in the first frame. Without information update in subsequent interactions, it will lead to limited target-background discrimination ability. Another method, as shown in (b) of Figure 1 , can perform both cross-relation modeling and self-relation modeling by self-attention between the concatenated templates and the tokens of the search region. However, the search region may contain a large amount of background, and there may be interfering objects similar to the target appearance. This method may introduce redundant background noise information and reduce the discrimination ability of the model. At the same time, the lack of highly discriminative representations extracted in some early layers will result in poor interaction effects between the template and the search region.

[0004] However, different relevant reference information may need to be considered for the search regions in different contexts, and the above solutions do not fully consider the appearance changes of the target and the dynamic changes of the environment, which may reduce the performance of target tracking. Summary of the Invention

[0005] Based on this, in order to solve the technical problems in the prior art, the present invention provides a visual target tracking method, device, medium and equipment.

[0006] The present invention provides a visual target tracking method, including:

[0007] Construct a neural network, which includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module;

[0008] Collect a video sequence to construct a dataset, and use the dataset to train the constructed neural network to obtain a target tracking model;

[0009] Input the video sequence containing the target to be tracked into the target tracking model. Extract the target historical template, target original template, and target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; perform embedding operations on the target historical template, target original template, and target search area through the embedding module to obtain target historical template embedding features, target original template embedding features, and target search area embedding features; perform feature extraction on the target historical template embedding features, target original template embedding features, and target search area embedding features through the first Transformer module, the second Transformer module, and the third Transformer module to obtain target historical template features, target original template features, and target search area features; map the target historical template features to a first key Key and a first value Value through the first attention mechanism module, map the target original template features to a first query query, and perform attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain target-associated local context aggregation features; map the target-associated local context aggregation features to a second key Key and a second value Value through the second attention mechanism module, map the target search area features to a second query query, and perform attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain target-associated local context assignment features; map the target-associated local context assignment features to a target detection result through the detection module, and output the target to be tracked in each frame image.

[0010] Further, before mapping the target historical template features to a first key Key and a first value Value through the first attention mechanism module, perform spatial feature selection on the target historical template features, specifically including:

[0011] Generate a spatial mask M pre :

[0012]

[0013] Among them, is the attention weight of the i-th head in the n-th self-attention layer of the VIT model, where i ∈ 1, 2, …, N h , N h is the number of heads in the self-attention layer; Attn n is the attention weight of all self-attention layers, where n ∈ 1, 2, …, N and N is the number of self-attention layers in the VIT model; A score is the attention score; denotes extracting the weights associated with the first score score from each head in A ; is the i-th mask corresponding to the i-th head;

[0014] Use the generated spatial mask M pre to perform spatial feature selection on the target historical template features:

[0015] Z pre = f pre ⊙ M pre

[0016] Among them, f pre is the patch feature in the target historical template features; ⊙ represents element-wise multiplication, and Z pre is the target historical template feature after spatial feature selection.

[0017] Furthermore, the mapping of the target historical template features to the first key Key and the first value Value, the mapping of the target original template features to the first query query, and the calculation of the attention mechanism based on the first key Key, the first value Value, and the first query query to obtain the target associated local context aggregation features specifically include:

[0018] Perform cross-attention operations on the target original template features and the target historical template features after spatial feature selection:

[0019]

[0020] Among them, Z is the target original template feature; Z pre is the target historical template feature after spatial feature selection; denotes the projection matrix; C is the number of feature channels; A t is the first cross-attention matrix;

[0021] After integrating the target original template features and the first cross-attention matrix, generate the target associated local context aggregation features based on a multi-layer perceptron:

[0022]

[0023] Among them, is the integration result of the target original template feature and the first cross-attention matrix, and LN(·) represents the layer normalization operation; is the target-associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

[0024] Furthermore, mapping the target-associated local context aggregation feature to the second key Key and the second value Value, mapping the target search region feature to the second query query, and performing attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain the target-associated local context allocation feature specifically includes:

[0025] Performing a cross-attention operation on the target search region feature and the target-associated local context aggregation feature:

[0026]

[0027] Among them, X is the target search region feature; is the target-associated local context aggregation feature; represents the projection matrix; C is the number of feature channels; A x is the second cross-attention matrix;

[0028] After integrating the target original template feature and the second cross-attention matrix, generating the target-associated local context allocation feature based on the multi-layer perceptron:

[0029]

[0030] Among them, is the integration result of the target original template feature and the cross-attention matrix, and LN(·) represents the layer normalization operation; is the target-associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

[0031] Furthermore, extracting the target historical template, the target original template, and the target search region in each frame image of the video sequence containing the target to be tracked specifically includes:

[0032] Taking the center of the target in the first frame of the video sequence as a benchmark, expanding the region by twice the side length outward, and this region contains the appearance information of the target and its surrounding local scenes to obtain the target original template;

[0033] Based on the target center coordinates in the previous frame image, expand the area with four times the side length outward. This area covers the moving range of the target to obtain the target search area;

[0034] Use the position in the previous frame image that has the same size as the target search area and contains the complete context information of the target and its surrounding states as the target historical template.

[0035] Further, the embedding operation on the target historical template, the target original template, and the target search area specifically includes:

[0036] Segment and flatten the target historical template, the target original template, and the target search area respectively to obtain the historical template block sequence P pre 、the initial template block sequence P z and the search area block sequence P X ;

[0037] Perform linear projection on the historical template block sequence P pre 、the initial template block sequence P z and the search area block sequence P X respectively to obtain the initial features of the target historical template, the target original template, and the target search area:

[0038] Z pre =P pre W0

[0039] Z0=P z W0

[0040] X0=P x W0

[0041] where W0 is the learnable parameter of the linear projection, and Z pre 、Z0 and X0 represent the initial features of the target historical template, the target original template, and the target search area respectively;

[0042] Correspondingly add the position encoding matrices E pre 、E z and E x to the initial features of the target historical template, the target original template, and the target search area to provide position prior information, and obtain the target historical template embedding features, the target original template embedding features, and the target search area embedding features with embedded position encoding.

[0043] The present invention provides a visual target tracking device, including:

[0044] A model construction module for constructing a neural network, where the neural network includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module;

[0045] A model training module for collecting video sequences to construct a data set, and using the data set to train the constructed neural network to obtain a target tracking model;

[0046] A target tracking module for inputting a video sequence containing a target to be tracked into the target tracking model, extracting the target historical template, the target original template, and the target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; performing an embedding operation on the target historical template, the target original template, and the target search area through the embedding module to obtain target historical template embedding features, target original template embedding features, and target search area embedding features; extracting features from the target historical template embedding features, the target original template embedding features, and the target search area embedding features through the first Transformer module, the second Transformer module, and the third Transformer module to obtain target historical template features, target original template features, and target search area features; mapping the target historical template features to a first key Key and a first value Value through the first attention mechanism module, mapping the target original template features to a first query query, and performing attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain target-associated local context aggregation features; mapping the target-associated local context aggregation features to a second key Key and a second value Value through the second attention mechanism module, mapping the target search area features to a second query query, and performing attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain target-associated local context allocation features; mapping the target-associated local context allocation features to a target detection result through the detection module, and outputting the target to be tracked in each frame image.

[0047] The present invention provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned visual target tracking method is implemented.

[0048] The present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned visual target tracking method is implemented.

[0049] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0050] The present invention provides a visual target tracking method. Through a first attention mechanism module, the target historical template features are mapped to a first key (Key) and a first value (Value), and the target original template features are used as a first query (Query). The attention calculation is used to extract the local context aggregation features strongly related to the target. This process is guided by the initial information of the original template, filters the core features of the target from the historical template, and suppresses the interference of background noise at the same time. Through a second attention mechanism module, the target-associated local context aggregation features are mapped to a second key (Key) and a second value (Value), and the target search area features are used as a second query (Query). Further, the context information associated with the target is dynamically allocated to the search area through attention calculation. The second attention mechanism module matches the fused context features with the potential target positions in the search area to enhance the detection of the target. Through the two-stage attention mechanism, the original template features serve as the medium for information transmission during the interaction process, retaining both the target initial appearance information and integrating the temporal state of the historical template and the dynamic context of the search area, achieving the improvement of the accuracy and stability of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation to the present invention. In the drawings:

[0052] Figure 1 are schematic diagrams of the principles of the two-stream modeling method and the single-stream modeling method provided by the present invention, Figure 1 where (a) is a schematic diagram of the principle of the two-stream architecture, Figure 1 and (b) is a schematic diagram of the principle of the single-stream architecture; Figure 1 where (c) is a schematic diagram of the principle of the bridging interactive architecture of the present invention;

[0053] Figure 2 is a schematic diagram of the overall framework provided by the present invention;

[0054] Figure 3 is a schematic diagram of the process of the template bridging interaction update module provided by the present invention;

[0055] Figure 4 is a schematic diagram of the dynamic selection of features based on space provided by the present invention;

[0056] Figure 5 Schematic diagram of the spatial feature selection module provided by the present invention;

[0057] Figure 6 Comparison chart of the tracking effects with other trackers on the OTB100 dataset provided by the present invention; among them, Figure 6 (a) in is the success rate, Figure 6 (b) in is the precision;

[0058] Figure 7 Comparison chart of the tracking effects with other trackers on the OTB100 dataset provided by the present invention. Detailed implementation manners

[0059] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the specific embodiments and corresponding drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0060] Although existing methods have achieved success to a certain extent, they do not fully consider the appearance changes of the target and the dynamic changes of the environment. In other words, search regions in different contexts may need to consider different relevant reference information. Although some two-stream modeling methods perform interactive calculations of template memory and search regions, it may be unnecessary to retrieve all information from the memory to the search region and may reduce the tracking performance. Although these methods have certain advantages, there are potential limitations in dealing with target and environment changes and do not fully consider context information.

[0061] In view of the deficiencies of the prior art, the present invention proposes a single-object tracking method based on adaptive context awareness, using template bridging interactive update to enhance the information interaction between the historical template and the search region. The present invention uses OSTrack as the benchmark tracking. The present invention proposes a new scheme to bridge the cross-channel interaction between the historical template and the search region through the original template of the target as a medium. At the same time, to prevent the introduction of irrelevant background noise in the historical template, spatial feature selection is gradually used to select the feature markers centered on the target. Specifically, a bridging template interactive update module dedicated to tracking is designed, such as Figure 1As shown in (c). The main advantage of this scheme is that the original template contains the original information of the target object, which can guide the extraction of target-related context from the historical template and the search area to adaptively enhance and supplement information. Since the historical template contains a large amount of background information, spatial feature selection is adopted to gradually select the feature markers centered on the target. In spatial feature selection, all spatial indices selected by the multi-head self-attention mechanism within a single channel are combined. After feature selection, the template and the historical template can gradually reduce the interference from background noise during the cross-channel interaction process through the bridging of the original template. Different from Figure 1 the method of directly replacing the old template with the new template in (b) of , the original template features are interactively updated from the target state and the search area of the previous frame through the bridging template interaction module. The principle of the Template Bridging Interaction Update (TBIU) module is as follows:

[0062] Using the cross-attention mechanism, the original template is used as the query, the target state features of the previous frame are used as the key and value, and the context information of the search area related to the target is collected into the original template medium. Then, the search area features are used as the query, and the fused original template is used as the key and value, and the target-related context is distributed from the medium to the search area. After multiple updates, the most representative target features in the entire video sequence are finally obtained. Finally, the context information of the fused template medium and the search area information are transmitted back to the original template to update the rich context collected between the historical template and the search area. The STS and TBIU modules are inserted between the Transformer blocks to select high-quality features and bridge the channel information flow within the Transformer to achieve effective single-object tracking.

[0063] Embodiment 1

[0064] A constructed neural network includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module; a video sequence is collected to construct a data set, and the constructed neural network is trained using the data set to obtain a target tracking model.

[0065] Input the video sequence containing the target to be tracked into the target tracking model. Through the template extraction module, extract the target historical template, target original template, and target search area in each frame image of the video sequence containing the target to be tracked; perform embedding operations on the target historical template, target original template, and target search area through the embedding module to obtain the target historical template embedding feature, target original template embedding feature, and target search area embedding feature; perform feature extraction on the target historical template embedding feature, target original template embedding feature, and target search area embedding feature through the first Transformer module, second Transformer module, and third Transformer module to obtain the target historical template feature, target original template feature, and target search area feature; map the target historical template feature to the first key Key and the first value Value through the first attention mechanism module, map the target original template feature to the first query query, and perform attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain the target-associated local context aggregation feature; map the target-associated local context aggregation feature to the second key Key and the second value Value through the second attention mechanism module, map the target search area feature to the second query query, and perform attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain the target-associated local context allocation feature; map the target-associated local context allocation feature to the target detection result through the detection module, and output the target to be tracked in each frame image.

[0066] The following specifically details the visual target tracking method based on the model as Figure 2 shown, which specifically includes the following steps:

[0067] Step 1: Crop picture triples from the enhanced training set. Through spatial segmentation and flattening, and then perform linear projection and add position encoding on them to respectively obtain the embedding feature X0 of the search area block, the embedding feature Z0 of the template token, and the embedding feature Z of the historical template block sequence pre , and through consecutive Transformer blocks, gradually extract the features of the search area, the original template, and the previous template.

[0068] Specifically as follows:

[0069] 1) Sample images from a video sequence in the video dataset to collect training samples. Use common data augmentation (such as translation and brightness jitter) to expand the training set.

[0070] 2) The original template picture The historical template picture and the search area picture They are respectively cropped into squares. Among them, the original template image has a side length that expands twice from the center of the target in the first frame of the video sequence. This side length includes the appearance information of the target and its surrounding local scenes. The historical template image contains complete context information, which should include the position of the target and its surrounding states, and needs to maintain the same size as the search area. The search area image is expanded by four times the side length from the center coordinates of the target in the previous frame. The search area typically covers the possible movement range of the target.

[0071] 3) Segment and flatten this group of images to obtain the initial template block sequence P z , the historical template block sequence P pre and the search area block sequence P X .

[0072] 4) Apply the block embedding layer of linear projection to these sequences to obtain the initial features of the original template, historical template, and search area as follows:

[0073] Z0 = P z W0;

[0074] Z pre = P pre ;

[0075] W0,X0 = P x W0;

[0076] where are the learnable parameters of linear projection, represents the embedded feature of the search area block, represents the embedded feature of the template token, represents the embedded feature of the historical template block sequence, C is the number of feature channels, and N is the number of blocks. Then, a learnable position encoding matrix and are added to provide position prior information.

[0077] 5) The embedded position-encoded X0 and Z0, Z pre are respectively fed into a series of Transformer blocks to extract the features of the search area, original template, and historical template. In each Transformer block, first project the Search feature X three times to obtain query, key, and value. Then, perform matrix multiplication to extract features, and the generation of attention weights is as follows:

[0078]

[0079] The feature extraction of the original template and historical template is also the same operation as above.

[0080] Step 2: Perform spatial feature selection on the features of the historical template, extract features from the historical template through the multi-head attention mechanism, gradually focus attention on the target area, and finally generate a selection mask M for it pre 。

[0081] 1) The historical template is extracted through VIT for N layers, and there are N h heads in the self-attention layer. The attention weights of the nth layer can be expressed as follows:

[0082]

[0083] where is the attention weight in the ith head of the nth layer. Therefore, the attention weights of all layers are organized as:

[0084]

[0085] 2) Iteratively calculate the attention scores through matrix multiplication:

[0086]

[0087] A score represents the comprehensive relationship between patches. Then, extract the weights associated with the first score score from each head in A .

[0088] 3) For each head, retain the key features and generate a mask

[0089]

[0090] where MASK represents converting the selected feature markers into mask form, and Top s retains the top s important markers (s ∈ N + ). In multi-head attention, different heads focus on different aspects. To capture more details in the channels, "Head-level Concatenation" is adopted to combine the selected feature markers from different heads. The spatial feature selection module is inserted between Transformer blocks to gradually focus on the key areas of the target. Finally, as Figure 4 shown, the mask M in the history pre and the historical template features can be expressed as:

[0091]

[0092] Z pre = f pre ⊙ M pre ;

[0093] Among them, f pre represents the patch feature of the previoustemplate. ⊙ represents element-wise multiplication, and the process of spatial feature selection is as Figure 5 shown.

[0094] Step 3: Insert the TBIU (Template Bridge Interaction Update) module between the Transformer blocks of the VIT (Vision Transformer) backbone multiple times to bridge the search area and the historical template features. Use cross-attention to aggregate the context of the original template and the historical template, and further adaptively distribute the target prior information contained therein to the search area features. Finally, transfer the features of the original template medium back to the original template for template update to prepare for the interaction of the next layer; the working principle of the template bridge interaction update module is as Figure 3 shown, specifically including:

[0095] 1) Perform cross-attention operation on the original template feature Z pre and the feature Z in the original template:

[0096]

[0097] Among them represents the parameters of the query, key, and value projection layers. Then, refine the previous template context A t related to the target and integrate it with Z.

[0098]

[0099] where LN and MLP represent LayerNorm and multi-layer perceptron.

[0100] 2) Use the search area features as the query, the original template medium as the key and value, and distribute information through similar cross-channel attention to further refine A x and integrate it with X to enhance the corresponding target-related search area markers:

[0101]

[0102]

[0103] 3) Use to represent the original template medium after historical template interaction, and use Z preTake key and value, and use the original template Z as the query for information transmission. Then, use similar LN (Layer Normalization) and MLP (Multi-Layer Perceptron) layers to refine the output. The output search area X of the previous layer Transformer, the updated template markers and the interactive search area markers are used as the input data for the next (i + 1)-th Transformer block in the VIT backbone. Figure 2 The output of the template bridging interaction update in Figure 2 is input into the three transforms of the next layer. The inputs of the three transforms of the next layer include the output x_hat of the template bridging interaction update, the output search area of the previous layer transform, and the output Zpre of the spatial feature selection.

[0104] Step 4: Use the output of the VIT as the input of the prediction head. The prediction head consists of a classification branch and a regression branch. Each branch is a three-layer perceptron with a hidden dimension d and a ReLU activation function. For the feature map generated by the feature fusion network the prediction head makes predictions for each vector to obtain (n = H z W z ) foreground / background classification results and (n = H z W z ) normalized coordinates of the search area.

[0105] Specifically as follows:

[0106] 1) Locate the approximate position of the target according to the classification result, that is, the position with the largest value in the classification result, and map this position back to the search area to obtain the center position of the target; select the offset values of the target's top, bottom, left, and right relative to the center position in the regression result according to the position with the largest value in the classification result.

[0107] 2) If there is a tracking failure or the classification score is too low, update the entire model by the gradient descent method.

[0108] 3) Finally, draw the tracking coordinate box according to the center position and offset value of the target to obtain the tracking result.

[0109] The single-object tracking method based on adaptive context awareness proposed by the present invention, in Figure 2The overall framework diagram of the present invention is given. First, a training set is obtained after data augmentation from a video dataset. Subsequently, the original template image, historical template image, and search area image are cropped from the training set. Then, this set of images is input into the Vit network for feature extraction. Secondly, the features extracted from the historical template image are sent to the spatial feature selection module for background screening. In the proposed template bridging interaction update module, the interaction between the historical template and the search area collects and distributes local context related to the target through the original template as a medium to enhance the features of the search area. The original template is also updated with the rich context of the medium. Finally, the updated original template and search area features are concatenated and enhanced and sent to the tracking head for classification tasks and regression tasks; and the target is accurately located by combining the classification result and the regression result. The invention point is to use template bridging interaction update to enrich context information and eliminate background interference: this set of images is spatially segmented and flattened into 3 block sequences, input into the Vit backbone network for feature extraction, and the template bridging interaction update module is used for interactive propagation of target and context information during the extraction process. Through consecutive Transformer blocks, the features of the search area, original template, and historical template are gradually extracted, and the matching relationship between them is captured. Finally, the features are sent to the prediction head for classification tasks and regression tasks; and the target is accurately located by combining the classification result and the regression result.

[0110] Preliminary test experiments were conducted on the proposed fusion feature decoding structure scheme of the present invention. The experiments were carried out on a server with an NVIDIA TITAN RTX 4090 using the Pytorch framework. The software platform was Pycharm, and the fusion feature decoding structure scheme was implemented by programming in the Python language.

[0111] In the experiment, first, the success rate and precision rate on the OTB100 dataset were tested, and then the comparison of the real tracking results in the OTB100 dataset was tested. Figure 6 The two figures in [Figure X] are the precision rate figure and the success rate figure respectively. What is represented in the precision rate figure is the Euclidean distance between the center point of the prediction box of the tracking algorithm and the center point of the Ground Truth box. Usually, the threshold is 20 pixels. That is, if their Euclidean distance is within 20 pixels, it is regarded as a successful tracking. What is represented in the success rate figure is the percentage of the overlap rate (overlap score, OS) between the tracking box drawn by the tracking algorithm and the manually annotated tracking box being greater than a given threshold. Figure 6 In [Table X], "Ours" is the name of the tracker proposed in the present invention, and ROMtrack, Mixformer, AiATrack, SparseTT, TransT, and OSTrack are the names of trackers proposed by other scholars in recent years. The numbers behind refer to the average precision rate and average accuracy of the trackers. For example, Figure 6As shown in the figure, when compared with advanced trackers in recent years on the general dataset OTB100, the tracking method Ours of the present invention has significantly improved in terms of precision and success rate. Figure 7 This is a comparison of the true tracking results of the present invention and advanced trackers OSTrack, Mixformer, ROMtrack, and TransT in recent years in the OTB100 dataset. As can be seen from Figure 7 it, when similar targets appear in the search area and during long-term tracking, the bounding boxes generated by the tracking method of the present invention are more accurate than those of other trackers.

[0112] In summary, the present invention proposes a new scheme that bridges the cross-channel interaction between the historical template and the search area through the original target template as a medium. Specifically, a bridging template interaction update module dedicated to tracking is designed, as shown in Figure 1 (c). The main advantage of this scheme is that the original template contains the original information of the target object, which can guide the extraction of target-related context from the historical template and the search area to adaptively enhance and supplement information. The template and the historical template can gradually reduce the interference from background noise during the cross-channel interaction through the bridging of the original template. The original template features are interactively updated from the target state and the search area of the previous frame through the bridging template interaction module. The principle of the template bridging interaction update (TBIU) module is as follows: Using the cross-attention mechanism, the original template is used as the query, the target state features of the previous frame are used as the key and value, and the target-related search area context information is collected into the original template medium. Then, the search area features are used as the query, and the fused original template is used as the key and value, and the target-related context is distributed from the medium to the search area. After multiple updates, the most representative target features in the entire video sequence are finally obtained. Finally, the context information of the fused template medium and the search area information are transmitted back to the original template to update the rich context collected between the historical template and the search area, realizing effective single-target tracking.

[0113] The above is the visual target tracking method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding visual target tracking device, including:

[0114] A model construction module for constructing a neural network, where the neural network includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module.

[0115] A model training module for collecting video sequences to construct a data set and using the data set to train the constructed neural network to obtain a target tracking model.

[0116] A target tracking module for inputting a video sequence containing a target to be tracked into the target tracking model, extracting the target historical template, the target original template, and the target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; performing an embedding operation on the target historical template, the target original template, and the target search area through the embedding module to obtain the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature; extracting features from the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature through the first Transformer module, the second Transformer module, and the third Transformer module to obtain the target historical template feature, the target original template feature, and the target search area feature; mapping the target historical template feature to a first key Key and a first value Value through the first attention mechanism module, mapping the target original template feature to a first query query, and performing an attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain the target-associated local context aggregation feature; mapping the target-associated local context aggregation feature to a second key Key and a second value Value through the second attention mechanism module, mapping the target search area feature to a second query query, and performing an attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain the target-associated local context assignment feature; mapping the target-associated local context assignment feature to a target detection result through the detection module and outputting the target to be tracked in each frame image.

[0117] For the specific limitations of the visual target tracking device, reference may be made to the limitations on the visual target tracking method in the foregoing text, which will not be elaborated herein. Each module in the foregoing visual target tracking device may be implemented in whole or in part by software, hardware, and their combination. Each of the foregoing modules may be embedded in the processor of the computer device in hardware form or independent thereof, or stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to each of the foregoing modules.

[0118] The present invention also provides a computer-readable storage medium storing a computer program, which can be used to execute the foregoing provided visual target tracking method.

[0119] The present invention also provides the structure of a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, other hardware required for other services may also be included. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the foregoing provided visual target tracking method.

[0120] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the foregoing method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided by the present invention may include at least one of non-volatile and volatile memories. The non-volatile memory may include a read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. The volatile memory may include a random access memory (RAM) or an external cache memory. By way of illustration and not limitation, the RAM may be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc.

[0121] The technical features of the foregoing embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the foregoing embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded by the present invention.

Claims

1. A visual target tracking method, characterized in that, Including: Construct a neural network, which includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes a first Transformer module, a second Transformer module, and a third Transformer module in parallel, a first attention mechanism module connected to the outputs of the first Transformer module and the second Transformer module, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module; Collect video sequences to construct a dataset, and use the dataset to train the constructed neural network to obtain a target tracking model; Input the video sequence containing the target to be tracked into the target tracking model, and extract the target historical template, the target original template, and the target search area in each frame image of the video sequence containing the target to be tracked through the template extraction module; Perform embedding operations on the target historical template, the target original template, and the target search area through the embedding module to obtain the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature; perform feature extraction on the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature through the first Transformer module, the second Transformer module, and the third Transformer module to obtain the target historical template feature, the target original template feature, and the target search area feature; Map the target historical template feature to a first key Key and a first value Value through the first attention mechanism module, map the target original template feature to a first query query, and perform attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain the target-associated local context aggregation feature; Map the target-associated local context aggregation feature to a second key Key and a second value Value through the second attention mechanism module, map the target search area feature to a second query query, and perform attention mechanism calculation based on the second key Key, the second value Value, and the second query query to obtain the target-associated local context assignment feature; Map the target-associated local context assignment feature to a target detection result through the detection module, and output the target to be tracked in each frame image.

2. The visual target tracking method according to claim 1, characterized in that, Before mapping the target historical template feature to the first key Key and the first value Value through the first attention mechanism module, perform spatial feature selection on the target historical template feature, specifically including: Generate the spatial mask M pre : Among them, is the attention weight of the i-th head in the n-th self-attention layer of the VIT model, where i ∈ 1, 2, …, N h , N h is the number of heads in the self-attention layer; Attn n is the attention weight of all self-attention layers, where n ∈ 1, 2, …, N, and N is the number of self-attention layers in the VIT model; A score is the attention score; denotes extracting the weights associated with the first score score from each head in A ; is the i-th mask corresponding to the i-th head; Use the generated spatial mask M pre Perform spatial feature selection on the target historical template features: Z pre = f pre ⊙M pre Among them, f pre is the patch feature in the target historical template feature; ⊙ represents element-wise multiplication, and Z pre is the target historical template feature after spatial feature selection.

3. The visual target tracking method according to claim 2, wherein Mapping the target historical template feature to the first key Key and the first value Value, mapping the target original template feature to the first query query, and performing attention mechanism calculation based on the first key Key, the first value Value, and the first query query to obtain the target-associated local context aggregation feature, specifically including: Perform cross-attention operations on the target original template feature and the target historical template feature after spatial feature selection; Among them, Z is the target original template feature; Z pre is the target historical template feature after spatial feature selection; represents the projection matrix; C is the number of feature channels; A t is the first cross-attention matrix; After integrating the target original template features and the first cross-attention matrix, generate target-associated local context aggregation features based on a multi-layer perceptron: Among them, is the integration result of the target original template feature and the first cross-attention matrix, and LN(·) represents the layer normalization operation; is the target associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

4. The visual target tracking method according to claim 3, wherein Mapping the target-associated local context aggregation features to a second key (Key) and a second value (Value), mapping the target search region features to a second query (query), and performing attention mechanism calculation based on the second key (Key), the second value (Value), and the second query (query) to obtain target-associated local context assignment features, specifically including: Perform cross-attention operation on the target search region features and the target-associated local context aggregation features: Among them, X is the target search area feature; is the target-associated local context aggregation feature; represents the projection matrix; C is the number of feature channels; A x is the second cross-attention matrix; After integrating the target original template features and the second cross-attention matrix, generate target-associated local context assignment features based on a multi-layer perceptron: Among them, is the integrated result of the target original template feature and the cross-attention matrix, and LN(·) represents the layer normalization operation; is the target-associated local context aggregation feature, and MLP(·) represents the multi-layer perceptron operation.

5. The visual target tracking method according to claim 1, characterized in that Extract the target historical template, the target original template, and the target search region in each frame image of the video sequence containing the target to be tracked, specifically including: Taking the center of the target in the first frame of the video sequence as a benchmark, expand the region by twice the side length outward, and this region contains the appearance information of the target and its surrounding local scene to obtain the target original template; Based on the target center coordinates in the previous frame image, expand the region by four times the side length outward, and this region covers the moving range of the target to obtain the target search region; Taking the position in the previous frame image that has the same size as the target search region and contains the target and the complete context information of its surrounding state as the target historical template.

6. The visual target tracking method according to claim 1, wherein Perform embedding operation on the target historical template, the target original template, and the target search region, specifically including: The target historical template, the target original template, and the target search area are respectively segmented and flattened to obtain a historical template block sequence P pre , an initial template block sequence P z , and a search area block sequence P X ; Perform linear projection on the historical die block sequence P respectively pre , the initial die block sequence P z and the search area block sequence P X to obtain the initial features of the target historical template, the target original template, and the target search area: Z pre = P pre W0 Z0 = P z W0 X0 = P x W0 where W0 is the learnable parameter of the linear projection, and Z pre , Z0, and X0 respectively represent the initial features of the target historical template, the target original template, and the target search area; Add the position encoding matrix E to the corresponding initial features of the target historical template, the target original template, and the target search area pre , E z and E x to provide position prior information and obtain the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature embedded with position encoding.

7. A visual target tracking device, characterized in that, Including: A model construction module for constructing a neural network, where the neural network includes a template extraction module, an embedding module, a context modeling module, and a detection module connected in sequence; the context modeling module includes parallel first, second, and third Transformer modules, a first attention mechanism module connected to the outputs of the first and second Transformer modules, and a second attention mechanism module connected to the output ends of the first attention mechanism module and the third Transformer module; A model training module for collecting video sequences to construct a data set and using the data set to train the constructed neural network to obtain a target tracking model; A target tracking module for inputting the video sequence containing the target to be tracked into the target tracking model, and extracting the target historical template, the target original template, and the target search region in each frame image of the video sequence containing the target to be tracked through the template extraction module; The embedding operation is performed on the target historical template, the target original template, and the target search area through an embedding module to obtain the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature; the feature extraction is performed on the target historical template embedding feature, the target original template embedding feature, and the target search area embedding feature through a first Transformer module, a second Transformer module, and a third Transformer module to obtain the target historical template feature, the target original template feature, and the target search area feature; The target historical template feature is mapped to a first key Key and a first value Value through a first attention mechanism module, the target original template feature is mapped to a first query query, and the attention mechanism calculation is performed based on the first key Key, the first value Value, and the first query query to obtain the target associated local context aggregation feature; The target associated local context aggregation feature is mapped to a second key Key and a second value Value through a second attention mechanism module, the target search area feature is mapped to a second query query, and the attention mechanism calculation is performed based on the second key Key, the second value Value, and the second query query to obtain the target associated local context allocation feature; The target associated local context allocation feature is mapped to a target detection result through a detection module, and the target to be tracked in each frame of image is output.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 6 above is implemented.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any one of claims 1 to 6 above is implemented.

Images