Single-target tracking method and device based on dynamic pruning and time sequence adaptive aggregation

Abstract

The application belongs to the technical field of single target tracking in computer vision, and discloses a single target tracking method and device based on dynamic pruning and time sequence adaptive aggregation; wherein the single target tracking method comprises: obtaining a target video sequence to be tracked, and labeling a bounding box of a target object in a first frame of the target video sequence as a template graph; based on the target video sequence and the template graph, a pre-trained deep target tracking network is used to perform single target tracking on the target object in the subsequent frames of the target video sequence, and a single target tracking result is obtained. In the technical scheme disclosed by the application, the target is tracked by the pre-trained deep target tracking network, a dynamic pruning module is arranged in the deep target tracking network, and a token time sequence adaptive aggregation method is proposed, which can effectively improve the accuracy of target tracking.

Description

Technical Field

[0001] This invention belongs to the field of single-object tracking technology in computer vision, and specifically relates to a single-object tracking method and apparatus based on dynamic pruning and temporal adaptive aggregation. Background Technology

[0002] Single-target tracking refers to the technique of real-time localization and tracking of a specific target (interpretably marked in the initial frame) in a video sequence. It can be widely used in fields such as autonomous driving, intelligent monitoring, robot navigation, and motion analysis, and is a key technology in computer vision for achieving target detection and continuous tracking in dynamic scenes.

[0003] Single-target tracking tasks typically face complex dynamic environmental interference (interpretably, such as changes in target appearance, background interference, occlusion, changes in illumination, motion blur, etc.), and the target's feature information is easily lost or confused, which affects the accuracy of single-target tracking tasks. More specifically, targets in single-target tracking usually have inter-class similarity and intra-class differences (for example, different targets have similar appearances or the same target has different appearances in different time frames), which makes target tracking more difficult.

[0004] Currently, existing single-object tracking methods can be divided into two categories based on their technical processes and development stages: traditional single-object tracking methods and deep learning-based single-object tracking methods. Traditional single-object tracking methods primarily rely on manually designed target features (such as color, texture, and shape) combined with target matching algorithms (such as particle filtering and Kalman filtering) for tracking. This approach depends on the participation of a large number of experienced professional researchers, resulting in high labor costs. It is unsuitable for target tracking tasks in complex and dynamic scenes, and with the increasing scale of video data, the tracking accuracy and efficiency of traditional methods are insufficient to meet practical needs, making it less feasible. In contrast, deep learning-based single-object tracking methods complete the target feature extraction and temporal information modeling process through deep networks, achieving end-to-end automatic tracking with higher tracking accuracy, efficiency, and robustness.

[0005] Existing deep learning-based single-object tracking methods mainly rely on Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). CNNs utilize the translation invariance of images as prior knowledge, aggregating pixel information layer by layer through convolution operations to effectively learn target features. While this approach offers high tracking efficiency and robustness, it is ineffective in capturing salient features and temporal information of targets in complex dynamic scenes (e.g., when targets are occluded or undergo significant appearance changes). Therefore, establishing an efficient visual cognitive computing module is crucial for enhancing salient information capture in complex scenes and improving the accuracy of single-object tracking. Current state-of-the-art methods largely employ Vision Transformers, which model long-distance dependencies based on Multi-Head Self-Attention (MHSA) mechanisms. In existing multi-stage Vision Transformer tracking tasks, feature extraction is often performed stage by stage, ultimately outputting a feature map for target localization.

[0006] However, the computational complexity of multi-head self-attention is proportional to the square of the input sequence length, resulting in high computational overhead, especially when dealing with long input sequences. Meanwhile, single-object tracking methods typically rely on high-confidence historical frames as dynamically updated template maps to model video-level temporal information. This method requires two-stage training of the network, which is not only complex and difficult to train, but also makes it difficult to effectively extract video-level temporal features. Therefore, further research is urgently needed to improve the success rate of visual Transformers in single-object tracking tasks. Summary of the Invention

[0007] The purpose of this invention is to provide a single-target tracking method and apparatus based on dynamic pruning and temporal adaptive aggregation, to solve the technical problem of high mistracking caused by insufficient capture of target appearance salience features and video-level temporal information features in existing technologies. The technical solution disclosed in this invention tracks the target using a pre-trained deep target tracking network. The deep target tracking network incorporates a dynamic pruning module and proposes a token temporal adaptive aggregation method, which can effectively improve the accuracy of target tracking. Specifically, this invention uses a dynamic pruning module to trim background information in the search graph, increasing the model's attention to target salience information. It also proposes a token temporal adaptive aggregation method to establish a long-term memory token library and a short-term memory token library. A directed attention encoder module models long-distance dependencies containing target feature information and video-level temporal feature information, thereby improving the tracking accuracy of single-target tracking tasks.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] In a first aspect, this invention provides a single-target tracking method based on dynamic pruning and temporal adaptive aggregation, comprising the following steps:

[0010] Obtain the target video sequence to be tracked, and annotate the bounding box of the target object in the first frame of the target video sequence as a template image;

[0011] Based on the target video sequence and the template image, a pre-trained deep target tracking network is used to perform single-target tracking on the target objects in subsequent frames of the target video sequence to obtain single-target tracking results.

[0012] The deep target tracking network includes: a memory token library for storing memory tokens, and a patch embedding and position encoding, stage 1, stage 2, stage 3, stage 4, and a convolutional network tracking head connected in sequence; wherein, the memory token library includes a long-term memory token library for storing long-term memory tokens and a short-term memory token library for storing short-term memory tokens; the patch embedding and position encoding is used to input template graphs and search graphs, and to tokenize and position-encode the template graphs and search graphs, outputting template graph tokens and search graph tokens; stage 1, stage 2, and stage 3 each include several directional attention encoder modules and dynamic... The dynamic pruning module includes several directional attention encoder modules connected in series. The directional attention encoder module takes into account long-term memory tokens, short-term memory tokens, and template graph tokens and search graph tokens output from the previous module, and outputs template graph tokens and search graph tokens after directional attention encoding. The dynamic pruning module takes into account search graph tokens output from the directional attention encoder modules in the same stage, and outputs high-quality search graph tokens after pruning. The memory token library takes into account tokens in the current memory token library and search graph tokens that meet preset quality requirements, and outputs long-term memory tokens and short-term memory tokens.

[0013] A further improvement of the technical solution of the present invention is that the directed attention encoder module includes a first LayerNorm normalization operation, a directed attention module, a second LayerNorm normalization operation, and a feedforward neural network layer connected in series. In addition, the input of the first LayerNorm normalization operation is added to the output of the directed attention module through a skip connection layer to serve as the input of the second LayerNorm normalization operation, and the input of the second LayerNorm normalization operation is added to the output of the feedforward neural network to serve as the output of the directed attention encoder module.

[0014] A further improvement of the technical solution of the present invention is that the directional attention module is composed of a multi-head attention module, and the input includes long-term memory tokens, short-term memory tokens, template graph input tokens and search graph input tokens, and the four types of tokens share linear mapping weights;

[0015] Among them, the key K is calculated through a linear mapping of the long-term memory token. LT Value V LT Matrix; Short-Time Memory tokens are used to calculate key K through linear mapping. ST Value V ST Matrix; Template diagram input token is used to calculate query Q through linear mapping. T Key K T Value V T Matrix; Search graph input token is used to calculate query Q through linear mapping. S Key K S Value V S matrix;

[0016] Multi-head attention module input query Q T Key K T Value V T Matrix, output template diagram output token; key K LT K ST K T K S Key K is obtained by splicing. [LT,ST,T,S] Value V LT V ST V T V S The value V is obtained by splicing. [LT,ST,T,S] Multi-head attention module input query Q S Key K [LT,ST,T,S] Value V [LT,ST,T,S] The matrix outputs the search graph and the tokens.

[0017] A further improvement of the technical solution of the present invention is that the dynamic pruning module includes: a scorer, a cross-attention aggregator, and an auxiliary tracking head; wherein, the scorer takes a learnable query token matrix and search graph tokens as input, outputs an attention heatmap as a score matrix, and adds the score matrix according to the query token dimension to obtain the search graph token score, and retains a preset number of items before scoring the search graph tokens; during the training phase, the cross-attention aggregator takes the output of the scorer and the search graph token value as input, and the auxiliary tracking head takes the output of the cross-attention aggregator as input and outputs the prediction result; during the inference phase, only the scorer is used; in addition, the dynamic pruning module stops the backpropagation operation on the search graph tokens.

[0018] A further improvement of the technical solution of the present invention is that, in the dynamic pruning module, during training, the prediction results are used to adjust the parameters in the learnable query token matrix Q, the scorer, the cross attention aggregator and the auxiliary tracking head using gradient descent.

[0019] The scoring mechanism of the scorer is represented as follows:

[0020]

[0021] X K =TopK(X|Score);

[0022] In the formula, Q is the learnable query matrix; K is the key, i.e., the search graph token; N is the number of query tokens in Q; Score is the score of the search graph tokens by the dynamic pruning module; X is the input search graph token; X K Reserve a predetermined number of tokens before the output score; (·) T This represents the matrix transpose; TopK(·) represents retaining a preset number of tokens before scoring;

[0023] The cross-attention aggregator is represented as:

[0024]

[0025] X″=MLP(LN(X′))+X′;

[0026] In the formula, V is the value, i.e., the search graph token; d is the scaling factor; X′ represents the output of the attention operation in the cross-attention aggregator; X″ represents the output of the cross-attention aggregator; Softmax(·) is the exponential normalization function; (·) T LN(·) represents matrix transpose; LN(·) represents linear mapping; MLP(·) represents multilayer perceptron.

[0027] A further improvement of the technical solution of the present invention lies in that, during the training process of the deep target tracking network, gradient propagation is simultaneously performed on the auxiliary tracking heads of the three stages of dynamic pruning modules and the convolutional network tracking head, and the total loss function is expressed as:

[0028] L = L cls +λ iou L iou +λ L1 L1;

[0029]

[0030]

[0031] In the formula, L cls L iouL1 and L1 represent the classification loss, intersection-over-union loss, and absolute value loss, respectively; λ iou and λ L1 These are the weight parameters for the crossover ratio loss and the absolute value loss, respectively; L and Let α represent the loss functions of the convolutional network tracking head and the i-th auxiliary tracking head, respectively; i is the weight parameter of the loss value of the i-th auxiliary tracking head in the total loss function; Loss represents the total loss function of the deep target tracking network.

[0032] A further improvement of the technical solution of the present invention is that, in the memory token library, the long-term memory tokens stored in the long-term memory token library are memory tokens with longer temporal associations, and are updated by a preset number of items before the memory token library scoring output in stage 4; the short-term memory tokens stored in the short-term memory token library are memory tokens with shorter temporal associations, and are updated by a preset number of items before the search graph token scoring output in stage 4.

[0033] in,

[0034] The scoring mechanism is expressed as follows:

[0035] K [LT,ST,T,S] =Concat(K LT， K ST， K T， K S );

[0036] attn = Q S (K [LT,ST,T,S] ) T ;

[0037]

[0038] In the formula, K LT K ST K T K S These are the keys output through a linear mapping of long-term memory tokens, short-term memory tokens, template graph tokens, and search graph tokens, respectively; K [ST,LT,T,S] A matrix connecting the keys of four types of tokens; Q S `attn` represents the query output by the search graph tokens through a linear mapping; `attn` represents the attention heatmap recorded in the last directional attention encoder module of stage 4; `C` is the classification score map output by the convolutional network tracking head; `M` is the number of remaining search graph tokens after three prunings; `Score` is the score result for long-term memory tokens, short-term memory tokens, template graph tokens, and remaining search graph tokens after three prunings; `Concat(·)` represents matrix concatenation; `(·)` T Indicates matrix transpose;

[0039] The update scheme for the long-term memory token library can be represented as follows:

[0040] Score LT =Score(Index) LT );

[0041] Score ST =Score(Index) ST );

[0042] Score [LT,ST] =Concat(Score) LT Score ST );

[0043] X [LT,ST] =Concat(X) LT， X ST )

[0044] X′ LT =TopK(X [LT,ST] Score [LT,ST] );

[0045] In the formula, Index LT and Index ST These are the indices in which long-term memory tokens and short-term memory tokens participate in the calculation of the Score; Score LT and Score ST The score represents the rating of long-term and short-term tokens. [LT,ST] The connection matrix for long-term and short-term token scores; X LT and X ST These are the long-term memory token and the short-term memory token before the update; X [LT,ST] This is the connection matrix of long-term memory tokens and short-term memory tokens before the update; X′ LT For the updated long-term memory tokens; TopK(·) indicates that a preset number of tokens should be retained before the rating;

[0046] The update scheme for the short-time memory token library can be represented as:

[0047] Score S =Score(Index) S );

[0048] X′ S =X S +Embeding(Round(C));

[0049] X′ ST =TopK(X′) S ScoreS );

[0050] In the formula, Index S This indicates that only the indexes in the Score search graph tokens are used in the calculation. S This represents the score of the remaining search graph tokens after three prunings; X S The output of stage 4 represents the search graph token; Round(·) represents the rounding function; Embedding(·) represents the embedding layer, which consists of a single linear layer; X′ S X′ represents the search graph feature token generated by adding the search graph token output from stage 4 to the embedding layer output. ST For the updated short-term memory token.

[0051] In a second aspect, the present invention provides a single-target tracking system based on dynamic pruning and temporal adaptive aggregation, comprising:

[0052] The data acquisition module is used to acquire the target video sequence to be tracked and to annotate the bounding box of the target object in the first frame of the target video sequence as a template image.

[0053] The tracking processing module is used to perform single-target tracking on target objects in subsequent frames of the target video sequence based on the target video sequence and the template image, using a pre-trained deep target tracking network, to obtain single-target tracking results.

[0054] The deep target tracking network includes: a memory token library for storing memory tokens, and a patch embedding and position encoding, stage 1, stage 2, stage 3, stage 4, and a convolutional network tracking head connected in sequence; wherein, the memory token library includes a long-term memory token library for storing long-term memory tokens and a short-term memory token library for storing short-term memory tokens; the patch embedding and position encoding is used to input template graphs and search graphs, and to tokenize and position-encode the template graphs and search graphs, outputting template graph tokens and search graph tokens; stage 1, stage 2, and stage 3 each include several directional attention encoder modules and dynamic... The dynamic pruning module includes several directional attention encoder modules connected in series. The directional attention encoder module takes into account long-term memory tokens, short-term memory tokens, and template graph tokens and search graph tokens output from the previous module, and outputs template graph tokens and search graph tokens after directional attention encoding. The dynamic pruning module takes into account search graph tokens output from the directional attention encoder modules in the same stage, and outputs high-quality search graph tokens after pruning. The memory token library takes into account tokens in the current memory token library and search graph tokens that meet preset quality requirements, and outputs long-term memory tokens and short-term memory tokens.

[0055] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a single-target tracking method based on dynamic pruning and temporal adaptive aggregation as described in any one of the first aspects of the present invention.

[0056] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the single-target tracking method based on dynamic pruning and temporal adaptive aggregation as described in any one of the first aspects of the present invention.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] This invention provides a single-target tracking method based on dynamic pruning and temporal adaptive aggregation. It uses a pre-trained deep target tracking network for tracking, addressing the high mistracking rate caused by insufficient capture of target appearance salience features and video-level temporal information features in existing technologies. Specifically, this invention uses a dynamic pruning module to trim background information in the search graph, increasing the deep target tracking network's attention to target salience information. Furthermore, this invention proposes a token temporal adaptive aggregation method to establish long-term memory and short-term memory token libraries, relying on a directed attention encoder module to model long-distance dependencies containing target feature information and video-level temporal feature information, effectively improving the tracking accuracy of single-target tracking tasks.

[0059] The deep target tracking network disclosed in this invention includes patch embedding and position encoding, four stages, a convolutional network tracking head, and a memory token library. Stages 1, 2, and 3 each include a directional attention encoder module and a dynamic pruning module, while stage 4 includes only the directional attention encoder module. The directional attention encoder module includes a directional attention module, which processes feature tokens interactively by category, extracting temporal and target features by jointly modeling four types of tokens: long-term memory tokens, short-term memory tokens, template graph tokens, and search graph tokens. The dynamic pruning module prunes the search graph tokens, effectively eliminating background noise information in the search graph and reducing model computational complexity. The memory token library stores high-quality historical frame feature tokens, and efficiently and adaptively aggregates video-level temporal information by updating the memory token library. In summary, this invention proposes a novel directed attention encoder module for information fusion between feature tokens and enhancement of salient features. It also designs a directed attention module and a memory token library for modeling long-range dependencies in video-level temporal information, and proposes a dynamic pruning module to eliminate background noise in the search graph and reduce network computation. Based on these, a deep target tracking network is constructed, addressing existing problems in target tracking solutions. This allows for salient feature extraction and improves target tracking accuracy. Furthermore, the proposed directed attention encoder module, dynamic pruning module, and memory token library are all based on basic matrix operations, with consistent input and output data formats, requiring minimal dependencies and facilitating application in various deep target tracking models, thus demonstrating broad application prospects. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 This is a flowchart illustrating a single-target tracking method based on dynamic pruning and temporal adaptive aggregation provided in an embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the structure of a deep target tracking network in an embodiment of the present invention.

[0063] Figure 3 This is a schematic diagram of the structure of the directional attention encoder module in an embodiment of the present invention.

[0064] Figure 4 This is a schematic diagram of the structure of the directional attention module in an embodiment of the present invention.

[0065] Figure 5 This is a schematic diagram of the dynamic pruning module during the training process in an embodiment of the present invention.

[0066] Figure 6 This is a schematic diagram of the dynamic pruning module in the reasoning process in an embodiment of the present invention.

[0067] Figure 7 This is a schematic diagram of target tracking results for three sets of video sequences in a specific embodiment of the present invention.

[0068] Figure 8 This is a schematic diagram of a single-target tracking system based on dynamic pruning and temporal adaptive aggregation provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention; obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0070] Based on the technical solutions disclosed in the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0071] Please see Figure 1 In this embodiment of the invention, a single-target tracking method based on dynamic pruning and temporal adaptive aggregation is provided, comprising the following steps:

[0072] Step 1: Obtain the target video sequence to be tracked; specifically, the video sequence to be tracked can be RGB video data captured in any scene, and all frames of the video contain the target to be tracked.

[0073] Step 2: Label the bounding box of the target object in the first frame of the video sequence as a template image; specifically, label the target object in the first frame of the video to be tracked, and use the processed data as a template image.

[0074] Step 3: Based on the target video sequence and template image to be tracked, a pre-trained deep target tracking network is used to track the target in subsequent frames of the sequence to obtain the tracking results of the video sequence; specifically, the tracking results are output in the form of the bounding box position of the target in each frame of the video.

[0075] Please see Figure 2 , Figure 2 The symbol C in the diagram represents concatenating two sets of input tokens. The aforementioned deep target tracking network includes, in sequence, patch embedding and position encoding, stage 1, stage 2, stage 3, stage 4, and a convolutional network tracking head, as well as a memory token library for storing memory tokens. Stages 1, 2, and 3 each include several directional attention encoder modules and a dynamic pruning module connected in series. Stage 4 includes several directional attention encoder modules connected in series. The convolutional network tracking head includes a three-layer convolutional neural network. The memory token library includes a long-term memory token library and a short-term memory token library. More specifically, patch embedding and position encoding are used to input the template image. The module takes a search graph as input and outputs template graph tokens and search graph tokens. The directed attention encoder module takes long-term memory tokens, short-term memory tokens, and template graph tokens and search graph tokens output by the previous module as input and outputs tokens (including template graph tokens and search graph tokens) from the directed attention encoder module after feature extraction and interaction. The dynamic pruning module takes the search graph tokens output by the directed attention encoder module in the same stage as input and outputs high-quality pruned search graph tokens, the number of which is 80% (rounded up) of the total number of input tokens. The memory token library takes high-quality tokens from the current memory token library and search graph tokens as input and outputs long-term memory tokens and short-term memory tokens.

[0076] In the technical solution of this invention, addressing the problem of insufficient capture of salient features of target appearance and video-level temporal information features in existing technologies, the single-target tracking method provided by this invention fully considers target feature extraction and video-level temporal information modeling mechanisms, which can effectively improve the accuracy of target tracking and reduce false tracking. The method of this invention is based on the visual Transformer framework, introducing visual cognitive computing into the field of target tracking, which can effectively improve the accuracy of target tracking. Furthermore, each module is simple to implement, has few dependencies, and possesses strong applicability.

[0077] In this embodiment of the invention, the tracking process specifically includes the following steps:

[0078] Step 1: Use the GOT10k open dataset for single-object tracking as the experimental dataset. This dataset contains a rich variety of object categories and diverse scenes, along with high-quality object annotations. The dataset is divided into training, validation, and test sets according to official standards. The training set includes 9335 video sequences, the validation set includes 180 video sequences, and the test set includes 180 video sequences. Load the dataset and transfer it to the computer executing the algorithm for further processing.

[0079] Step 2: Construct the target tracking network, including sequentially connected patch embedding and position encoding, directional attention encoder modules, dynamic pruning modules, and a convolutional network tracking head, as well as a memory token library. The deep target tracking network comprises four stages, each containing 4, 3, 3, and 2 directional attention encoder modules and 1, 1, 1, and 0 dynamic pruning modules, respectively. More specifically, patch embedding and position encoding tokenizes and positions the input template graph and search graph. 768 convolutional operations of size 16×16 with a stride of 16 are used for tokenization. Position encoding is a learnable parameter. The tokenized parameters are added to the position encoding parameters to output template graph tokens and search graph tokens.

[0080] Please see Figure 3 , Figure 3 The symbol C indicates concatenating two sets of input tokens, and the symbol + indicates summing the two sets of input tokens; the directed attention encoder module includes a first LayerNorm layer normalization operation, a directed attention module, a second LayerNorm layer normalization operation, and a feedforward neural network layer.

[0081] In a further explanatory technical solution, in the directional attention encoder module,

[0082] The first LayerNorm normalization operation is used to input long-term memory tokens, short-term memory tokens, and input tokens from the directional attention encoder module, and outputs tokens processed by the first LayerNorm normalization operation.

[0083] The directed attention module is used to input the token processed by the normalization operation of the first LayerNorm layer, and output the token processed by the directed attention module.

[0084] The second LayerNorm normalization operation is used to input the sum of the tokens processed by the directed attention module and the tokens input by the directed attention encoder module, and outputs the tokens processed by the second LayerNorm normalization operation;

[0085] The feedforward neural network layer is used as input for the token processed by the normalization operation of the second LayerNorm layer, and outputs the token processed by the feedforward neural network.

[0086] The output token of the directional attention encoder module is the sum of the token processed by the feedforward neural network and the input token of the normalization operation of the second LayerNorm layer.

[0087] Please see Figure 4 , Figure 4The symbol C indicates that two sets of input tokens are concatenated; the directional attention module is composed of a multi-head attention module, and the module inputs include long-term memory tokens, short-term memory tokens, template graph input tokens, and search graph input tokens;

[0088] The targeted attention module can be represented as:

[0089] K [LT,ST,T,S] =Concat(K LT， K ST， K T， K S );

[0090] V [LT,ST,T,S] =Concat(V LT， V ST， V T， V S );

[0091]

[0092] In the formula, K LT K ST K T K S The key K is output through a linear mapping of the long-term memory token, short-term memory token, template graph token, and search graph token, respectively. [LT,S,T,S] A K matrix is ​​formed by connecting the keys K of the four types of tokens; V LT V ST V T V S The keys V are the outputs of long-term memory tokens, short-term memory tokens, template graph tokens, and search graph tokens through a linear mapping, respectively. [LT,S,T,S] A V matrix is ​​formed by connecting the keys V of the four types of tokens; Q T Q S The query Q is output by linear mapping between the template graph token and the search graph token, respectively; Attention T Attention S These are the template graph token and search graph token output by the multi-head self-attention module, respectively; d is the scaling factor; Concat(·) represents matrix concatenation; Softmax(·) is the exponential normalization function; (·) T This indicates the matrix transpose.

[0093] In this embodiment of the invention, please refer to 5 and Figure 6 In the diagram, the symbol ∑ represents summing the input tokens along a certain dimension; the dynamic pruning module includes a scorer, a cross-attention aggregator, and an auxiliary tracking head; specifically, during model training, please refer to... Figure 5The output of the scorer, along with the search graph token value V, needs to be used as the input to the cross-attention aggregator. The structure of the auxiliary tracking head is consistent with that of the convolutional network tracking head, taking the output of the cross-attention aggregator as input and outputting the prediction result. During training, the dynamic pruning module uses the prediction result to perform gradient descent adjustments on the parameters in the learnable query token matrix Q, the scorer, the cross-attention aggregator, and the auxiliary tracking head. For model inference, please refer to [link to relevant documentation]. Figure 6 Only the scorer is needed; it is particularly important to note that the dynamic pruning module stops backpropagation of search graph tokens;

[0094] The scoring mechanism can be represented as:

[0095]

[0096] X k =TopK(X|Score);

[0097] In the formula, Q is the learnable query matrix, K is the key (i.e., the search graph token); N is the number of query tokens in the query matrix Q; Score is the score of the search graph tokens by the dynamic pruning module; X is the input search graph token; X K Reserve the top K tokens for the output score; (·) T This represents the matrix transpose; TopK(·) represents retaining the top K tokens by score; specifically, the output quantity of each dynamic pruning module is 80% (rounded up) of the total input tokens; according to this embodiment of the invention, the number of query tokens N is 256, the number of input tokens of the dynamic pruning module in stage 1 is 256, and the number of output tokens K is 205; the number of input tokens of the dynamic pruning module in stage 2 is 205, and the number of output tokens K is 164; the number of input tokens of the dynamic pruning module in stage 3 is 164, and the number of output tokens K is 132;

[0098] The cross-attention aggregator can be represented as:

[0099]

[0100] X″=MLP(LN(X′))+X′;

[0101] In the formula, Q is the learnable query matrix, K is the key (i.e., search graph token), V is the value (i.e., search graph token); d is the scaling factor; X′ represents the output of the attention operation in the cross-attention aggregator; X″ represents the output of the cross-attention aggregator; Softmax(·) is the exponential normalization function; (·) T LN(·) represents matrix transpose; LN(·) represents linear mapping; MLP(·) represents multilayer perceptron.

[0102] In this embodiment of the invention, the tracking module uses a convolutional network tracking head to generate the target bounding box; the convolutional network tracking head is a commonly used module in the field of target tracking.

[0103] Step 3: Data preprocessing. Before inputting the data into the network for training, sample three template images, crop and fill the target regions, and then scale them to a size of 128×128. Sample four search images, randomly perturb the target regions, crop and fill them, and then scale them to a size of 256×256.

[0104] Step 4: During training, the first two template images are used to extract features through all directional attention encoder modules in the network to initialize the memory token library. Then, the third template image and the first search image are used as image pairs input to the network. The convolutional network tracking head outputs the tracking results and updates the memory token library. This process is repeated until the fourth search image is processed. The above operations yield tracking results for the four search images. In each training step, backpropagation is performed starting from the loss function value. The AdamW optimizer is used to optimize the network parameters based on the gradient information obtained from backpropagation, thereby guiding the neural network to achieve accurate target tracking results based on the input images.

[0105] In this embodiment of the invention, the memory token library includes a long-term memory token library and a short-term memory token library;

[0106] The scoring mechanism can be represented as:

[0107] K [LT,ST,T,S] =Concat(K LT， K ST， K T， K S );

[0108] attn = Q S (K [LT,ST,T,S] ) T ;

[0109]

[0110] In the formula, K LT K ST K T K S The key K is output through a linear mapping of the long-term memory token, short-term memory token, template graph token, and search graph token, respectively. [ST,LT,T,S] A K matrix is ​​formed by connecting the keys K of the four types of tokens; Q SQuery Q is the output of the search graph tokens through a linear mapping; attn represents the attention heatmap recorded in the last directional attention encoder module of stage 4; C is the classification score map output by the convolutional network tracking head; M is the number of remaining search graph tokens after three prunings; Score is the score result for long-term memory tokens, short-term memory tokens, template graph tokens, and remaining search graph tokens after three prunings; Concat(·) represents matrix concatenation; (·) T Indicates matrix transpose;

[0111] The update scheme for the long-term memory token library can be represented as follows:

[0112] Score LT =Score(Index) LT );

[0113] Score ST =Score(Index) ST );

[0114] Score [LT,ST] =Concat(Score) LT Score ST );

[0115] X [LT,ST] =Concat(X) LT， X ST )

[0116] X′ LT =TopK(X) [LT,ST] Score [LT,ST] );

[0117] In the formula, Score represents the rating of long-term memory tokens, short-term memory tokens, template graph tokens, and remaining search graph tokens after three prunings; Index LT and Index ST These are the indices in which long-term memory tokens and short-term memory tokens participate in the calculation of the Score; Score LT and Score ST The score represents the rating of long-term and short-term tokens. [LT,ST] The connection matrix for long-term and short-term token scores; X LT and X ST These are the long-term memory token and the short-term memory token before the update; X [LT,ST] This is the connection matrix of long-term memory tokens and short-term memory tokens before the update; X′ LTFor the updated long-term memory tokens; TopK(·) means retaining the top K tokens in the score; Concat(·) means matrix concatenation; Specifically, the number of memory tokens in both the long-term memory token library and the short-term memory token library is 64, so the number of input tokens in the long-term memory token library is 128, and the number of output tokens K is 64;

[0118] The update scheme for the short-time memory token library can be represented as:

[0119] Score S =Score(Index) S );

[0120] X′ S =X S +Embeding(Round(C));

[0121] X′ ST =TopK(X′) S Score S );

[0122] In the formula, Score represents the rating of long-term memory tokens, short-term memory tokens, template graph tokens, and remaining search graph tokens after three prunings; Index S This indicates that only the search graph tokens are used in the calculation of the Score; Score S This represents the score of the remaining search graph tokens after three prunings; X S The search graph token is the output of stage 4; C is the classification score map output by the convolutional network tracking head; Round(·) represents the rounding function; Embedding(·) represents the embedding layer, which consists of a single linear layer; X′ S This represents the search graph feature token generated by adding the search graph token output from stage 4 to the embedding layer output; X′ ST The updated short-term memory tokens; TopK(·) represents retaining the top K tokens by score; specifically, the remaining search graph tokens after three prunings are 132, so the input tokens of the short-term memory token library are 132, and the output tokens K are 64.

[0123] In this embodiment of the invention, the deep target tracking network needs to be trained simultaneously on the auxiliary tracking heads of the three dynamic pruning modules and the convolutional network tracking head during the training process. The total loss function is expressed as:

[0124] L = L cls +λ iou L iou +λ L1 L1;

[0125]

[0126]

[0127] In the formula, L cls L iou L1 and L1 represent the classification loss, intersection-over-union loss, and absolute value loss, respectively; λ iou and λ L1 These are the weight parameters for the crossover ratio loss and the absolute value loss, respectively; L and Let α represent the loss functions of the convolutional network tracking head and the i-th auxiliary tracking head, respectively; i λ is the weight parameter of the loss value of the i-th auxiliary tracking head in the total loss function; Loss represents the total loss function of the deep target tracking network. Specifically, λ iou Set to 2, λ L1 Set to 5, α i (i = 1, 2, 3) are all set to 0.5.

[0128] This invention applies visual cognitive computing technology to the field of automatic tracking. A new directional attention encoder module is proposed for information fusion between feature tokens and enhancement of saliency features. A directional attention module and a memory token library are designed to model long-distance dependencies of video-level temporal information. A dynamic pruning module is proposed to eliminate background noise in the search graph and reduce network computation. Based on this, a target tracking network is constructed to solve target tracking tasks. Addressing the technical problem of high mistracking caused by insufficient capture of saliency features of target appearance and video-level temporal information features in existing technologies, this invention uses a visual cognitive method based on directional attention to improve the attention to the features of the tracked target. A directional attention module and a memory token library are proposed to further efficiently model information fusion between feature tokens and adaptively aggregate video-level temporal feature information. A dynamic pruning module is designed to eliminate background noise in the search graph, enabling saliency feature extraction and improving target tracking accuracy. The directional attention encoder module, dynamic pruning module, and memory token library proposed in this invention are all based on basic matrix operations, with consistent input and output data formats, requiring minimal dependencies and facilitating application in various deep target tracking models, thus possessing broad application prospects.

[0129] In a specific exemplary embodiment of the present invention: the GOT10k target tracking dataset used contains 9335 training set video sequences, 180 validation set video sequences, and 180 test set video sequences; 15000 sequence groups are sampled from the training set and 2500 sequence groups are sampled from the validation set, each sequence group including 3 template images and 4 search images. The template images are scaled to 128×128 pixels during preprocessing, and the search images are scaled to 256×256 pixels during preprocessing. In this embodiment, the target tracking network is initialized randomly, and the weights of the backbone network model (patch embedding and position encoding, and directional attention encoder modules at each stage) pre-trained based on the classification task are loaded. The operating environment is a computer with frameworks such as PyTorch, capable of reading a given image and completing the construction and training of the model of this method. The training time for this embodiment on a Gold6626R@2.90GHz CPU, 8GB of memory, and two NVIDIA GeForce RTX3090 GPUs is approximately 28 hours.

[0130] In this embodiment of the invention, the specific implementation steps include: first, setting relevant training parameters, setting the optimizer used for network updates in this invention to the AdamW optimizer, setting the initial learning rate to 0.0002, and setting the weight decay coefficient to 0.0001. The learning rate adjustment strategy is set to fixed step decay, the decay period to 80, and the decay rate to 0.1. The deep target tracking network of this embodiment consists of a patch embedding and location encoding module, a four-stage feature extraction module, a convolutional network tracking head module, and a memory token library. The four stages of the feature extraction module respectively include 4, 3, 3, and 2 directional attention encoder modules and 1, 1, 1, and 0 dynamic pruning modules. The network input is an image pair consisting of a template image and a search image. The directional attention encoder module and the dynamic pruning module extract target texture and abstract semantic information and eliminate background information layer by layer. The output feature tokens undergo feature extraction in multiple stages and are finally passed to the convolutional network tracking head module to obtain the target bounding box tracking result. Next, when training the network using the partitioned dataset, eight sequence groups are randomly selected from the training set and input into the network each time. The selected AdamW optimizer is used to update the parameters, and training is completed after 100 iterations on the dataset. Finally, the sequence groups from the test set are input into the trained network for tracking, and the results of the embodiment of the invention are obtained. Please refer to [link to relevant documentation]. Figure 7In the figure, the symbol # represents the tracking result of a certain frame in the sequence. For example, #48 refers to the tracking result of the 48th frame of the sequence. From the target detection bounding box results in each frame of the sequence given in the embodiment, the method proposed in this invention achieves good target tracking results. In the three sets of video sequences given, the position of the target in each frame of the sequence is accurately predicted, demonstrating the effectiveness of this invention in target detection tasks.

[0131] In summary, this invention discloses a single-target tracking method based on dynamic pruning and temporal adaptive aggregation. The method employs a directional attention encoder module for information fusion between feature tokens and enhancement of saliency features. A directional attention module and a memory token library are designed to model long-distance dependencies of video-level temporal information. A dynamic pruning module is proposed to eliminate background noise in the search graph and reduce network computation. Based on this, a target tracking network is constructed to solve target tracking tasks. This invention applies visual cognitive computing to the field of automatic tracking, effectively improving target tracking accuracy. Furthermore, the modules are simple to implement, require minimal dependencies, and have strong applicability. It can solve the technical problem of high false tracking caused by insufficient capture of target appearance saliency features and video-level temporal information features in existing technologies.

[0132] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0133] Please see Figure 8 In this embodiment of the invention, a single-target tracking system based on dynamic pruning and temporal adaptive aggregation is provided, comprising:

[0134] The video acquisition module acquires the target video sequence to be tracked;

[0135] The template image acquisition module marks the bounding box of the target object in the first frame of the video sequence as the template image;

[0136] The tracking module is used to track the target in subsequent frames of the sequence based on the target video sequence and template image to be tracked, and to obtain the tracking result of the video sequence.

[0137] The deep target tracking network includes, in sequence, patch embedding and position encoding, stage 1, stage 2, stage 3, stage 4, and a convolutional network tracking head, as well as a memory token library for storing memory tokens. Specifically, stage 1, stage 2, and stage 3 each include several directional attention encoder modules and dynamic pruning modules connected in series; stage 4 includes several directional attention encoder modules connected in series; the convolutional network tracking head includes a three-layer convolutional neural network; and the memory token library includes a long-term memory token library and a short-term memory token library.

[0138] The directed attention encoder module includes a first LayerNorm layer normalization operation, a directed attention module, a second LayerNorm layer normalization operation, and a feedforward neural network layer. It adds the input of the first LayerNorm layer normalization operation to the output of the directed attention module and adds the input of the second LayerNorm layer normalization operation to the output of the feedforward neural network layer through skip connection layers.

[0139] The dynamic pruning module includes a scorer, a cross-attention aggregator, and an auxiliary tracking head. The scorer takes a learnable query token matrix Q and search graph tokens K as input, outputs an attention heatmap as a score matrix, and sums the score matrices along the query token dimension to obtain the search graph token score, retaining the top K items of the search graph token score. During model training, the cross-attention aggregator takes the scorer's output and the search graph token value V as input. The auxiliary tracking head takes the cross-attention aggregator's output as input and outputs the prediction result. During model inference, only the scorer is used. It is important to note that the dynamic pruning module stops backpropagation operations on the search graph tokens.

[0140] The memory token library includes a long-term memory token library and a short-term memory token library; wherein, the long-term memory token library stores memory tokens with longer temporal associations and is updated by a preset number of items before the memory token library score output in stage 4; the short-term memory token library stores memory tokens with shorter temporal associations and is updated by a preset number of items before the search graph token score output in stage 4.

[0141] The directional attention module consists of a multi-head attention module. The module inputs include long-term memory tokens, short-term memory tokens, template graph input tokens, and search graph input tokens. These four types of tokens share linear mapping weights. The long-term memory tokens are used to calculate the key K through a linear mapping. LT Value V LT Matrix; Short-Time Memory tokens are used to calculate key K through linear mapping. ST Value V ST Matrix; Template diagram input token is used to calculate query Q through linear mapping. T Key KT Value V T Matrix; Search graph input token is used to calculate query Q through linear mapping. S Key K S Value V S Matrix. Multi-head attention module input query Q T Key K T Value V T Matrix, output template diagram output token; key K LT K ST K T K S Key K is obtained by splicing. [LT,ST,T,S] Value V LT V ST V T V S The value V is obtained by splicing. [LT,ST,T,S] Multi-head attention module input query Q S Key K [LT,ST,T,S] Value V [LT,ST,T,S] The matrix outputs the search graph and the tokens.

[0142] In one embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used to execute the operation of a single-target tracking method based on dynamic pruning and timing adaptive aggregation.

[0143] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the single-target tracking method based on dynamic pruning and timing adaptive aggregation in the above embodiments.

[0144] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0145] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A single-target tracking method based on dynamic pruning and temporal adaptive aggregation, characterized in that, Includes the following steps: Obtain the target video sequence to be tracked, and annotate the bounding box of the target object in the first frame of the target video sequence as a template image; Based on the target video sequence and the template image, a pre-trained deep target tracking network is used to perform single-target tracking on the target objects in subsequent frames of the target video sequence to obtain single-target tracking results. The deep target tracking network includes: a memory token library for storing memory tokens, and a patch embedding and position encoding, stage 1, stage 2, stage 3, stage 4, and a convolutional network tracking head connected in sequence; wherein, the memory token library includes a long-term memory token library for storing long-term memory tokens and a short-term memory token library for storing short-term memory tokens; the patch embedding and position encoding is used to input template graphs and search graphs, and to tokenize and position-encode the template graphs and search graphs, outputting template graph tokens and search graph tokens; stage 1, stage 2, and stage 3 each include several directional attention encoder modules and dynamic... The dynamic pruning module includes several directional attention encoder modules connected in series. The directional attention encoder module takes into account long-term memory tokens, short-term memory tokens, and template graph tokens and search graph tokens output from the previous module, and outputs template graph tokens and search graph tokens after directional attention encoding. The dynamic pruning module takes into account search graph tokens output from the directional attention encoder modules in the same stage, and outputs high-quality search graph tokens after pruning. The memory token library takes into account tokens in the current memory token library and search graph tokens that meet preset quality requirements, and outputs long-term memory tokens and short-term memory tokens.

2. The single-target tracking method based on dynamic pruning and temporal adaptive aggregation according to claim 1, characterized in that, The directed attention encoder module includes a first LayerNorm normalization operation, a directed attention module, a second LayerNorm normalization operation, and a feedforward neural network layer connected in series. Additionally, a skip connection layer is used to add the input of the first LayerNorm normalization operation to the output of the directed attention module to obtain the input of the second LayerNorm normalization operation, and to add the input of the second LayerNorm normalization operation to the output of the feedforward neural network to obtain the output of the directed attention encoder module.

3. The single-target tracking method based on dynamic pruning and temporal adaptive aggregation according to claim 2, characterized in that, The directional attention module consists of a multi-head attention module. The input includes long-term memory tokens, short-term memory tokens, template graph input tokens, and search graph input tokens. The four types of tokens share linear mapping weights. Among them, the key K is calculated through a linear mapping of the long-term memory token. LT Value V LT Matrix; Short-Time Memory tokens are used to calculate key K through linear mapping. ST Value V ST Matrix; Template diagram input token is used to calculate query Q through linear mapping. T Key K T Value V T Matrix; Search graph input token is used to calculate query Q through linear mapping. S Key K S Value V S matrix; Multi-head attention module input query Q T Key K T Value V T Matrix, output template diagram output token; key K LT K ST K T K S Key K is obtained by splicing. [LT,ST,T,S] Value V LT V ST V T V S The value V is obtained by splicing. [LT,ST,T,S] Multi-head attention module input query Q S Key K [LT,ST,T,S] Value V [LT,ST,T,S] The matrix outputs the search graph and the tokens.

4. The single-target tracking method based on dynamic pruning and temporal adaptive aggregation according to claim 1, characterized in that, The dynamic pruning module includes a scorer, a cross-attention aggregator, and an auxiliary tracking head. The scorer takes a learnable query token matrix and search graph tokens as input, outputs an attention heatmap as a score matrix, and sums the score matrices along the query token dimension to obtain the search graph token score. A preset number of terms are retained before the search graph token score is calculated. During the training phase, the cross-attention aggregator takes the scorer's output and the search graph token values ​​as input, and the auxiliary tracking head takes the cross-attention aggregator's output as input to output the prediction result. During the inference phase, only the scorer is used. Furthermore, the dynamic pruning module stops backpropagation operations on the search graph tokens.

5. A single-target tracking method based on dynamic pruning and temporal adaptive aggregation according to claim 4, characterized in that, In the dynamic pruning module, during training, the prediction results are used to adjust the parameters in the learnable query token matrix Q, the scorer, the cross-attention aggregator, and the auxiliary tracking head using gradient descent. The scoring mechanism of the scorer is represented as follows: X K =TopK(X|Score); In the formula, Q is the learnable query matrix; K is the key, i.e., the search graph token; N is the number of query tokens in Q; Score is the score of the search graph tokens by the dynamic pruning module; X is the input search graph token; X K Reserve a predetermined number of tokens before the output score; (·) T This represents the matrix transpose; TopK(·) represents retaining a preset number of tokens before scoring; The cross-attention aggregator is represented as: X″=MLP(LN(X′))+X′; In the formula, V is the value, i.e., the search graph token; d is the scaling factor; X′ represents the output of the attention operation in the cross-attention aggregator; X″ represents the output of the cross-attention aggregator; Softmax(·) is the exponential normalization function; (·) T LN(·) represents matrix transpose; LN(·) represents linear mapping; MLP(·) represents multilayer perceptron.

6. The single-target tracking method based on dynamic pruning and temporal adaptive aggregation according to claim 5, characterized in that, During the training of the deep target tracking network, gradient propagation is performed simultaneously on the auxiliary tracking heads of the three-stage dynamic pruning modules and the convolutional network tracking head. The total loss function is expressed as: L=L cls +λ iou L iou +λ L1 L1; L auxi =L cls +λ iou L iou +λ L1 L1,i=1,2,3; In the formula, L cls L iou L1 and L1 represent the classification loss, intersection-over-union loss, and absolute value loss, respectively; λ iou and λ L1 These are the weight parameters for the crossover ratio loss and the absolute value loss, respectively; L and Let α represent the loss functions of the convolutional network tracking head and the i-th auxiliary tracking head, respectively; i The weighting parameter of the total loss function for the loss value of the i-th auxiliary tracking head; Loss represents the total loss function of a deep object tracking network.

7. The single-target tracking method based on dynamic pruning and temporal adaptive aggregation according to claim 1, characterized in that, In the memory token library, the long-term memory tokens stored in the long-term memory token library are memory tokens with longer time-series associations, and are updated by the preset number of items before the memory token library score output in stage 4. The short-term memory tokens stored in the short-term memory token library are short-term memory tokens with short temporal associations, and are updated by the preset number of items before the token scoring of the search graph output in stage 4; in, The scoring mechanism is expressed as follows: K [LT,ST,T,S] =Concat(K LT， K ST， K T， K S ); attn=Q S (K [LT,ST,T,S] ) T ; In the formula, K LT K ST K T K S These are the keys output through a linear mapping of long-term memory tokens, short-term memory tokens, template graph tokens, and search graph tokens, respectively; K [ST,LT,T,S] A matrix connecting the keys of four types of tokens; Q S `attn` represents the query output by the search graph tokens through a linear mapping; `attn` represents the attention heatmap recorded in the last directional attention encoder module of stage 4; `C` is the classification score map output by the convolutional network tracking head; `M` is the number of remaining search graph tokens after three prunings; `Score` is the score result for long-term memory tokens, short-term memory tokens, template graph tokens, and remaining search graph tokens after three prunings; `Concat(·)` represents matrix concatenation; `(·)` T Indicates matrix transpose; The update scheme for the long-term memory token library can be represented as follows: Score LT =Score(Index LT ); Score ST =Score(Index ST ); Score [LT,ST] =Concat(Score LT ,Score ST ); X [LT,ST] =Concat(X LT， X ST ) X′ LT =TopK(X [LT,ST] |Score [LT,ST] ); In the formula, Index LT and Index ST These are the indices in which long-term memory tokens and short-term memory tokens participate in the calculation of the Score; Score LT and Score ST The score represents the rating of long-term and short-term tokens. [LT,ST] The connection matrix for long-term and short-term token scores; X LT and X ST These are the long-term memory token and the short-term memory token before the update; X [LT,ST] This is the connection matrix of long-term memory tokens and short-term memory tokens before the update; X′ LT For the updated long-term memory tokens; TopK(·) indicates that a preset number of tokens should be retained before the rating; The update scheme for the short-time memory token library can be represented as: Score S =Score(Index S ); X′ S =X S +Embeding(Round(C)); X′ ST =TopK(X′ S |Score S ); In the formula, Index S This indicates that only the indexes in the Score search graph tokens are used in the calculation. S This represents the score of the remaining search graph tokens after three prunings; X S The output of stage 4 represents the search graph token; Round(·) represents the rounding function; Embedding(·) represents the embedding layer, which consists of a single linear layer; X′ S X′ represents the search graph feature token generated by adding the search graph token output from stage 4 to the embedding layer output. ST For the updated short-term memory token.

8. A single-target tracking system based on dynamic pruning and temporal adaptive aggregation, characterized in that, include: The data acquisition module is used to acquire the target video sequence to be tracked and to annotate the bounding box of the target object in the first frame of the target video sequence as a template image. The tracking processing module is used to perform single-target tracking on target objects in subsequent frames of the target video sequence based on the target video sequence and the template image, using a pre-trained deep target tracking network, to obtain single-target tracking results. The deep target tracking network includes: a memory token library for storing memory tokens, and a patch embedding and position encoding, stage 1, stage 2, stage 3, stage 4, and a convolutional network tracking head connected in sequence; wherein, the memory token library includes a long-term memory token library for storing long-term memory tokens and a short-term memory token library for storing short-term memory tokens; the patch embedding and position encoding is used to input template graphs and search graphs, and to tokenize and position-encode the template graphs and search graphs, outputting template graph tokens and search graph tokens; stage 1, stage 2, and stage 3 each include several directional attention encoder modules and dynamic... The dynamic pruning module includes several directional attention encoder modules connected in series. The directional attention encoder module takes into account long-term memory tokens, short-term memory tokens, and template graph tokens and search graph tokens output from the previous module, and outputs template graph tokens and search graph tokens after directional attention encoding. The dynamic pruning module takes into account search graph tokens output from the directional attention encoder modules in the same stage, and outputs high-quality search graph tokens after pruning. The memory token library takes into account tokens in the current memory token library and search graph tokens that meet preset quality requirements, and outputs long-term memory tokens and short-term memory tokens.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the single-target tracking method based on dynamic pruning and temporal adaptive aggregation as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the single-target tracking method based on dynamic pruning and temporal adaptive aggregation as described in any one of claims 1 to 7.

Images