A target tracking method and system based on trajectory awareness

Abstract

This invention discloses a target tracking method and system based on trajectory awareness, belonging to the field of computer vision technology. The method includes: dynamically fusing historical frame features by constructing a PatchBlender hybrid matrix generator, and combining it with a two-dimensional video feature extraction network composed of a PatchBlender smoothing layer and a ViT Transformer to obtain a preliminary historical representation of the target motion; further introducing a three-dimensional spatiotemporal graph attention mechanism to model the spatiotemporal relationship of the historical features, updating to obtain more discriminative trajectory-aware features; finally, fusing these features with search frame features, and achieving high-precision target localization and bounding box regression through a prediction head network. This invention achieves efficient modeling and accurate tracking of target motion trajectories through the PatchBlender hybrid matrix generator, the PatchBlender Transformer feature extraction network, and the three-dimensional spatiotemporal graph attention mechanism.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a target tracking method and system based on trajectory perception. Background Technology

[0002] Object tracking is a crucial research task in computer vision, widely applied in scenarios such as video surveillance, intelligent transportation, autonomous driving, and human-computer interaction. Traditional object tracking methods are mostly based on a "template-search region" matching mechanism, determining the target's location by finding the region in the current frame that is most similar to the initial template. In simpler terms, it's about "finding the region most similar to the target within a frame." While these methods have low computational cost, they struggle to fully utilize the target's historical information and lack modeling of the target's trajectory over time. Therefore, they are prone to tracking drift and failure in long-term tracking or complex environments.

[0003] With the development of target detection technology, Tracking-by-Detection (TBD) has gradually become mainstream. This method mainly consists of two parts: a target detector and a tracker. First, the detector acquires the positions of candidate targets in each frame. Then, a data association algorithm matches targets from different frames to existing trajectories, achieving continuous tracking. Leveraging the high precision of modern detectors, TBD can maintain high detection rates and tracking robustness in complex scenarios such as changes in target appearance, lighting conditions, or occlusion. However, TBD still lacks explicit modeling of long-term target motion, and trajectory breakage or target identity switching may still occur in scenarios with continuous target occlusion or rapid movement.

[0004] In recent years, with the development of deep learning, Transformer-based video target tracking methods have received widespread attention. These methods can handle video-level input and utilize self-attention mechanisms to model inter-frame temporal correlations, thereby enhancing their adaptability to target motion and appearance changes. However, the standard Transformer architecture primarily relies on self-attention mechanisms to globally model the features of the entire image or video frame, lacking explicit temporal induction biases. This results in limitations in feature fusion and dynamic change capture across consecutive video frames, making it prone to target drift or loss during fast-moving, occluded, or long-term tracking. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a target tracking method and system based on trajectory awareness. By introducing temporal feature enhancement based on PatchBlender Transformer and a three-dimensional spatiotemporal graph attention mechanism, and fusing historical motion trajectory information of the target, high-precision and robust end-to-end target tracking is achieved.

[0006] On the one hand, a target tracking method based on trajectory awareness includes:

[0007] S1. Obtain the original video stream, extract video frames from the original video stream and preprocess them to obtain the original video embedding sequence, and input the original video embedding sequence into the two-dimensional video feature extraction network based on PatchBlender Transformer to obtain the preliminary feature representation of the target motion trajectory.

[0008] S2, extract the preset frame embedding from the preliminary feature representation of the target motion trajectory to construct a three-dimensional spatiotemporal graph structure, update the embedding along the three-dimensional spatiotemporal graph structure through the graph attention mechanism, and obtain the reference feature representation that integrates the target trajectory motion information;

[0009] S3, the reference feature representation and the search frame features extracted from the preliminary feature representation of the target motion trajectory are concatenated and input into the prediction head network based on the convolutional network for regression prediction to obtain target tracking information. The target tracking information includes the horizontal coordinate of the target box center, the vertical coordinate of the target box center, the target box width, and the target box height on the original image.

[0010] Furthermore, in S1, the two-dimensional video feature extraction network based on PatchBlender Transformer is specifically composed of several PatchBlender Transformer modules stacked together. Each PatchBlender Transformer module has the same structure but does not share parameters; each PatchBlender Transformer module includes a PatchBlender smoothing layer and a ViT Transformer layer.

[0011] PatchBlender smoothing layers accept dimensions of: The original video embedding sequence input Through a hybrid matrix generator The generated mixing matrix Based on hybrid matrix Temporal fusion of features from the original video frame sequence yields the enhanced input. Then, the input is fed into the ViT Transformer layer for multi-head self-attention feature extraction; after processing through multiple PatchBlender Transformer modules, a preliminary feature representation of the target motion trajectory is finally obtained; among which, The length of the extracted frame sequence, The number of patches per frame. For the embedded dimension.

[0012] Furthermore, the hybrid matrix generator The specific construction method is as follows:

[0013] Features of each input frame patch Average pooling is used to obtain the frame feature vector. :

[0014] ;

[0015] in, Indicates taking the first position The first frame The feature embedding vectors of each patch are concatenated in chronological order to form a frame feature matrix:

[0016] ;

[0017] A hybrid matrix generator is constructed using attention based on the frame feature matrix:

[0018] ;

[0019] in, , and It is a learnable matrix; This indicates element-wise multiplication within the matrix. This indicates that Softmax normalization is applied to the matrix; D represents the embedding dimension.

[0020] Furthermore, a mixing matrix is ​​obtained through a mixing matrix generator. Then, the PatchBlender smoothing layer applies the original video embedding sequence as input. The mixing process is as follows:

[0021] ;

[0022] The final shape is Enhanced input after initial integration of information from various time periods The data is then fed into a Vit Transformer layer for multi-head self-attention to further extract features, resulting in a preliminary feature representation of the target's motion trajectory. Its shape is ; Indicates the first Frame to the first The degree of impact of the frame; This indicates that the video number is being retrieved. Input characteristics of frames.

[0023] Furthermore, in S2, the embedding is updated along the 3D spatiotemporal graph structure through a graph attention mechanism to obtain a reference feature representation that fuses the target trajectory motion information, as follows:

[0024] First, from the preliminary characteristic representation of the target's motion trajectory Extract the nearest All patch embeddings of a frame form a set of nodes in a graph. The calculation formula is as follows:

[0025] ;

[0026] Among them, node features are defined as , indicating the time taken The The embedding vectors of patch; the total number of nodes is ; Indicates the current moment. Indicates the sampling length of the preset frame;

[0027] Describe the relationships between nodes Define each node neighborhood From spatial neighborhood and temporal neighborhood Together, they form the edge set used to construct the graph. The definition is as follows:

[0028] ;

[0029] Spatial neighborhood represents locally adjacent patches within the same frame, as follows:

[0030] ;

[0031] in Indicates section Local neighborhood in a two-dimensional patch mesh; Indicates the patch index of the neighboring node; Indicates at time The The embedding vector corresponding to each patch;

[0032] Temporal neighborhood represents the corresponding patch at the same position in adjacent frames, as follows:

[0033] ;

[0034] in, Indicates the time to which the embedding vector belongs; Indicates the first Index of each patch;

[0035] Edge set based on neighborhood definition graph The construction method is as follows:

[0036] ;

[0037] For each node in the graph, perform graph attention updates along the edges of the graph; for each edge... Calculate attention weights :

[0038] ;

[0039] in, It is a linear mapping matrix; This is the attention weight vector; Indicates feature splicing; Representing neighboring nodes The corresponding time index; Indicates the activation function;

[0040] node The update characteristics are given by the weighted sum of its neighboring nodes:

[0041] ;

[0042] For the recent After all nodes in the frame have been updated as described above, they are rearranged in chronological order to obtain a reference feature representation that fuses the target trajectory motion information. ,as follows:

[0043] .

[0044] Finally, preliminary characteristic representation of the target's motion trajectory. Extract search frame features , and After connection, the data is fed into the prediction head network for target bounding box prediction.

[0045] Furthermore, in S3, the prediction head network includes three structurally identical sub-convolutional networks (FCNs), each of which is... The system consists of stacked Conv-BN-ReLU layers, with independent sub-convolutional networks that do not share weights. Three structurally identical sub-convolutional networks (FCNs) are used to obtain the target classification score map. Local offset and normalized bounding box size ; This indicates the height of the image after scaling in the preprocessing stage. This indicates the height of the image after scaling during the preprocessing phase.

[0046] Furthermore, the formula for calculating target tracking information is as follows:

[0047]

[0048] ;

[0049] ;

[0050] in, , represents the coordinate where the maximum value is obtained in the classification score response map. The normalized x-coordinate of the center of the target bounding box. The normalized ordinate of the center of the target bounding box. To normalize the bounding box width, This is the normalized bounding box height;

[0051] , , and These are the normalized x-coordinate offset of the target box center, the normalized y-coordinate of the target box center, the normalized width of the target box, and the normalized height of the target box, respectively.

[0052] , , and These are the x-coordinate of the center of the target bounding box, the y-coordinate of the center of the target bounding box, the width of the target bounding box, and the height of the target bounding box on the original image, respectively.

[0053] The width of the search area in the cropping; The height of the search area to be clipped; The x-coordinate of the top left corner of the cropped area; The vertical coordinate of the top left corner of the cropped area.

[0054] On the other hand, a trajectory-aware target tracking system includes:

[0055] The preliminary feature representation extraction module is used to acquire the original video stream, extract video frames from the original video stream and preprocess them to obtain the original video embedding sequence. The original video embedding sequence is then input into a two-dimensional video feature extraction network based on PatchBlender Transformer to obtain the preliminary feature representation of the target motion trajectory.

[0056] The updated embedding module is used to extract preset frame embeddings from the preliminary feature representation of the target motion trajectory to construct a three-dimensional spatiotemporal graph structure. The embeddings are updated along the three-dimensional spatiotemporal graph structure through a graph attention mechanism to obtain a reference feature representation that integrates the target trajectory motion information.

[0057] The tracking module is used to connect the reference feature representation and the search frame features extracted from the preliminary feature representation of the target motion trajectory and input them into the prediction head network based on the convolutional network for regression prediction to obtain target tracking information. The target tracking information includes the horizontal coordinate of the target box center, the vertical coordinate of the target box center, the target box width, and the target box height on the original image.

[0058] The present invention adopts the above technical solution and has the following beneficial effects:

[0059] (1) This invention constructs a three-dimensional spatiotemporal graph structure that fuses the q-th closest frames and introduces a graph attention mechanism to model the target's motion trajectory, effectively capturing the dynamic evolution of the target in the spatiotemporal dimension and enhancing its resistance to interference factors such as occlusion, deformation, and rapid movement;

[0060] (2) The present invention uses PatchBlender Transformer to perform time-aware feature smoothing and enhancement on the original video embedding sequence, and combines it with subsequent graph attention update to obtain reference feature representation rich in trajectory context information, so that the prediction head can more accurately regress the target position and scale, and improve the bounding box localization accuracy;

[0061] (3) This invention organically combines trajectory feature extraction based on PatchBlender Transformer, three-dimensional spatiotemporal graph attention modeling and lightweight convolutional prediction head to form a unified end-to-end framework. Without significantly increasing computational overhead, it realizes the perception and accurate tracking of the target's long-term motion pattern, and is suitable for real-time visual tracking tasks. Attached Figure Description

[0062] Figure 1 This is a flowchart of the target tracking method based on trajectory perception according to an embodiment of the present invention;

[0063] Figure 2 This is a diagram of a target tracking system based on trajectory awareness, according to an embodiment of the present invention. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0065] like Figure 1 As shown, the present invention provides a target tracking method based on trajectory awareness, comprising:

[0066] S1. Obtain the original video stream, extract video frames from the original video stream and preprocess them to obtain the original video embedding sequence, and input the original video embedding sequence into the two-dimensional video feature extraction network based on PatchBlender Transformer to obtain the preliminary feature representation of the target motion trajectory.

[0067] Specifically, a video frame image can be viewed as having the shape of The matrix, where Represents the pixel height of the image. Indicates the pixel width of the image. This indicates the number of channels in the image. The initial target region is automatically provided by the detector or manually specified in the template frame.

[0068] Extract the first frame of the video as the template frame, and extract the nearest frame from the first frame to the current frame. Frame images are used as historical frames to obtain trajectory information (if insufficient). If a frame is selected, all frames are retrieved; the current frame image is used as the search frame.

[0069] For the template frame image, it is strictly cropped according to the initial target region, retaining only the pixels within the target region as subsequent encoding input. For historical frame images and search frame images, a fixed-size search region is cropped from the original image using the center of the target region as a reference point; the size of this search region remains constant throughout the tracking process. After cropping, the images are uniformly scaled to [size missing]. It is divided into non-overlapping two-dimensional image patches, each patch having a spatial size of [missing information]. The total number of patches after partitioning is

[0070] ;

[0071] The shape of each patch is The image shape becomes .

[0072] Then, each patch is flattened into its length according to channel order. The vector is encoded using a linear mapping as The embedding vector is 3D, and a one-dimensional learnable positional encoding is added. :

[0073]

[0074] in and For trainable parameters, Indicates the first The flattened vector of each patch. It is a learnable one-dimensional positional encoding.

[0075] After each frame of the image is divided and encoded using the patch method described above, they are concatenated in chronological order to obtain a shape of... Embedding matrix The data is then input into the PatchBlender Transformer backbone network, where temporal information fusion and multi-head self-attention are used to obtain a preliminary feature representation of the target motion trajectory. .in, This indicates the total number of frames extracted. This represents the number of patches that each video frame is divided into after processing. Indicates the embedding dimension.

[0076] Specifically, the 2D video feature extraction network based on PatchBlender Transformer is composed of several PatchBlender Transformer modules stacked together. Each PatchBlender Transformer module has the same structure but does not share parameters. Each PatchBlender Transformer module includes a PatchBlender smoothing layer and a ViTTransformer layer.

[0077] PatchBlender smoothing layers accept dimensions of: The original video embedding sequence input Through a hybrid matrix generator The generated mixing matrix Based on hybrid matrix Temporal fusion of features from the original video frame sequence yields the enhanced input. Then, the input is fed into the ViT Transformer layer for multi-head self-attention feature extraction; after processing through multiple PatchBlender Transformer modules, a preliminary feature representation of the target motion trajectory is finally obtained; among which, The length of the extracted frame sequence, The number of patches per frame. For the embedded dimension.

[0078] The hybrid matrix generator The specific construction method is as follows:

[0079] Features of each input frame patch Average pooling is used to obtain the frame feature vector. :

[0080] ;

[0081] in, Indicates extraction at the 1st Frame, number The feature embedding vectors of each patch are concatenated in chronological order to form a frame feature matrix:

[0082] ;

[0083] A hybrid matrix generator is constructed using attention based on the frame feature matrix:

[0084]

[0085] in, , and It is a learnable matrix; represents element-wise multiplication within a matrix; Softmax represents applying Softmax normalization to the matrix; D represents the embedding dimension.

[0086] Specifically, the mixture matrix is ​​obtained through the mixture matrix generator. Then, the PatchBlender smoothing layer applies the original video embedding sequence as input. The mixing process is as follows:

[0087] ;

[0088] The final shape is Enhanced input after initial integration of information from various time periods The data is then fed into a Vit Transformer layer for multi-head self-attention to further extract features, resulting in a preliminary feature representation of the target's motion trajectory. Its shape is ; Indicates the first Frame to the first The degree of impact of the frame; Indicates the extraction of video number 1 Input characteristics of frames.

[0089] S2, extract the most recent q frames from the preliminary feature representation of the target trajectory and embed them to construct a three-dimensional spatiotemporal graph structure. Update the embedding along the three-dimensional spatiotemporal graph structure through a graph attention mechanism to obtain a reference feature representation that integrates the target trajectory motion information.

[0090] Specifically, the embedding is updated along the 3D spatiotemporal graph structure using a graph attention mechanism to obtain a reference feature representation that fuses the target trajectory motion information, as follows:

[0091] First, from the preliminary characteristic representation of the target's motion trajectory Extract the nearest All patch embeddings of a frame form a set of nodes in a graph. The calculation formula is as follows:

[0092] ;

[0093] Among them, node features are defined as ; indicates at time No. The embedding vectors of patch; the total number of nodes is ; Indicates the current time; q represents the sampling length of the preset frame;

[0094] To describe the relationships between nodes Define each node neighborhood From spatial neighborhood and time neighborhood Together they form the following:

[0095] ;

[0096] Spatial neighborhood represents locally adjacent patches within the same frame, as follows:

[0097] ;

[0098] in Indicates section Local neighborhood in a 2D patch grid; Indicates the patch index of the neighboring node; Indicates at time The The embedding vector corresponding to each patch;

[0099] Temporal neighborhood represents the corresponding patch at the same position in adjacent frames:

[0100] ;

[0101] in, Indicates the time to which the embedding vector belongs; This represents the index of the nth patch;

[0102] The edge set based on the aforementioned neighborhood definition graph for:

[0103] ;

[0104] For each node in the graph, perform graph attention updates along the edges of the graph; for each edge... Calculate attention weights :

[0105] ;

[0106] in, It is a linear mapping matrix; This is the attention weight vector; Indicates feature splicing; Representing neighboring nodes The corresponding time index; Indicates the activation function;

[0107] node The update characteristics are given by the weighted sum of its neighboring nodes:

[0108] ;

[0109] For the recent After all nodes in the frame have been updated as described above, they are rearranged in chronological order to obtain a reference feature representation that fuses the target trajectory motion information. ,as follows:

[0110] .

[0111] Finally, preliminary characteristic representation of the target's motion trajectory. Extract search frame features , and After connection, the data is fed into the prediction head network for target bounding box prediction.

[0112] S3, the reference feature representation and the search frame features extracted from the preliminary feature representation of the target motion trajectory are concatenated and input into the prediction head network based on the convolutional network for regression prediction to obtain target tracking information. The target tracking information includes the horizontal coordinate of the target box center, the vertical coordinate of the target box center, the target box width, and the target box height on the original image.

[0113] Specifically, in this embodiment, the prediction head network consists of three structurally identical sub-convolutional networks (FCNs), each network structure being composed of... The system consists of stacked Conv-BN-ReLU layers. The sub-convolutional networks are independent and do not share weights. They are used separately to obtain the target classification score image. Local offset Used to correct discretization errors and normalize bounding box sizes. The prediction head network receives a shape of The result of connecting the reference frame and the search frame. Each frame... Each patch embedding is arranged and reorganized into a two-dimensional grid according to the original two-dimensional space, resulting in a feature map of each frame with the following shape: The specific process is as follows:

[0114] For the Frame, take its Embedded patch ;

[0115] Based on the position of the patch in the two-dimensional image grid, Rearranged to the corresponding spatial position of the two-dimensional feature map ;

[0116] The rearranged features of all frames are fused in chronological order to obtain the final shape. Spatial feature map , which serves as the input to three independent sub-convolutional networks (FCNs).

[0117] Subsequently, the three sub-convolutional networks generate object classification score maps respectively. Local offset and normalized bounding box size ; This indicates the height of the image after scaling in the preprocessing stage. This represents the height of the image after scaling in the preprocessing section. The target center position is determined by the highest score position. The final normalized bounding box is obtained by combining local offset and normalized size.

[0118] Specifically, the formula for calculating target tracking information is as follows:

[0119]

[0120] ;

[0121] ;

[0122] in, The normalized x-coordinate of the center of the target bounding box. The normalized ordinate of the center of the target bounding box. To normalize the bounding box width, This is the normalized bounding box height;

[0123] These are the normalized x-coordinate offset of the target box center, the normalized y-coordinate of the target box center, the normalized width of the target box, and the normalized height of the target box, respectively.

[0124] , , and These are the x-coordinate of the center of the target bounding box, the y-coordinate of the center of the target bounding box, the width of the target bounding box, and the height of the target bounding box on the original image, respectively.

[0125] The width of the search area in the cropping; The height of the search area to be clipped; The x-coordinate of the top left corner of the cropped area; The vertical coordinate of the top left corner of the cropped area.

[0126] This embodiment also includes a model training step, where the network is trained under supervision on video sequence samples in the training set. Specifically, this includes: first, processing historical frames and search frames to obtain the classification response map and bounding box predictions output by the prediction head; then, comparing the classification response map with the true target location to calculate the classification loss, which uses focal loss; and finally, calculating the regression error between the bounding box predictions and the true bounding boxes, including L1 loss and GIoU loss based on overlap. The resulting total loss is as follows:

[0127]

[0128] in, For classifying losses, For regression loss It is a regularization parameter used for balancing The weights among the three.

[0129] Finally, the loss of each frame is averaged over the sequence length dimension to obtain the overall training loss. The parameters of each module in the network are optimized through backpropagation, so that the network can accurately predict the target position and maintain trajectory consistency in new video sequences.

[0130] In summary, this invention effectively alleviates the problem of insufficient feature fusion between different frames in long-term video modeling by introducing a PatchBlender smoothing layer into the visual frequency-level Transformer, making the feature representation smoother and more robust, thereby improving the adaptability to fast target motion.

[0131] By constructing a 3D spatiotemporal graph structure and combining it with a graph attention network to weighted update node features, explicit modeling of the target's motion trajectory in the temporal dimension is achieved. This fully utilizes historical information and maintains tracking continuity and stability under scenarios involving changes in target appearance, rapid target movement, long-term tracking, or occlusion. The graph attention mechanism dynamically calculates the correlation between nodes, adaptively allocating weights based on the importance of neighborhood features, avoiding unnecessary computation and improving the model's resistance to complex background interference and target identity switching. Employing a lightweight modular design, it inherits the global modeling capabilities of the Transformer while combining the spatiotemporal structural advantages of graph neural networks, reducing computational complexity while maintaining accuracy. This meets the needs of real-time video target tracking and is suitable for fields such as video surveillance and intelligent security.

[0132] As shown in Table 1, this model (TATrack) was trained on a server with two 24GB Nvidia GeForce RTX3090 GPUs. LaSOT, TrackingNet, and COCO were used as the complete training sets, while only the training set for GOT-10k was used. Data augmentation was performed using horizontal flipping and brightness jitter. The batch size was set to 16. Input video frames were cropped and uniformly scaled to 256×256 pixels. AdamW was used to optimize the network parameters, with an initial learning rate of... The weight decay is set to The total number of training epochs was set to 300, with 60,000 image pairs randomly sampled in each epoch for training. After 240 epochs, the learning rate was set to decrease to a certain value. After training, the model's performance was compared with that of state-of-the-art trackers on three large-scale tracking benchmarks:

[0133] Table 1 compares the results with advanced trackers on three large-scale tracking benchmarks;

[0134]

[0135] like Figure 2 As shown, this embodiment also discloses a target tracking system based on trajectory awareness, including:

[0136] The preliminary feature representation extraction module 21 is used to acquire the original video stream, extract video frames from the original video stream and preprocess them to obtain the original video embedding sequence, and input the original video embedding sequence into the two-dimensional video feature extraction network based on PatchBlenderTransformer to obtain the preliminary feature representation of the target motion trajectory.

[0137] The embedding module 22 is updated to extract the preset frame embedding from the preliminary feature representation of the target motion trajectory in order to construct a three-dimensional spatiotemporal graph structure. The embedding is updated along the three-dimensional spatiotemporal graph structure through the graph attention mechanism to obtain a reference feature representation that integrates the target trajectory motion information.

[0138] The tracking module 23 is used to connect the reference feature representation and the search frame features extracted from the preliminary feature representation of the target motion trajectory and input them into the prediction head network based on the convolutional network for regression prediction to obtain target tracking information. The target tracking information includes the horizontal coordinate of the target box center, the vertical coordinate of the target box center, the target box width, and the target box height on the original image.

[0139] A specific implementation of a trajectory-aware target tracking system is described in this embodiment, which is the same as the trajectory-aware target tracking method.

[0140] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A trajectory-aware based target tracking method, characterized in that, The method comprises the following steps: S1, obtaining an original video stream, extracting video frames from the original video stream, and obtaining an original video embedding sequence after preprocessing, and inputting the original video embedding sequence into a two-dimensional video feature extraction network based on a PatchBlender Transformer to obtain a preliminary feature representation of a target motion trajectory; The two-dimensional video feature extraction network based on the PatchBlender Transformer is specifically stacked by a plurality of PatchBlender Transformer modules, each of which has the same structure but does not share parameters; each PatchBlender Transformer module comprises a PatchBlender smoothing layer and a ViTTransformer layer; PatchBlender smoothing layers accept dimensions of: The original video embedding sequence input Through a hybrid matrix generator The generated mixing matrix Based on hybrid matrix Temporal fusion of features from the original video frame sequence yields the enhanced input. Then, the input is fed into the ViT Transformer layer for multi-head self-attention feature extraction; after processing through multiple PatchBlender Transformer modules, a preliminary feature representation of the target motion trajectory is finally obtained; among which, The length of the extracted frame sequence, The number of patches per frame. For the embedded dimension; S2, extracting a preset frame embedding from the preliminary feature representation of the target motion trajectory to construct a three-dimensional spatio-temporal graph structure, updating the embedding along the three-dimensional spatio-temporal graph structure through a graph attention mechanism, and obtaining a reference feature representation fused with target trajectory motion information; S3, connecting the reference feature representation and a search frame feature extracted from the preliminary feature representation of the target motion trajectory and inputting them into a prediction head network based on a convolutional network for regression prediction to obtain target tracking information, wherein the target tracking information comprises a target box center horizontal coordinate, a target box center vertical coordinate, a target box width and a target box height on an original image. 2.The trajectory-aware based target tracking method according to claim 1, characterized in that, The hybrid matrix generator The configuration mode is specifically: for each frame of input patch features average pooling to obtain a frame feature vector : ； wherein, represents the feature embedding vector of the patch at the frame at the patch; concatenated in time order as a frame feature matrix: ； An attention-based hybrid matrix generator is constructed based on the frame feature matrix: ； wherein, , and are learnable matrices; denotes multiplication of corresponding elements within matrices, denotes applying Softmax for normalization on matrices; denotes embedding dimension. 3.The trajectory-aware based target tracking method according to claim 2, characterized in that, The mixing matrix is obtained by a mixing matrix generator After, the PatchBlender smooth layer inputs the original video embedding sequence The process of mixing is as follows: ； The shape of the final result is The enhanced input of each time information is fused preliminarily , and sent into the VitTransformer layer to further extract features by multi-head self-attention, to obtain a preliminary feature representation of the target motion trajectory , which has a shape of ; Indicates the influence degree of the first frame on the first frame; Indicates the input feature of the first frame of the video.

4. The trajectory-aware based target tracking method according to claim 3, wherein, In S2, the embedding is updated along the three-dimensional spatio-temporal graph structure through a graph attention mechanism to obtain a reference feature representation fused with target trajectory motion information, which is specifically as follows: First, all patch embeddings of the most recent frame are extracted from the preliminary feature representation of the target motion trajectory forming a set of nodes of a graph The formula is as follows:​ ； Wherein, the node feature is defined as , which represents the embedding vector of the th patch at time ; the total number of nodes is ; represents the current time, represents the sampling length of the preset frame; Describing the association between nodes , defining the neighborhood of each node is composed of a spatial neighborhood and a temporal neighborhood , used to build the edge set of the graph, defined as follows:​ ； The spatial neighborhood represents local adjacent patches in the same frame, as follows: ； wherein representing a segment a local neighborhood in a two-dimensional patch grid; representing patch indices of neighboring nodes; representing an embedding vector corresponding to the th patch at time th patch at time The temporal neighborhood represents corresponding patches at the same position in adjacent frames, as follows: ； wherein, represents the time instant to which the embedding vector belongs; represents the index of the th patch; Construction of the edge set based on the neighborhood definition graph is as follows: ； For a node in the graph, perform a graph attention update along edges of the graph; for each edge Computing attention weights : ； wherein, is a linear mapping matrix; is an attention weight vector; denotes feature concatenation; denotes neighbor nodes the time index of the belonging; denotes an activation function; Node The update feature of a node is given by the weighted sum of its neighborhood nodes: ； To the most recent After performing the above update to all nodes of the frame, it is rearranged in chronological order to obtain a reference feature representation that fuses the target trajectory motion information As follows: ； Finally, the search frame features are extracted from the preliminary feature representation of the target motion trajectory , which are concatenated with the prediction head network for target bounding box prediction.​ 5.The trajectory-aware based target tracking method according to claim 1, wherein, In S3, the prediction head network comprises three sub-convolutional networks FCN with the same structure, each of which is The layer Conv-BN-ReLU is stacked, the sub-convolutional networks are independent of each other and do not share weights, and the three sub-convolutional networks FCN with the same structure are respectively used to obtain a target classification score map , local offset , and normalized bounding box size ; represents the height of the image after scaling in the preprocessing part. represents the width of the image after scaling in the preprocessing part. 6.The trajectory-aware target tracking method of claim 1, wherein, In S3, the calculation formula of the target tracking information is as follows: ； ； ； wherein, denotes the coordinates at which the classification score response graph takes a maximum value; is a normalized horizontal coordinate of the center of the target bounding box, is a normalized vertical coordinate of the center of the target bounding box, is a normalized width of the bounding box, is a normalized height of the bounding box; , , and are a normalized horizontal coordinate offset of the target box center, a normalized vertical coordinate of the target box center, a normalized width of the target box, and a normalized height of the target box, respectively. , , and are the horizontal coordinate of the center of the target frame, the vertical coordinate of the center of the target frame, the width of the target frame and the height of the target frame on the original image respectively; is a cropped search region width; is a cropped search region height; is a cropped top-left corner horizontal coordinate; is a cropped top-left corner vertical coordinate.

7. A trajectory-aware based target tracking system, characterized in that, The method comprises the following steps: The preliminary feature representation extraction module is configured to obtain an original video stream, extract video frames from the original video stream, and obtain an original video embedding sequence after preprocessing, and input the original video embedding sequence into a two-dimensional video feature extraction network based on a PatchBlender Transformer to obtain a preliminary feature representation of a target motion trajectory; The two-dimensional video feature extraction network based on the PatchBlender Transformer is specifically stacked by a plurality of PatchBlender Transformer modules, each of which has the same structure but does not share parameters; each PatchBlender Transformer module comprises a PatchBlender smoothing layer and a ViTTransformer layer; PatchBlender smooth layer accepts input of original video embedding sequence with dimension ​​​​​​​​​ The update embedding module is configured to extract a preset frame embedding from the preliminary feature representation of the target motion trajectory to construct a three-dimensional spatio-temporal graph structure, update the embedding along the three-dimensional spatio-temporal graph structure through a graph attention mechanism, and obtain a reference feature representation fused with target trajectory motion information; The tracking module is configured to connect the reference feature representation and the search frame features extracted from the preliminary feature representation of the target motion track and input the prediction head network based on the convolution network to perform regression prediction, so as to obtain target tracking information, wherein the target tracking information includes a horizontal coordinate of a center of a target frame on an original image, a vertical coordinate of the center of the target frame, a width of the target frame and a height of the target frame.

Images