Target tracking method based on time sequence information enhancement
By using the M-STEP model to fuse features and enhance information between video frames and template frames, the robustness and accuracy issues of existing target tracking methods in complex scenarios are solved, and efficient target tracking results are achieved.
Patent Information
- Application Number
- CN202511110978.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-08
- Publication Date
- 2025-11-18
AI Technical Summary
Existing target tracking methods lack robustness and generalization ability when faced with challenges such as target occlusion, rapid movement, pose changes and complex backgrounds, making it difficult to achieve a balance between speed, accuracy and robustness in practical deployments.
The M-STEP model is used to preprocess video search frames and multiple template frames. Feature extraction and fusion are performed through SFE, MSTE, SIF and TPE modules. Information enhancement is performed using multi-head self-attention and MLP layers. Finally, target localization is performed in the tracking and localization head.
It improves the accuracy and robustness of target tracking, effectively adapts to complex scenarios, enables continuous target localization and tracking, and enhances the model's anti-interference ability.
Smart Images
Figure CN120976576A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image target tracking, specifically relating to a target tracking method based on temporal information enhancement. Background Technology
[0002] Object tracking is one of the fundamental tasks in computer vision. Its goal is to accurately locate a target in a video sequence, given its initial position in the first frame, and continue to locate that target in subsequent frames. Depending on the task requirements, object tracking can be divided into single-object tracking (SOT) and multi-object tracking (MOT). SOT, as the most basic form, has been widely applied in fields such as intelligent surveillance, human-computer interaction, autonomous driving, augmented reality, and robot perception, demonstrating significant theoretical value and promising application prospects.
[0003] Early tracking methods primarily relied on hand-designed features (such as HOG, color histograms, and optical flow information) and traditional discriminative models (such as support vector machines and correlation filters) to model targets. These methods exhibited certain tracking performance in ideal environments, but they were prone to drift or tracking failure when faced with challenges such as target occlusion, rapid motion, pose changes, scale transformations, and complex backgrounds, lacking sufficient robustness and generalization ability.
[0004] In recent years, with the rapid development of deep learning, especially convolutional neural networks (CNNs), target tracking technology has made significant progress. The advantages of deep features in terms of representational power and discriminative ability have greatly improved the performance of trackers in complex scenes. Among them, Siamese network structures, represented by SiamFC, model the tracking task as a template matching problem, achieving end-to-end feature similarity learning and opening up a tracking paradigm based on deep matching. Subsequently, researchers have continuously introduced region proposal networks (such as SiamRPN), boundary regression modules, attention mechanisms, and occlusion modeling strategies to further improve the accuracy and robustness of tracking.
[0005] Meanwhile, the introduction of the Transformer architecture has provided a new direction for object tracking. Compared with CNNs, Transformers can more effectively model global dependencies and long-range spatiotemporal information, exhibiting stronger context modeling capabilities in complex scenes. For example, methods such as STARK, TransT, and OSTrack introduce self-attention mechanisms and cross-attention modules into the tracking architecture, achieving more accurate information interaction between the template and the search region. Through spatiotemporal feature fusion, dynamic update mechanisms, and lightweight design, these methods improve accuracy while also maintaining real-time performance to some extent, becoming current research hotspots.
[0006] While current methods have achieved excellent performance on multiple public datasets, trackers still need to find a more reasonable balance between speed, accuracy, and robustness in practical deployments. Meanwhile, better adapting to changes in target appearance, addressing occlusion and re-identification issues, and improving the model's cross-scene generalization ability remain core challenges in target tracking research. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention proposes a target tracking method based on temporal information enhancement, which includes:
[0008] S1: Obtain the video search frame and multiple template frames, and preprocess the video search frame and multiple template frames;
[0009] S2: Input the pre-processed video search frames and multiple template frames into the pre-trained M-STEP model for processing to obtain image fusion features; the M-STEP model includes the SFE module, MSTE module, SIF module and TPE module;
[0010] S3: Input the image fusion features into the tracking and localization head for processing to obtain the target tracking result.
[0011] Preferably, the preprocessing process for the video search frame and multiple template frames includes: unifying the image size of the video search frame and multiple template frames; and performing noise reduction processing on the unified-size video search frame and multiple template frames to obtain the preprocessed video search frame and multiple template frames.
[0012] Preferably, the M-STEP model's processing of video search frames and multiple template frames includes:
[0013] S21: Input the video search frame and multiple template frames into the SFE module, and output the compressed search image features and compressed template image features;
[0014] S22: Input multiple template frames into the MSTE module to obtain multi-scale template image features;
[0015] S23: The SIF module fuses compressed search image features, compressed template image features, multi-scale template image features, and tracking temporal features from the previous frame to obtain image fusion features;
[0016] S24: Input the multi-scale template image features and image fusion features into the TPE module to obtain the tracking temporal features of the next frame.
[0017] Furthermore, the SFE module includes a cascaded LayerNorm layer, MLP layer, and PatchMerge layer.
[0018] Furthermore, the MSTE module includes three branches, each consisting of three types of convolutional layers and one pyramid pooling layer. Multiple template frames are passed through the three branches to obtain image features at three different scales. The image features at the three different scales are then stitched together and input into the FFN layer for processing to obtain multi-scale template image features.
[0019] Furthermore, the convolution kernels of the three types of convolutional layers are respectively , and The pooling kernel step sizes for the three branches of the pyramid pooling layer are 1, 2, and 5, respectively.
[0020] Furthermore, the process of outputting image fusion features by the SIF module includes: stitching and compressing the search image features, compressing the template image features, multi-scale template image features, and the tracking temporal features of the previous frame to obtain stitched features; inputting the stitched features into an attention module consisting of at least 20 multi-head self-attention modules connected in series for processing to obtain image fusion features; wherein, each attention module includes a multi-head self-attention layer and an MLP layer.
[0021] Furthermore, the TPE module includes an attention module, a Mamba module, and a fusion layer; the Mamba module contains a MambaLayer and an MLP layer, and the attention module includes a multi-head self-attention layer and an MLP layer.
[0022] The beneficial effects of this invention are as follows:
[0023] This invention designs an M-STEP model to achieve target localization and tracking. By utilizing the M-STEP model to transmit contextual information from the video, the accuracy of the tracking model is improved. Compared with existing visual target tracking methods, this invention has higher tracking accuracy. At the same time, this invention can effectively track and locate targets even when they are occluded, and can effectively adapt to various complex tracking scenarios, showing good application prospects. Attached Figure Description
[0024] Figure 1This is a flowchart of the target tracking method based on time-series information enhancement in this invention;
[0025] Figure 2 This is a schematic diagram of the M-STEP model framework according to a preferred embodiment of the present invention;
[0026] Figure 3 This is a schematic diagram of the MSTE module framework in this invention;
[0027] Figure 4 This is a schematic diagram of the TPE module framework in this invention. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] This invention proposes a target tracking method based on temporal information enhancement, such as... Figure 1 As shown, the method includes the following:
[0030] S1: Obtain the video search frame and multiple template frames, and preprocess the video search frame and multiple template frames.
[0031] The system acquires video search frames and multiple template frames. Template frames represent objects to be tracked, while search frames represent images in the video that contain the target to be tracked.
[0032] The video search frame and multiple template frames are preprocessed, specifically: the image size of the video search frame and multiple template frames is unified; the unified-size video search frame and multiple template frames are then denoised to obtain the preprocessed video search frame and multiple template frames.
[0033] S2: Input the pre-processed video search frames and multiple template frames into the pre-trained M-STEP model for processing to obtain image fusion features; the M-STEP model includes the SFE module, MSTE module, SIF module and TPE module.
[0034] This invention uses a pre-trained M-STEP model to process video search frames and multiple template frames to obtain image fusion features. For example... Figure 1 , Figure 2 As shown, the processing procedure includes:
[0035] S21: Input the video search frame and multiple template frames into the SFE module, and output compressed search image features and compressed template image features; specifically:
[0036] The SFE module consists of a cascaded LayerNorm layer, an MLP layer, and a PatchMerge layer. This module takes the image information of the search frame and multiple template frames as input data, respectively, with feature vector scales of [missing information]. Where B represents the batch size of the vector, N represents the length of the vector, and C represents the number of channels in the vector. The input data is first standardized by a LayerNorm layer, then shallow features of the image are extracted by an MLP layer (Multilayer Perceptron), and finally the feature vector is scaled down by a PatchMerge layer. After PatchMerge, the entire vector dimension is reduced to [value missing]. Size.
[0037] S22: Input multiple template frames into the MSTE module to obtain multi-scale template image features; specifically:
[0038] like Figure 3 As shown, the MSTE module consists of three branches, each composed of three types of convolutional layers and a pyramid pooling layer. Multiple template frames are passed through these three branches, resulting in image features at three different scales for each template. The image features at all three scales are then concatenated and fed into the FFN (feedforward neural network) layer for further processing to obtain multi-scale template image features. The convolutional layers of the MSTE module use three different sized convolutional kernels. , and The pyramid pooling layer also uses three pooling kernels with different step sizes: 1, 2, and 5.
[0039] This module can effectively extract multi-scale information from template frames. It primarily extracts multi-scale information from multiple template images, with the input scale being... N represents the number of templates, C represents the color channels of the image, H represents the height of the image, and W represents the width of the image. Multiple input templates are passed through three convolutional kernels of different sizes to obtain features at three different scales. These features at different scales are then fed into a pyramid pooling layer for pooling operations. The final output feature dimension is... .
[0040] S23: The SIF module fuses compressed search image features, compressed template image features, multi-scale template image features, and the tracking temporal features of the previous frame to obtain image fusion features; specifically:
[0041] The SIF module is primarily used to fuse temporal and image feature information. This module first concatenates compressed search image features, compressed template image features, multi-scale template image features, and the tracking temporal features from the previous frame. Specifically, the temporal information of the first frame is initialized with a single dimension. A random array. The dimensions of the four types of information are all... The stitched features are obtained; the stitched features are input into an attention module consisting of at least 20 multi-head self-attention modules connected in series for processing to obtain image fusion features; each attention module includes a multi-head self-attention layer and an MLP layer, the MLP layer is mainly responsible for the dimensionality change operation of the features.
[0042] S24: Input the multi-scale template image features and image fusion features into the TPE module to obtain the tracking temporal features for the next frame; specifically:
[0043] In the design of the M-STEP model, the TPE module is mainly used to perform temporal information filtering and enhancement operations. This module includes two inputs: one is the template information extracted from the MSTE module, i.e., multi-scale template image features, and the other is the temporal information extracted from the SIF module, i.e., image fusion features. At the same time, the temporal information output by TPE, i.e., tracking temporal features, will also be passed to the tracking process of subsequent search frames.
[0044] like Figure 4 As shown, the TPE module includes an attention module, a Mamba module, and a fusion layer. The Mamba module contains a MambaLayer and an MLP layer, while the attention module includes a multi-head self-attention layer and an MLP layer. The fusion layer uses Sigmoid to calculate the corresponding weight distribution of the temporal information from the previous frame and the enhanced temporal information, then performs weighted calculations and finally obtains the subsequent temporal information through the MLP.
[0045] S3: The image fusion features are input into the tracking and localization head for processing. The tracking and localization head typically uses a convolutional network for target positioning. This invention selects three convolutional network branches to calculate scores for three different score categories. Bounding box score Offset score The three scores are combined to represent the tracking result of the target.
[0046] The image fusion features obtained by processing the current video search frame, multiple template frames, and the tracking temporal features of the previous frame are input into the tracking and positioning head for processing, which can obtain the target in the current video search frame, i.e., the target tracking result.
[0047] By repeatedly performing the above process, the target in each frame of the video can be obtained, enabling continuous positioning and tracking of the target.
[0048] In summary, this invention processes video search frames and multiple template frames using M-STEP to obtain image fusion features, and then uses a tracking and positioning head to process these features to obtain the target tracking result. This invention can effectively achieve video target localization and tracking, with a strong anti-interference model, high tracking accuracy, and significantly improved accuracy and convenience of video target tracking.
[0049] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A target tracking method based on temporal information enhancement, characterized in that, include: S1: Obtain the video search frame and multiple template frames, and preprocess the video search frame and multiple template frames; S2: Input the pre-processed video search frames and multiple template frames into the pre-trained M-STEP model for processing to obtain image fusion features; the M-STEP model includes the SFE module, MSTE module, SIF module and TPE module; S3: Input the image fusion features into the tracking and localization head for processing to obtain the target tracking result.
2. The target tracking method based on time-series information enhancement according to claim 1, characterized in that, The preprocessing process for the video search frame and multiple template frames includes: unifying the image size of the video search frame and multiple template frames; and performing noise reduction processing on the unified-size video search frame and multiple template frames to obtain the preprocessed video search frame and multiple template frames.
3. The target tracking method based on time-series information enhancement according to claim 1, characterized in that, The M-STEP model's processing of video search frames and multiple template frames includes: S21: Input the video search frame and multiple template frames into the SFE module, and output the compressed search image features and compressed template image features; S22: Input multiple template frames into the MSTE module to obtain multi-scale template image features; S23: The SIF module fuses compressed search image features, compressed template image features, multi-scale template image features, and tracking temporal features from the previous frame to obtain image fusion features; S24: Input the multi-scale template image features and image fusion features into the TPE module to obtain the tracking temporal features of the next frame.
4. The target tracking method based on time-series information enhancement according to claim 3, characterized in that, The SFE module includes a cascaded LayerNorm layer, MLP layer, and PatchMerge layer.
5. The target tracking method based on time-series information enhancement according to claim 3, characterized in that, The MSTE module includes three branches, each consisting of three types of convolutional layers and a pyramid pooling layer. Multiple template frames are passed through the three branches to obtain image features at three different scales. The image features at the three different scales are then stitched together and input into the FFN layer for processing to obtain multi-scale template image features.
6. The target tracking method based on time-series information enhancement according to claim 5, characterized in that, The convolution kernels of the three convolutional layers are as follows: , and The pooling kernel step sizes for the three branches of the pyramid pooling layer are 1, 2, and 5, respectively.
7. The target tracking method based on time-series information enhancement according to claim 3, characterized in that, The process of outputting image fusion features by the SIF module includes: stitching and compressing the search image features, compressing the template image features, multi-scale template image features, and the tracking temporal features of the previous frame to obtain stitched features; inputting the stitched features into an attention module consisting of at least 20 multi-head self-attention modules connected in series for processing to obtain image fusion features; wherein, each attention module includes a multi-head self-attention layer and an MLP layer.
8. The target tracking method based on time-series information enhancement according to claim 3, characterized in that, The TPE module includes an attention module, a Mamba module, and a fusion layer; wherein, the Mamba module contains a MambaLayer layer and an MLP layer, and the attention module includes a multi-head self-attention layer and an MLP layer.