A visual target tracking method and system
By generating high-quality segmentation masks through self-supervised correspondence learning and temporal attention, the robustness and computational resource consumption of visual target tracking methods in complex scenarios are solved, and efficient real-time tracking is achieved.
Patent Information
- Application Number
- CN202411058640.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-08-02
AI Technical Summary
Existing visual target tracking methods lack robustness in complex scenarios such as changes in target appearance, occlusion, and rapid movement, and consume high computational resources, making it difficult to achieve efficient real-time tracking.
We employ a neighbor-memory-based segmentation network approach, which generates high-quality segmentation masks through self-supervised correspondence learning, temporal attention, and segmentation quality evaluation. This reduces noise and ambiguity, and enhances tracking accuracy and stability.
It improves the robustness and accuracy of target tracking, reduces computational resource consumption, and enables efficient real-time tracking in complex scenarios, meeting the speed and accuracy requirements of practical applications.
Smart Images

Figure CN119027451B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of pattern recognition and computer vision, and specifically relates to a visual target tracking method and system based on a neighboring memory segmentation network. Background Technology
[0002] Visual object tracking is a fundamental and highly challenging task in computer vision, primarily aimed at accurately identifying and tracking specified target objects within video sequences. Traditional visual tracking methods largely rely on target templates in the initial frame, predicting the target's position and state in subsequent frames through template matching. However, these traditional methods often exhibit a sharp performance degradation when the target undergoes changes in appearance, rapid movement, occlusion, or changes in lighting conditions within the video.
[0003] In recent years, significant technological advancements have been achieved in the field of target tracking with the introduction and development of twin networks and transformer technologies. These emerging methods effectively improve the accuracy and robustness of the tracking process by establishing a similarity metric between the target template and the search region. They can dynamically adjust and optimize tracking strategies across consecutive video frames, thereby coping with rapid changes in target morphology and complex environmental interference.
[0004] Despite this, these tracking methods based on advanced neural network models still face several key challenges. First, these methods typically use static target templates cropped from the initial frame, which makes it difficult to maintain long-term tracking stability when the target's appearance changes significantly in subsequent frames. Furthermore, many existing tracking algorithms tend to focus only on the similarity between the target and the template, neglecting the spatiotemporal consistency throughout the video sequence, which can sometimes lead to target drift. Additionally, to achieve higher tracking accuracy, some methods attempt to dynamically adjust the tracking template by introducing more complex network structures or online template update strategies; however, these strategies often require significant computational resources, increasing the algorithm's operating cost and time, and affecting its feasibility in practical applications.
[0005] To address these challenges, researchers have begun exploring various strategies, including spatial-temporal attention mechanisms, multimodal feature fusion, and online template updates. For example, spatial-temporal attention mechanisms enable algorithms to more accurately identify and respond to real-time changes in the state of a target in a video, while multimodal feature fusion techniques can leverage information from different sensors and data sources to enhance the tracking algorithm's adaptability to complex environments. Online template updates allow the tracking algorithm to dynamically adjust the target template during tracking to adapt to continuous changes in the target's appearance. These improved strategies have enhanced the overall performance and adaptability of tracking technology to some extent, but they also introduce higher computational burdens, which is a problem that current research needs to further address. Summary of the Invention
[0006] To address the robustness issues of existing visual target tracking methods in complex scenarios such as changes in target appearance, occlusion, and rapid movement, this invention provides a visual target tracking method and system based on a neighbor-memory segmentation network. The method includes the following steps:
[0007] Step S1: Initialize the adjacent memory segmentation network and receive the first frame I in the video stream. z Previous frame I p and current frame I x Together, they constitute the network input, which passes through the ConvNext network to obtain the first frame image features E. z Previous frame image features E p and current frame image features E x Among them, the previous frame I p For the current frame I x The previous frame;
[0008] Step S2: Define and optimize the self-supervised corresponding learning loss function L CLL The model is constrained to learn time-invariant features of image pixels / blocks at each time step, thereby enhancing the features of the current frame image.
[0009] Step S3: Combine with previous frame I p And the first frame I z Image features and mask features for the current frame I x Perform feature enhancement to obtain the target features F of the current frame. x ;
[0010] Step S4: Transfer the target features F of the current frame x Generate a segmentation mask Y m ;
[0011] Step S5: Evaluate the current frame segmentation mask Y m The quality;
[0012] Step S6: In the online tracking phase, the adjacent memory segmentation network trained in steps S1-S5 is used to initialize and run the online target tracking model.
[0013] Corresponding to the above method, the present invention also provides a visual target tracking system, comprising the following modules:
[0014] The image encoder module is used to extract features from the input frame and execute step S1;
[0015] The self-supervised correspondence learning module is used to learn the spatiotemporal correspondences in the video and executes step S2.
[0016] The temporal attention module is used to enhance the feature representation of the current frame, and step S3 is executed;
[0017] The segmentation network module is used to generate the segmentation mask, and step S4 is executed;
[0018] The segmentation quality evaluation module is used to evaluate the quality of the segmentation mask, and steps S5 are executed.
[0019] The tracking decision module is used to track the target based on the system output during the online tracking phase and executes step S6.
[0020] Compared with existing technologies, this invention has the following advantages: (1) The self-supervised correspondence learning mechanism does not require additional annotation information and uses the spatiotemporal coherence of the video itself as a supervision signal, which enhances the robustness of feature representation. (2) The use of a temporal attention module can better adapt to changes in the appearance of the target and improve the response capability to changes in the target state during tracking. (3) The proposed segmentation network directly generates segmentation masks instead of relying solely on bounding boxes, providing a more accurate representation of the target, reducing noise and ambiguity, and enhancing the accuracy of tracking. (4) The proposed segmentation quality evaluation module evaluates the quality of the segmentation mask, ensuring that the information stored in the memory is of high quality, avoiding error accumulation, and improving the stability of long-term tracking. (5) The tracking algorithm of this invention is highly efficient and can achieve real-time tracking while maintaining high accuracy, meeting the dual requirements of speed and accuracy in practical applications. Attached Figure Description
[0021] Figure 1 This is a diagram showing the composition of the visual tracking system of the adjacent memory segmentation network of the present invention;
[0022] Figure 2 This is a flowchart of the visual tracking method of the adjacent memory segmentation network of the present invention;
[0023] Figure 3 This is a flowchart of the adjacent memory segmentation network tracker in an embodiment of the present invention. Detailed Implementation
[0024] See Figure 1 The visual target tracking system of the adjacent memory segmentation network of the present invention includes the following modules: an image encoder module 51 for extracting features of the input frame; a self-supervised correspondence learning module 52 for learning the spatiotemporal correspondence in the video; a temporal attention module 53 for enhancing the feature representation of the current frame; a segmentation network module 54 for generating a segmentation mask; a segmentation quality evaluation module 55 for evaluating the quality of the segmentation mask; and a tracking decision module 56 for performing target tracking based on the system output during the online tracking phase.
[0025] See Figure 2The present invention provides a visual target tracking method based on a neighbor-memory segmentation network, which mainly includes the following steps:
[0026] Step S1: Initialize the adjacent memory segmentation network and receive the first frame I in the video stream. z Previous frame I p and current frame I x Together, they constitute the network input, which passes through the ConvNext network to obtain the first frame image features E. z Previous frame image features E p and current frame image features E x Among them, the previous frame I p For the current frame I x The previous frame. This step is implemented by the image encoder module 51, such as... Figure 3 The image encoder section is shown below.
[0027] Step S2: Define and optimize the self-supervised corresponding learning loss function L CLL The model is constrained to learn time-invariant features of image pixels / blocks at each time step, thereby enhancing the features of the current frame image. This step is implemented by the self-supervised correspondence learning module 52.
[0028]
[0029] Among them, L CLL The corresponding learning loss is used to optimize the model to learn the pixel-level correspondences between consecutive frames; E p (i) represents the feature vector of the i-th pixel in the previous frame; E x (j) represents the feature vector of the j-th pixel in the current frame; <E p (i),E x (j)> The dot product is used to calculate the similarity between two pixel feature vectors; ∑ j It sums the values of all pixels j in the current frame and is used to normalize the weights.
[0030] Through L CLL The loss function optimizes the current frame image features E. x and previous frame image features E p This updates the image encoder parameters, thereby enhancing image feature representation. CLL The closer the loss function value is to 0, the better the image features E of the current frame. x and previous frame image features E p There is a stronger spatial correspondence.
[0031] In target tracking, especially long-term target tracking, the appearance of the target in the current frame differs significantly from that in the initial frame. This means there's a possibility that the background target in the current frame is more similar in appearance to the target in the first frame than the actual target in the current frame. However, targets in adjacent frames often have very small offsets; that is, the tracked target in the current frame is approximately equal in spatial location to the tracked target in the previous frame. This property can also be transferred to the image feature level. Based on these findings, a criterion is proposed: feature points at the same location on the feature maps of the current and previous frames have the highest similarity. Utilizing this spatial characteristic, the current frame image feature E is constrained... x and previous frame image features E p The correspondence.
[0032] By using the similarity criterion between pixels to define the corresponding learning loss, the model's representation ability is trained and optimized. In this progressive way, the model can learn the correspondence between frames in the video in a self-supervised manner, and the background and background features correspond to each other, thereby capturing the target tracking appearance change information better and more accurately.
[0033] This step leverages the spatial invariance of continuous video to design a self-supervised learning optimization strategy. This enhances the stability and robustness of feature learning for the current frame, significantly improving the model's performance in video tracking tasks. This self-supervised learning method does not rely on labeled data; it enhances the tracking model's ability to adapt to target changes by utilizing the inherent coherence within the video.
[0034] Step S3: Combine with previous frame I p And the first frame I z Image features and mask features for the current frame I x Perform feature enhancement to obtain the target features F of the current frame. x Implemented by the time attention module 53.
[0035] The specific method is as follows:
[0036] Step S31: Use a mask encoder to divide the segmentation mask Y of the first frame. z The segmentation mask Y of the previous frame p Mask features M converted to the first frame z Mask features M from previous frames p The segmentation masks for the first frame and previous frames are labels provided during training.
[0037] Step S32: Calculate the current frame image features E using a multi-head cross-attention mechanism layer and a feedforward network. x With the first frame image feature E z Previous frame image features E p Correlation plot F z→x ,F p→xThe formula is as follows:
[0038] F z→x =MHCA(E x +P x E z +P z E z +M z );
[0039] F p→x =MHCA(E x +P x E p +P p E p +M p );
[0040] Where F z→x F represents the features from the first frame to the current frame calculated by the multi-head cross-attention layer; p→x The multi-head attention mechanism (MHCA) represents the feature obtained from the previous frame to the current frame through multi-head cross-attention computation; P z ,P p ,P x Indicates the spatial location embedding of the first frame, previous frames, and current frame; E x +P x This indicates that the features of the current frame are added to the spatial location embedding to provide the query to the multi-head cross-attention layer; E z +P z and E p +P p This means that the features of the first frame and the previous frame are added together with their respective spatial locations and provided to the multi-head cross-attention layer as keys and values.
[0041] In this invention, M Z M p Mask features from the first and previous frames are added to the corresponding image features to enhance the model’s ability to recognize the target object; the ability to consider the temporal correlation between the current frame and the first and previous frames simultaneously helps to capture the dynamic changes of the target in the time series.
[0042] Step S33: Through layer normalization and feedforward network, merge the correlation maps to obtain the target features F of the current frame. x The formula is as follows:
[0043] F x =Norm(Norm(E x +F z→x +F p→x )+FFN(E x ))
[0044] Here, Norm represents layer normalization, used to stabilize the training process, and FFN represents a feedforward network used to extract deep representations of features.
[0045] In this invention, the target feature F of the current frame x After layer normalization and final processing by the feedforward network, it not only focuses on the target itself, but also utilizes information in the time dimension, thereby improving the robustness of tracking.
[0046] Step S4: Transfer the target features F of the current frame x Generate a segmentation mask Y m This is implemented by the segmentation network module 54. The specific method is as follows:
[0047] Step S41: Transfer the target features F of the current frame x The input is fed into the detection head to obtain the corresponding classification feature map Y. cls Classification feature map Y cls This is used to predict the target's position in the current frame, which is a common practice in neural network-based target tracking.
[0048] Step S42: Based on the classification feature map Y cls Using a multi-layer convolutional neural network H θ For classification feature map Y cls and target features F x Encoding is performed to obtain enhanced mask-coded features x. cm ;
[0049] Step S43: Encode the enhanced mask feature x cm The data is fed into the mask decoder to generate the final segmentation mask Y. m Mask decoders are a common practice in the field of object segmentation; here, the AlphaRefine method is used.
[0050] Segmentation mask Y m This invention not only provides an accurate description of the target's shape for the current frame but also serves as a feature of the previous frame for tracking in the next frame. Unlike traditional methods that rely solely on bounding boxes, this invention uses a segmentation mask to provide a more accurate description of the target's appearance, reducing noise and ambiguity. This process involves upsampling and convolution operations to generate pixel-level segmentation results from the feature map.
[0051] Step S5: Evaluate the current frame segmentation mask Y m The quality is achieved by the segmentation quality assessment module 55, and the network structure is as follows: Figure 3 The segmentation quality assessment module is shown below. The specific method is as follows:
[0052] Step S51: Use a mask encoder to divide the mask Y mMapping to mask embedding f s .
[0053] Step S52: Transfer the target features F of the current frame x With mask embedding f s By combining convolution and linear block processing, a segmentation quality score S is obtained. q The segmentation quality score S q Reflects the segmentation mask Y m The quality.
[0054] Step S53: Calculate the segmentation quality loss function L qua The formula used to evaluate the quality of the segmentation mask is as follows:
[0055] L qua =(S q -GT iou ) 2
[0056] Among them, GT iou It is the segmentation mask Y m And the real segmentation mask Y gt The intersection-union ratio between them. The so-called true segmentation mask Y. gt It is the labeled data used during training and evaluation to determine the accurate location and shape of the target object in the image. qua The closer S is to 0, the better. q The closer the value is to the intersection-union ratio (GT) between the predicted and actual segmentation masks, the better. iou After completing the training, S q The higher the value, the more likely the segmentation mask Y is to be considered. m The closer it is to the true segmentation mask Y gt The currently predicted segmentation mask Y m The more reliable, the better. Therefore, L is adopted. qua The loss function can evaluate the quality of the segmentation mask.
[0057] By introducing segmentation quality assessment loss L qua This method quantifies and monitors the quality of the predicted segmentation mask. This ensures that relevant information is only added to the memory bank when the quality of the predicted mask exceeds a preset strict threshold, effectively avoiding interference from low-quality data and reducing accumulated errors during tracking. This approach significantly improves the accuracy and robustness of the tracking algorithm, especially in environments with changing target appearance or complex tracking conditions, ensuring that the tracking system can continuously and stably capture and update the target's true state.
[0058] Step S6: In the online tracking phase, the adjacent memory segmentation network trained in steps S1-S5 is used to initialize and run the online target tracking model. This is implemented by the tracking decision module 56.
[0059] At the start of target tracking, the first frame is copied as the previous frame, and the target features of both the first and previous frames are obtained. During continuous tracking from the second frame onwards, the target features of the previous frame are updated in real time. Therefore, it is necessary to select a reliable, high-quality current frame as the previous frame.
[0060] Determine a quality threshold τ s This threshold is used to determine whether the segmentation quality of the current frame is high enough; it is determined empirically or through cross-validation. If S q Above the threshold τ s If the current frame's segmentation mask quality is deemed high enough, then the current frame should be updated to the previous frame in the network input. This indicates that the current frame provides useful and accurate information about the target object. If S q Below the threshold τ s If the network input is not updated, it avoids including low-quality or potentially erroneous segmentation masks in the network input, which helps maintain the quality and consistency of the network input data. Experiments in this invention have shown that a suitable threshold of 0.8 yields good performance. This threshold may need to be adjusted based on specific application scenarios and datasets to achieve the best tracking results.
[0061] Steps S1 to S5 above constitute the offline training phase, which aims to train an adjacent memory segmentation network that can accurately generate segmentation masks and evaluate their quality; step S6 is used for online tracking.
[0062] The online tracking phase of this invention can be updated in real time and utilize high-quality intermediate frames to cope with dynamic changes in the target.
[0063] This invention relates to a visual target tracking method based on a neighboring memory segmentation network, which is specially designed to enhance robustness to complex scenarios such as changes in target appearance, occlusion, and rapid movement.
[0064] To better verify the effectiveness of the visual target tracking method provided by this invention, the embodiments of this invention select publicly available target tracking datasets such as TrackingNet, LaSOT, and GOT-10k for testing.
[0065] Table 1: Performance comparison of the method of the present invention with the state-of-the-art trackers on different datasets.
[0066]
[0067]
[0068] Result comparison description:
[0069] AUC measures the area under the curves for tracking accuracy and position accuracy, reflecting the overall performance of the algorithm at different thresholds.Norm P represents the normalized accuracy, i.e., the proportion of correct predictions out of all positive predictions. P directly reflects the proportion of correct predictions made by the algorithm. AO is the average overlap between the tracking prediction and the real target, measuring the accuracy of the tracking. SR 0.5 and SR 0.75 These are success rate metrics, representing the success rate when the overlap between the predicted bounding box and the true bounding box exceeds 50% and 75%, respectively. These two metrics can reflect the robustness of the algorithm under different overlap thresholds.
[0070] Test results on the TrackingNet, LaSOT, and GOT-10K datasets demonstrate that the proposed method outperforms existing state-of-the-art trackers in multiple metrics. On the TrackingNet dataset, the proposed method outperforms existing state-of-the-art trackers in AUC, P... Norm The proposed method achieved AUC scores of 80.6%, 85.6%, and 77.5% on the three metrics (AO, SR, and P), respectively, representing a significant improvement over existing methods. On the LaSOT dataset, the proposed method achieved an AUC score of 66.8%, surpassing most existing methods. Particularly on the GOT-10K dataset, the proposed method demonstrated superior performance in AUC, SR, and P. 0.5 and SR 0.75 The metrics reached 71.0%, 82.2%, and 64.6% respectively, indicating that the method of the present invention exhibits superior tracking performance and robustness when processing different datasets.
[0071] The above tables and detailed description of the comparison results fully demonstrate the superior performance of the method of the present invention on different datasets, proving its effectiveness and robustness in visual target tracking.
[0072] This invention calculates sample weights, thus prioritizing important samples during training; that is, important samples are classified correctly first. Furthermore, by combining the ranking score and the weighted classification score, more accurate candidate boxes can be selected, improving target tracking accuracy. The tracking algorithm of this invention has significant efficiency advantages, achieving real-time tracking while maintaining high accuracy, meeting the dual requirements of speed and accuracy in practical applications.
[0073] The above embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. All equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.
Claims
1. A visual target tracking method, characterized in that, Includes the following steps: Step S1: Initialize the adjacent memory segmentation network and receive the first frame from the video stream. Previous frame I p and the current frame I x Together, they constitute the network input, which is then processed by the ConvNext network to obtain the features of the first frame image. E z Previous frame image features E p and current frame image features E x Among them, the previous frame I p For the current frame The previous frame; Step S2: Define and optimize the self-supervised corresponding learning loss function The model is constrained to learn time-invariant features of image pixels or image blocks at each time step, thereby enhancing the features of the current frame image. Step S3: Combine with previous frames I p and the first frame Image features and mask features for the current frame Perform feature enhancement to obtain the target features of the current frame. ; Step S4: Extract target features from the current frame Generate a segmentation mask ; Step S5: Evaluate the segmentation mask for the current frame The quality; Step S6: In the online tracking stage, the adjacent memory segmentation network trained in steps S1-S5 is used to initialize and run the online target tracking model; In step S2, the loss function The corresponding learning loss is used to optimize the model to learn the pixel-level correspondences between consecutive frames. , in, E p ( i ) indicates the first frame in the previous frame Feature vector of 1 pixel; E x ( j ) indicates the first in the current frame Feature vector of 1 pixel; E p ( i ), E x ( j ) The dot product is used to calculate the similarity between two pixel feature vectors; ∑ j It refers to all pixels in the current frame. Summation is used to normalize weights; Step S3 further includes: Step S31: Use a mask encoder to segment the mask of the first frame. Segmentation mask of previous frames Mask features converted to the first frame Mask features of previous frames ; Step S32: Calculate the image features of the current frame using a multi-head cross-attention mechanism layer and a feedforward network. Features of the first frame image E z Previous frame image features Correlation plot → , → The formula is as follows: → =MHCA( + P , z + P z , z + M z ); → =MHCA( + P , p + P p , p + M p ); in → This represents the features from the first frame to the current frame calculated by the multi-head cross-attention layer; → This represents the features from the previous frame to the current frame obtained by multi-head cross attention calculation; MHCA stands for multi-head attention mechanism. , , This indicates the spatial position embedding of the first frame, previous frames, and current frame; + This means adding the features of the current frame to the spatial location embedding to provide the multi-head cross-attention layer as a query; + and + This means adding the features of the first frame and the previous frame to their respective spatial location embeddings, and providing them to the multi-head cross-attention layer as keys and values; Step S33: Through layer normalization and feedforward network, merge the correlation maps to obtain the target features of the current frame. The formula is as follows: F x =Norm(Norm( + → + → ) +FFN ( )) Norm represents the layer normalization operation, used to stabilize the training process. FFN This indicates that the feedforward network is used to extract a deep representation of the features.
2. The method according to claim 1, characterized in that: Step S4 further includes: Step S41: Add target features to the current frame The input is fed into the detection head to obtain the corresponding classification feature map. ; Step S42: Based on classification feature map Using multi-layer convolutional neural networks Classification feature map and target features Encoding is performed to obtain enhanced mask-coded features. ; Step S43: Encode the enhanced mask features The data is fed into the mask decoder to generate the final segmentation mask. .
3. The method according to claim 2, characterized in that: Step S5 further includes: Step S51: Use a mask encoder to divide the mask. Mapping to mask embedding ; Step S52: Extract target features from the current frame With mask embedding By combining convolution and linear block processing, a segmentation quality score is obtained. The segmentation quality score Reflects the segmentation mask The quality; Step S53: Calculate the segmentation quality loss function The formula used to evaluate the quality of the segmentation mask is as follows: =( S q - GT iou ) 2, in, It is a segmentation mask and real segmentation mask The intersection-union ratio between them.
4. A visual target tracking system for implementing the method according to any one of claims 1 to 3, characterized in that, Includes the following modules: The image encoder module (51) is used to extract features from the input frame and execute step S1; The self-supervised correspondence learning module (52) is used to learn the spatiotemporal correspondence in the video and execute step S2. The temporal attention module (53) is used to enhance the feature representation of the current frame and executes step S3; The segmentation network module (54) is used to generate a segmentation mask and execute step S4; The segmentation quality assessment module (55) is used to assess the quality of the segmentation mask and performs step S5. The tracking decision module (56) is used to track the target based on the system output during the online tracking phase and execute step S6.
Citation Information
Patent Citations
Event camera optical flow estimation method and device based on self-attention mechanism
CN115601403A
Interactive video character tracking method and system based on self-supervised optical flow learning
CN117392180A
