Visual target tracking method and system
Through self-supervising corresponding learning and time attention modules, segmentation masks are generated, combined with segmentation quality evaluation modules, the robustness of visual target tracking in complex scenarios is solved, and efficient and real-time target tracking is achieved to meet the accuracy of practical applications. Sex and speed requirements.
Patent Information
- Application Number
- CN202411058640.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing visual target tracking methods are not robust enough in complex scenarios such as target appearance changes, occlusion and rapid movement, making it difficult to maintain long-term tracking stability, and the computing resources are consumed, which affects the feasibility of practical applications.
Using a method based on adjacent memory segmentation network, through self-supervising corresponding learning, time attention module and segmentation quality evaluation module, segmentation mask is generated and tracked online, reducing noise and ambiguity, and enhancing the responsiveness and robustness to target state changes. sex.
It improves the accuracy and stability of target tracking, reduces computing resource consumption, realizes real-time tracking in complex scenarios, and meets the speed and accuracy requirements in actual applications.
Smart Images

Figure CN119027451A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of pattern recognition and computer vision, and in particular relates to a visual target tracking method and system based on an adjacent memory segmentation network. Background Art
[0002] Visual object tracking is a fundamental and challenging task in the field of computer vision. Its main purpose is to accurately identify and track a specified target object in a video sequence. Traditional visual tracking methods mostly rely on the target template in the initial frame and predict the position and state of the target in subsequent frames by template matching. However, these traditional methods often show a sharp drop in performance when the target changes in appearance, moves quickly, is occluded, or has changing lighting conditions in the video.
[0003] In recent years, with the introduction and development of Siamese networks and transformer technology, the field of target tracking has achieved significant technological progress. These emerging methods effectively improve the accuracy and robustness of the tracking process by establishing a similarity measure between the target template and the search region. They are able to dynamically adjust and optimize the tracking strategy in consecutive video frames to cope with the rapid changes in target morphology and complex environmental interference.
[0004] Nevertheless, these tracking methods based on advanced neural network models still face several key challenges. First, these methods usually use a static target template cropped in the initial frame, which makes it difficult to maintain long-term tracking stability when the target undergoes significant appearance changes in subsequent frames. In addition, many existing tracking algorithms tend to focus only on the similarity between the target and the template, while ignoring the spatiotemporal consistency in the entire video sequence, which sometimes causes the drift of the tracked target. In addition, in order to achieve higher tracking accuracy, some methods try to dynamically adjust the tracking template by introducing more complex network structures or online template update strategies, but these strategies often require a lot of computing resources, which increases the running cost and time of the algorithm, affecting its feasibility in practical applications.
[0005] In response to these challenges, researchers have begun to explore a variety of strategies, including space-time attention mechanisms, multimodal feature fusion, and online template updates. For example, through the space-time attention mechanism, the algorithm can more accurately identify and respond to real-time changes in the state of the target in the video, while the multimodal feature fusion technology can use information from different sensors and data sources to enhance the tracking algorithm's ability to adapt to complex environments. Online template update technology allows the tracking algorithm to dynamically adjust the target template during the tracking process to adapt to the continuous changes in the target's appearance. These improved strategies have improved the overall performance and adaptability of tracking technology to a certain extent, but at the same time have also brought a higher computational burden, which is a problem that needs to be further addressed in current research. Summary of the invention
[0006] In order to solve the robustness problem of existing visual target tracking methods in complex scenes such as target appearance changes, occlusion, and rapid motion, the present invention provides a visual target tracking method and system based on a neighboring memory segmentation network. The method comprises the following steps:
[0007] Step S1: Initialize the adjacent memory segmentation network and receive the first frame I in the video stream z , previous frame I p and the current frame I x Together they constitute the network input, and after passing through the ConvNext network, the first frame image feature E is obtained. z , previous frame image features E p and the current frame image feature E x ; Wherein, the previous frame I p The current frame I x The previous frame;
[0008] Step S2: Define and optimize the self-supervised learning loss function L CLL , constraining the model to learn the time-invariant features of the image pixels / blocks of the time step and enhance the features of the current frame image;
[0009] Step S3: Combine with previous frame I p and the first frame I z The image features and mask features of the current frame I x Perform feature enhancement to obtain the current frame target feature F x ;
[0010] Step S4: Set the current frame target feature F x Generate a segmentation mask Y m ;
[0011] Step S5: Evaluate the current frame segmentation mask Y m quality;
[0012] Step S6: In the online tracking phase, the neighboring memory segmentation network trained in steps S1-S5 is used to initialize and run the online target tracking model.
[0013] Corresponding to the above method, the present invention also provides a visual target tracking system, comprising the following modules:
[0014] An image encoder module, used to extract features of an input frame, and execute step S1;
[0015] A self-supervised correspondence learning module is used to learn the spatiotemporal correspondence relationship in the video and execute step S2;
[0016] A temporal attention module is used to enhance the feature representation of the current frame and execute step S3;
[0017] A segmentation network module, used to generate a segmentation mask and execute step S4;
[0018] A segmentation quality assessment module, used to assess the quality of the segmentation mask, and execute step S5;
[0019] The tracking decision module is used to track the target according to the system output in the online tracking stage, and executes step S6.
[0020] Compared with the prior art, the present invention has the following advantages: (1) A self-supervised corresponding learning mechanism that does not require additional annotation information and uses the spatiotemporal coherence of the video itself as a supervisory signal to enhance the robustness of feature representation. (2) The use of a temporal attention module can better adapt to changes in the target's appearance and improve the ability to respond to changes in the target's state during tracking. (3) The proposed segmentation network directly generates a segmentation mask instead of relying solely on a bounding box, providing a more accurate representation of the target, reducing noise and ambiguity, and enhancing tracking accuracy. (4) A proposed segmentation quality assessment module evaluates the quality of the segmentation mask, ensuring that the information stored in the memory is of high quality, avoiding error accumulation, and improving the stability of long-term tracking. (5) The tracking algorithm of the present invention is highly efficient and can achieve real-time tracking while maintaining high precision, meeting the dual requirements of speed and accuracy in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a composition diagram of the visual tracking system of the adjacent memory segmentation network of the present invention;
[0022] Figure 2 is a flow chart of the visual tracking method of the adjacent memory segmentation network of the present invention;
[0023] Figure 3 is a flow chart of a neighboring memory partitioning network tracker in an embodiment of the present invention. DETAILED DESCRIPTION
[0024] See also Figure 1 The visual target tracking system of the adjacent memory segmentation network of the present invention includes the following modules: an image encoder module 51: used to extract the features of the input frame; a self-supervised correspondence learning module 52: used to learn the spatiotemporal correspondence in the video; a temporal attention module 53: used to enhance the feature representation of the current frame; a segmentation network module 54: used to generate a segmentation mask; a segmentation quality assessment module 55: used to assess the quality of the segmentation mask; a tracking decision module 56: used to track the target according to the system output in the online tracking stage.
[0025] See also Figure 2The visual target tracking method based on the adjacent memory segmentation network of the present invention mainly includes the following steps:
[0026] Step S1: Initialize the adjacent memory segmentation network and receive the first frame I in the video stream z , previous frame I p and the current frame I x Together they constitute the network input, and after passing through the ConvNext network, the first frame image feature E is obtained. z , previous frame image features E p and the current frame image feature E x Among them, the previous frame I p The current frame I x This step is implemented by the image encoder module 51, such as Figure 3 The image encoder section is shown in FIG.
[0027] Step S2: Define and optimize the self-supervised learning loss function L CLL , the constraint model learns the time-invariant features of the image pixels / blocks of the time step, and enhances the features of the current frame image. This step is implemented by the self-supervised correspondence learning module 52.
[0028]
[0029] Among them, L CLL Correspondence learning loss, used to optimize the model to learn pixel-level correspondences between consecutive frames; E p (i) represents the feature vector of the i-th pixel in the previous frame; E x (j) represents the feature vector of the jth pixel in the current frame; <E p (i),E x (j)>The dot product is used to calculate the similarity between two pixel feature vectors;∑ j It is the sum of all pixels j in the current frame and is used to normalize the weights.
[0030] By L CLL Loss function, which can optimize the current frame image feature E x and the previous frame image features E p , and then update the image encoder parameters to enhance the image feature expression. CLL The closer the value of the loss function is to 0, the more the current frame image feature E x and the previous frame image features E p There is a stronger spatial correspondence.
[0031] In target tracking, especially long-term target tracking, the target appearance of the current frame is very different from that of the initial frame, that is, there is a possibility that the background target in the current frame is more similar to the target appearance in the first frame than the real target in the current frame. However, the offset between the targets of two adjacent frames is often very small, that is, the tracked target of the current frame and the tracked target of the previous frame are approximately equal in spatial position. This property can also be transferred to the image feature level. Based on the above findings, a criterion is proposed, that is, the feature points at the same position on the feature map of the current frame and the feature map of the previous frame have the highest similarity. Using this spatial characteristic, the current frame image feature E is constrained x and the previous frame image features E p The corresponding relationship.
[0032] The similarity criterion between pixels is used to define the corresponding learning loss, train and optimize the representation ability of the model. Through this progressive method, the model can self-supervise and learn the correspondence between frames in the video, and the correspondence between the background and background features, so as to better and accurately capture the appearance change information of target tracking.
[0033] This step is based on the spatial invariance of continuous videos. Using this principle, we design a self-supervised learning optimization strategy to promote the stability and robustness of the learning of the current frame image features, significantly improving the performance of the model in video tracking tasks. This self-supervised learning method does not rely on labeled data, but only uses the inherent coherence in the video to enhance the tracking model's ability to cope with target changes.
[0034] Step S3: Combine with previous frame I p and the first frame I z The image features and mask features of the current frame I x Perform feature enhancement to obtain the current frame target feature F x . This is implemented by the temporal attention module 53.
[0035] The specific method is as follows:
[0036] Step S31: Use a mask encoder to transform the segmentation mask Y of the first frame z , the segmentation mask Y of the previous frame p Converted to the mask feature M of the first frame z , the mask feature M of the previous frame p The segmentation masks of the first and previous frames are the labels provided during training.
[0037] Step S32: Calculate the current frame image feature E using the multi-head cross attention mechanism layer and feedforward network x With the first frame image feature E z , previous frame image features E p Correlation diagram F z→x ,F p→x, the formula is as follows:
[0038] F z→x =MHCA(E x +P x ,E z +P z ,E z +M z );
[0039] F p→x =MHCA(E x +P x ,E p +P p ,E p +M p );
[0040] where F z→x represents the features from the first frame to the current frame calculated by the multi-head cross attention layer; F p→x represents the features from the previous frame to the current frame obtained by multi-head cross attention calculation, MHCA represents the multi-head attention mechanism; P z ,P p ,P x represents the spatial position embedding of the first frame, previous frame, and current frame; E x +P x Indicates adding the features of the current frame to the spatial position embedding to provide it to the multi-head cross attention layer as a query; E z +P z and E p +P p Respectively, the features of the first frame and the previous frame are added with their respective spatial position embeddings and provided to the multi-head cross attention layer as keys and values.
[0041] In the present invention, M Z ,M p The mask features of the first frame and the previous frame are added to the corresponding image features to enhance the model's recognition ability of the target object; the temporal correlation between the current frame and the first frame and the previous frame can be considered at the same time, which helps to capture the dynamic changes of the target in the time series.
[0042] Step S33: Through layer normalization and feedforward network, the correlation graph is merged to obtain the current frame target feature F x , the formula is as follows:
[0043] F x =Norm(Norm(E x +F z→x +F p→x )+FFN(E x ))
[0044] Among them, Norm represents the layer normalization operation, which is used to stabilize the training process, and FFN represents the feedforward network used to extract the deep representation of features.
[0045] In the present invention, the current frame target feature F x After layer normalization and final processing by the feedforward network, it not only focuses on the target itself, but also utilizes information in the time dimension, thereby improving the robustness of tracking.
[0046] Step S4: Set the current frame target feature F x Generate a segmentation mask Y m , which is implemented by the segmentation network module 54. The specific method is as follows:
[0047] Step S41: The current frame target feature F x Input to the detection head to get the corresponding classification feature map Y cls , classification feature map Y cls Used to predict the position of the target in the current frame, which is a common practice for target tracking based on neural networks.
[0048] Step S42: Based on the classification feature map Y cls , using a multi-layer convolutional neural network H θ For the classification feature map Y cls and the target feature F x Encode and obtain the enhanced mask encoding feature x cm ;
[0049] Step S43: Encode the enhanced mask feature x cm Send to the mask decoder to generate the final segmentation mask Y m The mask decoder is a common practice in the field of target segmentation, and the AlphaRefine method is used here.
[0050] Segmentation mask Y m It not only provides an accurate description of the target shape for the current frame, but also can be used as the previous frame feature for tracking the next frame. Unlike the traditional method that only relies on the bounding box, the present invention uses the segmentation mask to provide a more accurate description of the target appearance, reducing noise and ambiguity. This process involves upsampling and convolution operations to generate pixel-level segmentation results from the feature map.
[0051] Step S5: Evaluate the current frame segmentation mask Y m The quality of the segmentation is achieved by the segmentation quality assessment module 55. The network structure is as follows Figure 3 The segmentation quality assessment module is shown in Figure 2. The specific method is as follows:
[0052] Step S51: Use the mask encoder to convert the segmentation mask Y mMapped to the mask embedding f s .
[0053] Step S52: Set the current frame target feature F x With the mask embedded f s Combined, through convolution and linear block processing, the segmentation quality score S is obtained q , the segmentation quality score S q reflects the segmentation mask Y m quality.
[0054] Step S53: Calculate the segmentation quality loss function L qua , used to evaluate the quality of the segmentation mask, the formula is as follows:
[0055] L qua =(S q -GT iou ) 2
[0056] Among them, GT iou is the segmentation mask Y m and the true segmentation mask Y gt The so-called true segmentation mask Y gt It is the annotation data used to determine the exact position and shape of the target object in the image during training and evaluation. qua The closer to 0, the higher the S q The closer the value of is to the intersection of the predicted segmentation mask and the true segmentation mask, the better the iou After completing the training, S q The higher the value, the more likely the segmentation mask Y m The closer it is to the true segmentation mask Y gt , the current predicted segmentation mask Y m Therefore, L qua The loss function can evaluate the quality of the segmentation mask.
[0057] By introducing the segmentation quality evaluation loss L qua , quantify and monitor the quality of the predicted segmentation mask. This ensures that relevant information is only included in the memory bank when the quality of the predicted mask exceeds a preset strict threshold, thereby effectively avoiding the interference of low-quality data and reducing the cumulative error in the tracking process. This method significantly improves the accuracy and robustness of the tracking algorithm, especially in the case of target appearance changes or complex tracking environments, ensuring that the tracking system can continuously and stably capture and update the true state of the target.
[0058] Step S6: In the online tracking stage, the neighboring memory segmentation network trained in steps S1-S5 is used to initialize and run the online target tracking model, which is implemented by the tracking decision module 56.
[0059] At the beginning of target tracking, the first frame is copied as the previous frame to obtain the target features of the first frame and the previous frame respectively. In the continuous tracking after the second frame, the target features of the previous frame are updated in real time. Therefore, it is necessary to select a reliable and high-quality current frame as the previous frame.
[0060] Determine a quality threshold τ s , is used to determine whether the segmentation quality of the current frame is high enough. This threshold is determined based on experience or through cross-validation. q Above the threshold τ s , then the segmentation mask quality of the current frame is considered to be high enough and the current frame should be updated to the previous frame in the network input, which means that the current frame provides useful and accurate information about the target object. q Below the threshold τ s , the network input is not updated to avoid including low-quality or possibly erroneous segmentation masks into the network input, which helps maintain the quality and consistency of the network input data. The appropriate threshold value verified by the present invention is 0.8, and the performance is good. This threshold value may need to be adjusted according to the specific application scenario and data set to achieve the best tracking effect.
[0061] The above steps S1 to S5 are an offline training stage, which aims to train a neighboring memory segmentation network that can accurately generate segmentation masks and evaluate their quality; step S6 is used for online tracking.
[0062] The online tracking stage of the present invention can update and utilize high-quality intermediate frames in real time to cope with dynamic changes of the target.
[0063] The visual target tracking method of the present invention is based on a neighboring memory segmentation network, which is specially designed to enhance the robustness to complex scenes such as target appearance changes, occlusions and rapid motions.
[0064] In order to better verify the effect of the visual target tracking method provided by the present invention, the embodiments of the present invention select target tracking public datasets such as TrackingNet, LaSOT and GOT-10k for testing.
[0065] Table 1: Performance comparison between the proposed method and the existing state-of-the-art trackers on different datasets
[0066]
[0067]
[0068] Result comparison description:
[0069] Among them, AUC measures the area percentage under the tracking accuracy and position accuracy curve, reflecting the comprehensive performance of the algorithm under different thresholds.Norm It represents the normalized accuracy, that is, the proportion of correct predictions among all positive predictions. P directly reflects the proportion of correct predictions made by the algorithm. AO is the average overlap between the tracking prediction and the true target, which measures the accuracy of tracking. SR 0.5 and SR 0.75 They are success rate indicators, which respectively represent the success rate when the overlap between the predicted bounding box and the true bounding box exceeds 50% and 75%. These two indicators can reflect the robustness of the algorithm under different overlap thresholds.
[0070] The test results on TrackingNet, LaSOT and GOT-10K datasets show that the proposed method outperforms the existing state-of-the-art trackers in multiple indicators. Norm The AUC scores of the proposed method on the LaSOT dataset are 66.8%, which exceeds most existing methods. In particular, on the GOT-10K dataset, the proposed method has a significant improvement in AO, SR 0.5 and SR 0.75 The indicators reached 71.0%, 82.2% and 64.6% respectively, indicating that the method of the present invention exhibits superior tracking performance and robustness when processing different data sets.
[0071] The above table and detailed description of the comparison results fully demonstrate the superior performance of the method of the present invention on different data sets, proving its effectiveness and robustness in visual target tracking.
[0072] The implementation of the present invention calculates the weight of the samples, so that more attention is paid to important samples during the training process, that is, important samples will be classified correctly with priority. And by combining the ranking score and the weight classification score, more accurate candidate boxes can be selected to improve the accuracy of target tracking. The tracking algorithm of the present invention has a significant advantage in efficiency, and can achieve real-time tracking while maintaining high accuracy, meeting the dual requirements of speed and accuracy in practical applications.
[0073] The above embodiments are only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principle of the present invention should all be included in the scope of the present invention.
Claims
1. A visual target tracking method, characterized in that: The following steps are involved: Step S1: Initialize the adjacent memory segmentation network and receive the first frame I in the video stream z , previous frame I p and the current frame I x Together they constitute the network input, and after passing through the ConvNext network, the first frame image feature E is obtained. z , previous frame image features E p and the current frame image feature E x ; Wherein, the previous frame I p The current frame I x The previous frame; Step S2: Define and optimize the self-supervised learning loss function L CLL , constraining the model to learn the time-invariant features of the image pixels / blocks of the time step and enhance the features of the current frame image; Step S3: Combine with previous frame I p and the first frame I z The image features and mask features of the current frame I x Perform feature enhancement to obtain the current frame target feature F x ; Step S4: Set the current frame target feature F x Generate a segmentation mask Y m ; Step S5: Evaluate the current frame segmentation mask Y m quality; Step S6: In the online tracking phase, the neighboring memory segmentation network trained in steps S1-S5 is used to initialize and run the online target tracking model.
2. The method according to claim 1, characterized in that: In step S2, the loss function L CLL Correspondence learning loss, used to optimize the model to learn pixel-level correspondences between consecutive frames, Among them, E p (i) represents the feature vector of the i-th pixel in the previous frame; E x (j) represents the feature vector of the jth pixel in the current frame; <E p (i),E x (j)>The dot product is used to calculate the similarity between two pixel feature vectors;∑ j It is the sum of all pixels j in the current frame and is used to normalize the weights.
3. The method according to claim 2, characterized in that: The step S3 further comprises: Step S31: Use a mask encoder to transform the segmentation mask Y of the first frame z , the segmentation mask Y of the previous frame p Converted to the mask feature M of the first frame z , the mask feature M of the previous frame p ; Step S32: Calculate the current frame image feature E using the multi-head cross attention mechanism layer and feedforward network x With the first frame image feature E z , previous frame image features E p Correlation diagram F z→x ,F p→x , the formula is as follows: F z→x =MHCA(E x +P x ,E z +P z ,E z +M z ); AND p→x =MHCA(E x +P x ,AND p +P p ,AND p +M p ); where F z→x represents the features from the first frame to the current frame calculated by the multi-head cross attention layer; F p→x represents the features from the previous frame to the current frame obtained by multi-head cross attention calculation, MHCA represents the multi-head attention mechanism; P z ,P p ,P x represents the spatial position embedding of the first frame, previous frame, and current frame; E x +P x Indicates adding the features of the current frame to the spatial position embedding to provide it to the multi-head cross attention layer as a query; E z +P z and E p +P p Respectively, the features of the first frame and the previous frame are added to their respective spatial position embeddings and provided to the multi-head cross attention layer as keys and values; Step S33: Through layer normalization and feedforward network, the correlation graph is merged to obtain the current frame target feature F x , the formula is as follows: F x =Norm(Norm(E x +F z→x +F p→x )+FFN(E x )) Among them, Norm represents the layer normalization operation, which is used to stabilize the training process, and FFN represents the feedforward network used to extract the deep representation of features.
4. The method according to claim 3, characterized in that: The step S4 further comprises: Step S41: The current frame target feature F x Input to the detection head to get the corresponding classification feature map Y cls ; Step S42: Based on the classification feature map Y cls , using a multi-layer convolutional neural network H θ For the classification feature map Y cls and the target feature F x Encode to obtain the enhanced mask encoding feature x cm ; Step S43: Encode the enhanced mask feature x cm Send to the mask decoder to generate the final segmentation mask Y m .
5. The method according to claim 4, characterized in that: The step S5 further comprises: Step S51: Use the mask encoder to convert the segmentation mask Y m Mapped to the mask embedding f s ; Step S52: Set the current frame target feature F x With the mask embedded f s Combined, through convolution and linear block processing, the segmentation quality score S is obtained q , the segmentation quality score S q reflects the segmentation mask Y m quality; Step S53: Calculate the segmentation quality loss function L qua , used to evaluate the quality of the segmentation mask, the formula is as follows: L qua =(S q -GT iou ) 2, Among them, GT iou is the segmentation mask Y m and the true segmentation mask Y gt The intersection ratio between them.
6. A visual target tracking system, used to implement the method according to any one of claims 1 to 5, characterized in that: Includes the following modules: An image encoder module (51) is used to extract features of an input frame and execute step S1; A self-supervised correspondence learning module (52) is used to learn the spatiotemporal correspondence relationship in the video and execute step S2; A temporal attention module (53) is used to enhance the feature representation of the current frame and execute step S3; A segmentation network module (54), used to generate a segmentation mask and execute step S4; A segmentation quality assessment module (55), used for assessing the quality of the segmentation mask, and executing step S5; The tracking decision module (56) is used to track the target according to the system output in the online tracking stage and execute step S6.
Citation Information
Cited By
Medical image focus tracking method and device, electronic equipment and storage medium
CN121391927A
