Multi-target tracking method, system and device based on memory enhancement and motion perception

Abstract

This application discloses a multi-target tracking method, system, and apparatus based on memory enhancement and motion perception. The method receives consecutive frames of a target video sequence containing multiple targets, inputs each frame in the consecutive frames into a hierarchical image encoder, and extracts multi-scale visual feature embeddings shared by multiple targets. For each target with shared feature embeddings, target memory features and object pointers pre-stored in a preset memory bank are obtained in parallel and cross-attention calculation is performed with the feature embeddings of the current frame to generate conditional features. The conditional features corresponding to each target are received by a mask decoder, and the prediction results corresponding to each target are output. The prediction results of the current frame corresponding to each target are fused with the corresponding feature embeddings to generate new target memory features and object pointers and update them to the memory bank. Tracking can start as long as the user provides an initial point or bounding box prompt, and it maintains high stability even in target occlusion, fast movement, or complex scenes.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to multi-target tracking methods, systems and devices based on memory enhancement and motion perception. Background Technology

[0002] Multi-object tracking (MOT) is a core task in computer vision, aiming to continuously locate and identify multiple objects from video sequences. Existing methods are mostly based on detection-tracking paradigms, such as combining detectors like YOLO with Kalman filtering or correlation filtering algorithms. However, these methods have the following limitations:

[0003] 1. Poor generalization: The model needs to be retrained for new targets and cannot flexibly adapt to unknown objects.

[0004] 2. Insufficient handling of occlusion and rapid movement: The accuracy of recapture after target loss is low, especially in complex scenes where it is easily interfered with.

[0005] 3. Insufficient utilization of temporal information: Traditional methods lack effective modeling of historical frame information, leading to inconsistencies in tracking.

[0006] In recent years, segmentation models such as SAM (Segment Anything Model) have provided zero-shot segmentation capabilities, but SAM lacks temporal modeling capabilities and cannot be directly used for video tracking. Existing improvement schemes attempt to integrate motion models or memory mechanisms, but these are often inefficient and struggle to balance real-time performance and accuracy. Therefore, there is an urgent need for a method that requires no additional training, can handle multi-object tracking in real time, and possesses high robustness. Summary of the Invention

[0007] The main purpose of this application is to provide a multi-target tracking method, system and device based on memory enhancement and motion perception. The method only requires the user to provide an initial point or box prompt to start tracking, and maintains high stability in target occlusion, fast movement or complex scenes.

[0008] To achieve the above objectives, in a first aspect, this application proposes a multi-target tracking method based on memory enhancement and motion perception, characterized by comprising:

[0009] Receive consecutive frames of a target video sequence containing multiple targets, input each frame in the consecutive frames into a hierarchical image encoder, and extract multi-scale visual feature embeddings shared by multiple targets;

[0010] To share the embedded features of each target, target memory features and object pointers pre-stored in a preset memory bank are obtained in parallel and cross-attention calculation is performed with the feature embedding of the current frame to generate conditional features that incorporate temporal context information. The memory bank is configured to store the target memory features and object pointers corresponding to each target in the most recent N frames.

[0011] The mask decoder receives the conditional features corresponding to each target, and through the lightweight Transformer decoding structure, outputs the prediction results corresponding to each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction.

[0012] The memory encoder fuses the prediction results of the current frame corresponding to each target with the corresponding feature embedding to generate new target memory features and object pointers and updates them to the memory bank.

[0013] In one embodiment, prior to the method of receiving the conditional features corresponding to each target by the mask decoder, and outputting the prediction results for each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure:

[0014] If the target user provides interactive prompts in the current frame or a historical frame, the interactive prompt data, including point operations, box operations, or mask prompts, is encoded into a prompt vector through the prompt encoder.

[0015] The mask decoder is further configured to receive the conditional features and cue vectors corresponding to each target, and output the prediction results corresponding to each target, including the target segmentation mask, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure.

[0016] In one embodiment, the mask decoder is configured to support multiple mask outputs;

[0017] The target visibility state prediction is configured to determine whether the target is completely occluded or does not exist in the current frame.

[0018] In one embodiment, during the process of the mask decoder outputting the prediction result through the lightweight Transformer decoding structure:

[0019] The decoded features are gradually restored to the input image resolution using a 3-layer bilinear upsampling network.

[0020] High-resolution shallow features from the output of a hierarchical image encoder are fused by skip connections to supplement the edge spatial details of the target.

[0021] When the target scenario is a scenario where the interactive prompts are semantically ambiguous, output 3 sets of candidate segmentation masks and their corresponding confidence scores, and select the mask with the highest confidence score as the final output result.

[0022] In one embodiment, the memory bank is configured as a first-in, first-out queue;

[0023] The cross-attention calculation is performed using a cross-attention mechanism with two-dimensional rotational position encoding.

[0024] In one embodiment, before generating conditional features that incorporate temporal context information, the method further includes:

[0025] Extract the optical flow motion features of the current frame and historical frames, combine them with the historical motion state parameters of the target in the memory bank, and predict the spatial position and motion trajectory of the target in the current frame through Kalman filtering to generate prior information of target motion;

[0026] When performing the cross-attention calculation, the attention weights are weighted and guided by prior information about the target motion in order to focus on the target motion region and suppress background interference.

[0027] In one embodiment, the method for generating new target memory features and object pointers and updating them in the memory bank includes:

[0028] Based on the target visibility state prediction results and confidence scores, a differentiated memory update strategy is implemented, including:

[0029] If the target visibility status is visible and the confidence score is greater than or equal to the preset threshold, the newly generated target memory features and object pointer are stored in the memory bank of the corresponding target. If the memory bank is full, the oldest frame of memory data is removed.

[0030] If the target's visibility status is invisible or the confidence score is less than a preset threshold, the memory bank for the corresponding target is not updated, and the historical memory data is retained for target recapture.

[0031] In one embodiment, the hierarchical image encoder employs a Hiera model pre-trained based on MAE;

[0032] The hierarchical image encoder is configured to receive preprocessed video frames, perform multi-scale feature extraction processing through a four-stage hierarchical Transformer block, and generate visual feature maps at four scales. The four scale visual feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively. The deep semantic features at the 1 / 16 scale are used as the feature embedding input to subsequent modules, and the high-resolution shallow features at the 1 / 4 scale are output to the mask decoder to optimize the boundary details of the segmentation mask.

[0033] Furthermore, to achieve the above objectives, in a second aspect, this application also proposes a multi-target tracking system based on memory enhancement and motion perception, comprising:

[0034] The video stream access module is used to receive consecutive frames of a target video sequence containing multiple targets, and input each frame in the consecutive frames into a hierarchical image encoder;

[0035] A hierarchical image coding module is used to extract multi-scale visual feature embeddings shared by multiple targets;

[0036] The memory attention fusion module is used to acquire target memory features and object pointers pre-stored in a preset memory bank in parallel for each target with shared feature embedding, and to perform cross-attention calculation with the feature embedding of the current frame to generate conditional features that fuse temporal context information. The memory bank is configured to store target memory features and object pointers corresponding to each target in the most recent N frames.

[0037] The mask decoding module is used to receive the conditional features corresponding to each target from the mask decoder, and output the prediction results corresponding to each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure.

[0038] The memory update module is used to fuse the prediction results of the current frame corresponding to each target with the corresponding feature embedding through the memory encoder, generate new target memory features and object pointers, and update them to the memory bank.

[0039] Furthermore, to achieve the above objectives, in a third aspect, this application also proposes a multi-target tracking device based on memory enhancement and motion perception, the device comprising: a memory, a processor, and a multi-target tracking program based on memory enhancement and motion perception stored in the memory and executable on the processor, the multi-target tracking program based on memory enhancement and motion perception being configured to implement the steps of the method as described in any one of the first aspects.

[0040] The multi-target tracking method, system and device based on memory enhancement and motion perception provided in this application are based on large-scale basic model training, which does not require retraining for specific targets. It can segment any category of targets in the tracking video with only simple interactive prompts, and has high generalization and zero-shot capability.

[0041] By explicitly modeling long-term dependencies through memory attention mechanisms, it effectively addresses challenging scenarios such as long-term occlusion, target disappearance and reappearance, and drastic deformation, demonstrating high tracking consistency and strong robustness in long-term modeling.

[0042] By unifying video segmentation and tracking tasks into a single end-to-end model, the error accumulation and module compatibility issues of traditional detection-tracking pipelines are avoided, resulting in high computational efficiency.

[0043] Employing a high-efficiency streaming architecture and a lightweight decoder, it can achieve real-time frame rates when processing high-definition video and supports interactive corrections by users at any frame, providing a smooth experience. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a first flowchart illustrating an embodiment of the multi-target tracking method based on memory enhancement and motion perception in this application;

[0047] Figure 2 This is a second flowchart illustrating an embodiment of the multi-target tracking method based on memory enhancement and motion perception in this application;

[0048] Figure 3 This is a first structural schematic diagram of an embodiment of the multi-target tracking system based on memory enhancement and motion perception in this application;

[0049] Figure 4 This is a second structural schematic diagram provided for an embodiment of the multi-target tracking system based on memory enhancement and motion perception in this application;

[0050] Figure 5 This is a schematic diagram of a structure provided for an embodiment of the multi-target tracking device based on memory enhancement and motion perception in this application.

[0051] Explanation of icon numbers:

[0052] 10. Memory; 20. Processor.

[0053] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0055] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0056] Existing technologies cannot simultaneously achieve the goals of requiring no additional training, real-time multi-object tracking, and high robustness. To address these issues, the basic models used in this invention have all been pre-trained on large-scale general datasets, requiring no additional training from the user. Model details are as follows:

[0057] Hierarchical image encoder: The Hiera-Base model based on MAE pre-training is adopted. This model has been pre-trained with mask self-supervised training on the SA-1B large-scale segmentation dataset. It has strong zero-shot generalization ability. The input image size is 1024×1024 and it can output multi-scale feature maps of 4 scales.

[0058] Optical flow model: The RAFT model, which is pre-trained on general optical flow datasets such as FlyingChairs, Sintel, and KITTI, is used to extract dense optical flow features between frames without the need for fine-tuning for specific scenes;

[0059] The memory attention module, cue encoder, mask decoder, and memory encoder are all jointly pre-trained end-to-end on large-scale general video segmentation and tracking datasets such as YouTube-VOS, DAVIS, MOSE, and BDD100K, enabling zero-shot tracking capabilities in general scenarios without requiring additional training from the user.

[0060] During the model inference stage, TensorRT is used to perform INT8 quantization and inference graph optimization on the pre-trained model. While ensuring that the accuracy loss is less than 1%, the inference speed is further improved to meet the real-time requirements.

[0061] In addition, the terms used in this invention are explained as follows:

[0062] Deep features (salient features): 1 / 16 scale deep semantic features based on the output of the Hiera image encoder pre-trained by MAE, with dimensions of 64×64×256. They are rich in high-level semantic information of the target and are the core input features for cross-attention calculation and memory encoding.

[0063] Conditional features: After being processed by the memory attention fusion module, the feature map is fused with the target's historical temporal context information. It has the same dimension as the deep features and is the core input of the mask decoder.

[0064] Target memory features: The 64×64×64 feature tensor output by the memory encoder is used to characterize the spatial appearance features of the target in the corresponding frame. It is stored in the first-in-first-out memory and is the core carrier of long-term modeling.

[0065] Object pointer: A 64-dimensional CLS token extracted from the output layer of the mask decoder Transformer, used to encode the target's high-level semantic identity information, which corresponds one-to-one with the target's memory features and together form the historical memory set.

[0066] Two-dimensional Rotational Position Encoding (2D-RoPE): An encoding method used to overlay spatial location information onto historical memory features. It encodes the horizontal and vertical coordinates of the feature map through a rotational position encoding function, thereby enhancing the model's ability to model the spatiotemporal positional relationship of the target.

[0067] Based on this, embodiments of this application provide a multi-target tracking method based on memory enhancement and motion perception, referring to... Figure 1 This includes the following steps:

[0068] Step S1: Receive consecutive frames of a target video sequence containing multiple targets, input each frame in the consecutive frames into a hierarchical image encoder, and extract multi-scale visual feature embeddings shared by multiple targets.

[0069] Step S2: For each target with shared feature embedding, the target memory features and object pointers pre-stored in a preset memory bank are obtained in parallel, and cross-attention calculation is performed with the feature embedding of the current frame to generate conditional features that integrate temporal context information. The memory bank is configured to store the target memory features and object pointers corresponding to each target in the most recent N frames.

[0070] Step S3: The mask decoder receives the conditional features corresponding to each target, and outputs the prediction results corresponding to each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure.

[0071] Step S4: Using a memory encoder, the prediction results of the current frame corresponding to each target are fused with the corresponding feature embeddings to generate new target memory features and object pointers and update them in the memory bank.

[0072] Furthermore, in an optional implementation, prior to the method of receiving the conditional features corresponding to each target by the mask decoder, and outputting the prediction results for each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure:

[0073] If the target user provides interactive prompts in the current frame or a historical frame, the interactive prompt data, including point operations, box operations, or mask prompts, is encoded into a prompt vector through the prompt encoder.

[0074] The mask decoder is further configured to receive the conditional features and cue vectors corresponding to each target, and output the prediction results corresponding to each target, including the target segmentation mask, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure.

[0075] Specifically, the prompt encoder uses a visual feature encoding space derived from the hierarchical image encoder, and outputs a standardized prompt vector with a fixed dimension of 256. Differentiated encoding processing is performed for different types of interactive prompts.

[0076] For point operation prompts, the foreground target point clicked by the user is encoded as a positive vector and the background click point is encoded as a negative vector. The coordinate information is encoded through two linear mapping layers and a sine position encoding layer to generate point prompt vectors of the corresponding dimensions.

[0077] For bounding box operation prompts, the coordinates of the top left and bottom right corners, the center coordinates, and the width and height dimensions of the bounding box drawn by the user are extracted. Spatial position encoding is completed through the bounding box encoding layer. At the same time, average pooling is performed on the multi-scale visual features corresponding to the selected area to supplement the semantic information of the area and generate the bounding box prompt vector.

[0078] For mask cues, a lightweight convolutional encoder with a stride of 2 is used to perform progressive downsampling encoding on the binary mask image drawn by the user. Each convolutional layer is followed by batch normalization and ReLU activation function, and finally mapped to a standardized mask cue vector.

[0079] The prompt encoder allows users to input interactive prompts at any frame in the video sequence. It can provide initialization guidance for the target in the first frame or correction guidance for targets that drift during tracking. After the prompt vector is generated, it will be cached in the tracking metadata of the corresponding target. When there are no new interactive prompts, the valid prompt vectors of the historical frames will be used automatically.

[0080] This implementation method achieves zero-sample initialization and dynamic correction during tracking of any target by adapting a differentiated coding structure for various interactive prompts, including points, boxes, and masks. No pre-training or data annotation is required for the target being tracked; the tracking task can be started with just a single simple interaction, significantly lowering the barrier to entry for tracking tasks. At the same time, the standardized prompt vector space design ensures the consistency of the guiding effect of different types of prompts on the tracking task, improving the flexibility and stability of interactive tracking.

[0081] Furthermore, in an alternative implementation, the mask decoder is configured to support multiple mask outputs;

[0082] The target visibility state prediction is configured to determine whether the target is completely occluded or does not exist in the current frame.

[0083] Furthermore, during the process of the mask decoder outputting the prediction result through the lightweight Transformer decoding structure:

[0084] The decoded features are gradually restored to the input image resolution using a 3-layer bilinear upsampling network.

[0085] High-resolution shallow features from the output of a hierarchical image encoder are fused by skip connections to supplement the edge spatial details of the target.

[0086] When the target scenario is a scenario where the interactive prompts are semantically ambiguous, output 3 sets of candidate segmentation masks and their corresponding confidence scores, and select the mask with the highest confidence score as the final output result.

[0087] Specifically, the mask decoder has a built-in 4-layer lightweight Transformer decoding structure. Each decoding structure includes a multi-head self-attention layer and a cross-attention layer. The input of the decoding structure is conditional features and cue vectors, and the output is a decoding feature map with a dimension of 64×64×256.

[0088] The upsampling ratios of the three-layer bilinear upsampling network are 2x, 2x and 4x respectively, which gradually restore the decoded feature map from 1 / 16 of the size of the input image to the original input resolution. After each upsampling layer, a 1×1 convolutional layer is set to align the feature dimensions.

[0089] The skip connection performs channel-dimension concatenation and fusion of the 1 / 4-size high-resolution shallow features output by the hierarchical image encoder with the feature map after upsampling in the second layer. After fusion, the feature is purified by a convolutional layer to supplement spatial details such as the edge and texture of the target.

[0090] The mask decoder is equipped with three independent prediction heads: a segmentation mask head, an IoU prediction head, and an occlusion prediction head. The segmentation mask head outputs three sets of candidate segmentation masks with the same resolution as the input image. The IoU prediction head synchronously outputs the confidence score in the range of 0 to 1 for each set of masks. The occlusion prediction head outputs a 1×1 binary classification result, where 0 represents that the target is visible in the current frame and 1 represents that the target is completely occluded or has left the frame and does not exist.

[0091] For scenarios where interactive prompts are semantically ambiguous, the three candidate masks are automatically sorted in descending order of confidence score, and the mask with the highest confidence score is selected as the final segmentation result of the target in the current frame.

[0092] This implementation achieves high-resolution, refined reconstruction of segmentation masks by fusing shallow features through multi-scale upsampling and skip connections, solving the problem of blurred mask edges in traditional methods. Through multi-mask output and confidence optimization mechanisms, it effectively addresses segmentation errors when interactive prompts contain semantic ambiguity, improving segmentation accuracy in complex scenes. An independent occlusion prediction head enables accurate judgment of target visibility, providing a reliable basis for subsequent differentiated updates to the memory bank and a preliminary judgment foundation for recapture after target occlusion.

[0093] Furthermore, in an alternative implementation, the memory bank is configured as a first-in, first-out queue;

[0094] The cross-attention calculation is performed using a cross-attention mechanism with two-dimensional rotational position encoding.

[0095] Furthermore, before generating conditional features that incorporate temporal context information, the process also includes:

[0096] Extract the optical flow motion features of the current frame and historical frames, combine them with the historical motion state parameters of the target in the memory bank, and predict the spatial position and motion trajectory of the target in the current frame through Kalman filtering to generate prior information of target motion;

[0097] When performing the cross-attention calculation, the attention weights are weighted and guided by prior information about the target motion in order to focus on the target motion region and suppress background interference.

[0098] Specifically, the memory bank maintains an independent first-in-first-out (FIFO) circular queue for each tracked target. The queue capacity N is configurable from 5 to 30 frames, with a default value of 6 frames. Each storage unit in the queue contains the 64×64×64-dimensional spatial memory features, a 64-dimensional object pointer, a temporal position code, and target motion state parameters for the corresponding frame. When the queue is full, the oldest frame of memory data at the head of the queue is automatically removed to ensure the temporal validity of the memory bank. The two-dimensional rotational position encoding (2D-RoPE) is applied to the key vector and query vector in the cross-attention calculation, and the encoding formula is as follows:

[0099]

[0100] Where x and y are the horizontal and vertical coordinates of the pixels in the feature map, dmodel is the feature dimension, and i is the dimension index. This encoding method superimposes absolute position information on the spatial memory features, enhancing the model's ability to model the spatiotemporal positional relationship of the target.

[0101] Optical flow motion features are extracted using a RAFT optical flow model pre-trained on the FlyingChairs, Sintel, and KITTI datasets. The input is the RGB images of the current frame and the previous frame, and the output is a dense optical flow feature map of the same size as the input image, representing the lateral and longitudinal motion displacement of each pixel.

[0102] Kalman filtering uses a linear constant acceleration model, with the state vector being... Where x and y are the coordinates of the target center. These are the lateral and longitudinal velocities, respectively. The horizontal and vertical accelerations are respectively used. Based on the historical motion state parameters stored in the memory bank, the spatial position, motion range and motion trajectory of the target in the current frame are predicted, and the target motion prior information is generated.

[0103] During the cross-attention calculation process, an attention weight mask is generated based on the prior information of the target motion. The feature positions within the motion prediction region are assigned a weight coefficient of 1.2 to 2.0 times, while the weights of the background region are suppressed by 0 to 0.5 times, thus completing the weighted guidance of the attention weights.

[0104] This implementation method achieves efficient caching and dynamic updating of long-term historical information of the target through a first-in-first-out memory queue structure, avoiding redundant accumulation of invalid historical information. Through a cross-attention mechanism using two-dimensional rotational position encoding, it achieves accurate modeling of the target's spatiotemporal positional relationship, solving the problem of feature confusion between different spatial locations and improving the accuracy of temporal context fusion. By using motion prior information generated through optical flow feature extraction and Kalman filter motion prediction, it weights and guides the cross-attention calculation, achieving precise focusing on fast-moving targets, effectively suppressing interference from background features, significantly improving tracking stability in scenarios with rapid target movement and severe image shaking, and reducing the probability of losing target tracking.

[0105] Further, in an optional implementation, the method for generating new target memory features and object pointers and updating them in the memory bank includes:

[0106] Based on the target visibility state prediction results and confidence scores, a differentiated memory update strategy is implemented, including:

[0107] If the target visibility status is visible and the confidence score is greater than or equal to the preset threshold, the newly generated target memory features and object pointer are stored in the memory bank of the corresponding target. If the memory bank is full, the oldest frame of memory data is removed.

[0108] If the target's visibility status is invisible or the confidence score is less than a preset threshold, the memory bank for the corresponding target is not updated, and the historical memory data is retained for target recapture.

[0109] Specifically, the preset threshold for the confidence score can be configured in the range of 0.4 to 0.7, with a default value of 0.5. This threshold is determined through offline calibration of a large-scale video tracking dataset, balancing the effectiveness of memory updates with the resistance to interference.

[0110] The process of generating target memory features is as follows:

[0111] The feature embedding of the current frame is multiplied element-wise with the predicted segmentation mask to separate the target foreground features from the background features. After filtering the background noise, the feature embedding is encoded through a 2-layer 3×3 convolutional neural network. Batch normalization and ReLU activation function follow the convolutional layers to finally generate 64×64×64-dimensional spatial memory features.

[0112] The object pointer is extracted from the CLS token of the Transformer output layer of the mask decoder and mapped to a 64-dimensional feature vector through a single linear mapping layer, which is used to encode the high-level semantic identity information of the target.

[0113] The execution logic of the differentiated update strategy is as follows:

[0114] The newly generated memory features, object pointers, and motion state parameters are packaged and stored in the FIFO memory of the corresponding target only when the target visibility state output by the occlusion prediction head is 0 (visible) and the confidence score output by the IoU prediction head is ≥ a preset threshold.

[0115] When the target visibility status is 1 (invisible) or the confidence score is less than the preset threshold, the memory update of the corresponding target is paused, and the historical memory data is fully preserved until the target reappears and the tracking quality recovers to above the threshold, and then the memory update is continued.

[0116] This implementation avoids the pollution of the memory bank by invalid memories when the target is occluded or the tracking quality is poor, by adopting a differentiated memory update strategy based on target visibility and mask confidence. This ensures that the features stored in the memory bank are all valid appearance and semantic information of the target. The first-in-first-out memory elimination mechanism ensures the temporal freshness of the memory bank while retaining the long-term temporal history information of the target. When the target reappears after a long period of occlusion, the retained historical memory can be used to quickly complete the target's identity matching and tracking recovery, which greatly improves the target recapture accuracy and tracking consistency in occluded scenarios.

[0117] Furthermore, in an optional implementation, the hierarchical image encoder employs a Hiera model pre-trained based on MAE.

[0118] The hierarchical image encoder is configured to receive preprocessed video frames, perform multi-scale feature extraction processing through a four-stage hierarchical Transformer block, and generate visual feature maps at four scales. The four scale visual feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively. The deep semantic features at the 1 / 16 scale are used as the feature embedding input to subsequent modules, and the high-resolution shallow features at the 1 / 4 scale are output to the mask decoder to optimize the boundary details of the segmentation mask.

[0119] Specifically, the hierarchical image encoder uses a Hiera-Base model pre-trained on the SA-1B large-scale segmentation dataset based on MAE (masked autoencoder). The input image is a pre-processed 1024×1024×3 RGB format image. The preprocessing operations include size scaling, aspect ratio preservation padding, and pixel value normalization to the [-1,1] range.

[0120] The Hiera model consists of four stages of hierarchical Transformer blocks, each containing multiple Transformer units. Each stage reduces the spatial size of the feature map and increases the feature channel dimension through pooling operations. The first stage outputs high-resolution shallow features at 1 / 4 of the input size with a channel dimension of 64. The second stage outputs medium-resolution features at 1 / 8 of the input size with a channel dimension of 128. The third stage outputs deep semantic features at 1 / 16 of the input size with a channel dimension of 256. The fourth stage outputs global features at 1 / 32 of the input size with a channel dimension of 512.

[0121] The deep semantic features at the 1 / 16 scale are flattened into a 4096×384 token sequence, which serves as the input memory attention module for the query vector and as the base input memory encoder for foreground feature fusion.

[0122] High-resolution shallow features at the 1 / 4 scale are fed into the upsampled network of the input mask decoder via skip connections to supplement the edge texture and spatial detail information of the segmentation mask.

[0123] This implementation method achieves accurate multi-scale feature extraction of input images by using the Hiera hierarchical visual Transformer model pre-trained based on MAE, taking into account both high-level semantic information and shallow spatial details of the target. The pre-training based on a large-scale dataset ensures the model's strong zero-shot generalization ability, and feature extraction of any target can be achieved without retraining for a specific target category.

[0124] By sharing the feature extraction results of the same image encoder among multiple targets, only one image encoding is needed to provide feature input for all tracked targets, avoiding redundant calculations in multi-target scenarios, significantly reducing computing power consumption, and improving the processing speed and real-time performance of multi-target tracking.

[0125] By outputting features at different scales in a hierarchical manner, the semantic requirements of temporal context fusion and the detailed requirements of mask segmentation are respectively adapted, thus achieving efficient utilization of feature resources.

[0126] like Figures 2 to 3 As shown, this application also provides an embodiment of a multi-target tracking system based on memory enhancement and motion perception, including:

[0127] The video stream access module is used to receive consecutive frames of a target video sequence containing multiple targets, and input each frame in the consecutive frames into a hierarchical image encoder;

[0128] A hierarchical image coding module is used to extract multi-scale visual feature embeddings shared by multiple targets;

[0129] The memory attention fusion module is used to acquire target memory features and object pointers pre-stored in a preset memory bank in parallel for each target with shared feature embedding, and to perform cross-attention calculation with the feature embedding of the current frame to generate conditional features that fuse temporal context information. The memory bank is configured to store target memory features and object pointers corresponding to each target in the most recent N frames.

[0130] The mask decoding module is used to receive the conditional features corresponding to each target from the mask decoder, and output the prediction results corresponding to each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure.

[0131] The memory update module is used to fuse the prediction results of the current frame corresponding to each target with the corresponding feature embedding through the memory encoder, generate new target memory features and object pointers, and update them to the memory bank.

[0132] For details, see Figure 4 Using this implementation method, the complete execution flow of the multi-target tracking method includes the following steps:

[0133] Step 1: The process begins with loading the pre-trained model. After the system starts, it loads the pre-trained model weights and initializes the inference engine. The loaded model includes the complete pre-trained weights of the image encoder, cue encoder, memory attention module, mask decoder, and memory encoder. TensorRT is used to optimize the inference graph and perform INT8 quantization to verify the consistency of the input and output dimensions of each module, ensuring a smooth model inference chain. All models loaded in this stage are pre-trained weights for general scenarios, requiring no training or fine-tuning by the user for the current tracking task.

[0134] Step 2: Initialize the memory bank to be empty or for existing targets. Based on the user's tracking task requirements, perform the memory bank initialization operation: If it is a brand new tracking task, initialize an empty first-in-first-out (FIFO) memory bank for each target to be tracked, and configure the memory bank capacity N (default value is 6, configurable range 5~30); if it is a continued tracking task, load the historical memory bank, tracking ID and motion state parameters of existing targets to complete the breakpoint continuation of the tracking task; at the same time, initialize the GPUCUDA stream resources for multi-target parallel processing, and allocate an independent computing stream for each target.

[0135] Step 3: Video Stream End Determination. The system continuously checks whether the input video stream has been completely read: If the video stream has ended, it jumps to the end processing stage to complete the archiving and storage of the tracking results and release system resources; if the video stream has not ended, it proceeds to the next step.

[0136] Step 4: Read the next frame. Read the next frame of the video sequence in a streaming manner. Decode the video frame using the FFmpeg hardware decoding component and output the original image in RGB format. Assign a globally unique frame number to each frame and maintain the frame timing sequence and historical frame buffer.

[0137] Step 5: Image encoding and multi-scale feature extraction. The decoded current frame image is preprocessed, including scaling to 1024×1024, pixel value normalization, and channel alignment. It is then input into the image encoder to perform multi-scale visual feature extraction. Through the four-stage hierarchical Transformer blocks of the Hiera model, feature maps at four scales are output. The deep features at the 1 / 16 scale are used as the main features input to subsequent modules, and the high-resolution shallow features at the 1 / 4 scale are output to the mask decoder. All features are shared by multiple targets and only one encoding calculation is performed.

[0138] Step 6: Determine if there is a new target or if correction is needed. The system determines whether there is a new tracking target in the current frame, or if there is an existing target that needs user interaction correction. If there is a new target or correction is needed, the user interaction and prompt encoding process is executed. If there is an existing target and no correction is needed, the historical memory reading process is executed.

[0139] Step 7 (Branch 1, corresponding to new target / scene requiring correction): User interaction prompts (point / box / mask) receive user interaction operations in the current frame. It supports three zero-shot interaction forms: point operation of clicking foreground / background points, box operation of drawing target boundary, and mask prompt of outlining target contour. It provides initialization information for new targets or correction information for tracking drifting existing targets. No annotation or training is required. Tracking can be started / corrected with only a single interaction.

[0140] Step 8: Cue Encoding. Input the user's interactive cues into the cue encoder, perform differential encoding processing, generate a standardized 256-dimensional cue vector, and output it to the memory attention module and mask decoder.

[0141] Step 9 (Branch 2, corresponding to existing target no need to modify the scene): Read the historical memory features and object pointers from the memory bank. Read the complete historical memory set from the FIFO memory bank of the corresponding target, including the spatial memory features of the last N frames, high-level semantic object pointers, and time position encoding information, to provide temporal context data for memory attention fusion.

[0142] Step 10: The memory attention module integrates current features, historical memory, and cue vectors. The main features of the current frame, historical memory features, object pointers, and cue vectors (new target / corrected scene) are input into the memory attention module. Two-dimensional rotation position encoding is superimposed on the spatial memory features. Cross-attention fusion is performed in combination with motion prior information to generate conditional features that integrate temporal context information, which are then output to the mask decoder.

[0143] Step 11: The mask decoder receives conditional features, high-resolution features, and cue vectors. The conditional features output by the memory attention module, the high-resolution shallow features output by the image encoder, and the cue vector output by the cue encoder are synchronously input into the mask decoder. The decoder first performs skip connection fusion of the low-resolution conditional features and the high-resolution shallow features through the multi-scale feature fusion upsampling module to supplement the edge space details of the target and generate a high-resolution decoded feature map.

[0144] Step 12: Output three prediction results. The mask decoder outputs three prediction results simultaneously through three independent prediction heads: segmentation mask, confidence score, and occlusion status. The segmentation mask head outputs three sets of candidate segmentation masks, the IoU prediction head outputs the confidence score corresponding to each set of masks, and the occlusion prediction head outputs the visibility status of the target. Finally, the mask with the highest confidence score is selected as the final segmentation mask of the target in the current frame.

[0145] Step 13: Memory Encoding and Update, Generating New Memory Features and Object Pointers. Input the main features of the current frame and the predicted segmentation mask into the memory encoder to perform new memory generation: perform element-wise multiplication of the main features and the segmentation mask to separate the target foreground features, and generate new spatial memory features through a convolutional neural network; at the same time, extract the CLS token from the output layer of the mask decoder and map it to the object pointer to complete the encoding of the target's high-level semantic features.

[0146] Step 14: Update the memory bank. Following the FIFO strategy, perform differentiated memory bank updates based on the target's occlusion status and confidence score. If the target is visible and the confidence score is greater than or equal to the preset threshold, store the newly generated memory features and object pointers into the corresponding target's FIFO memory bank. If the memory bank is full, remove the oldest frame of memory data. If the target is not visible or the confidence score is less than the preset threshold, do not update the memory bank, and retain historical memory data for recapture after the target occlusion ends.

[0147] Step 15: Output the segmentation and tracking results of the current frame. Output the segmentation mask, tracking ID, confidence score, occlusion status, and motion trajectory information of all targets in the current frame as the final segmentation and tracking results. At the same time, overlay the segmentation mask onto the original video frame to complete the visualization rendering, which supports real-time display and local storage.

[0148] Step 16: Whether to process multiple targets. The system determines whether the current tracking task contains multiple targets: if it is only single target tracking, it directly prepares to process the next frame and jumps to the video stream end judgment stage in Step 3; if it is multi-target tracking, it executes the multi-target parallel processing flow.

[0149] Step 17: Execute the above process in parallel for each target, sharing image features. Based on the multi-target processing mechanism, perform multi-target parallel scheduling: all tracked targets share the multi-scale features output by the image encoder to avoid redundant encoding calculations; allocate an independent CUDA stream for each target on the GPU, and execute the complete tracking process from Step 9 to Step 15 in parallel for each target through a single instruction multi-threaded architecture; maintain an independent FIFO memory bank, tracking ID, and motion state parameters for each target to achieve synchronous parallel tracking of multiple targets.

[0150] Step 18: Prepare to process the next frame. After completing the tracking and processing of all targets in the current frame, jump to the video stream end judgment step in step 3 and enter the processing loop of the next frame until the entire video stream is processed.

[0151] Step 19: End Processing. After the video stream has been read, the end processing flow is executed: archive and store all tracking results, synthesize and export the tracking video, release GPU memory and system memory resources, and complete the entire multi-target tracking task.

[0152] like Figure 5 As shown, this application also provides an embodiment of a multi-target tracking device based on memory enhancement and motion perception. The device includes: a memory, a processor, and a multi-target tracking program based on memory enhancement and motion perception stored in the memory and executable on the processor. The multi-target tracking program based on memory enhancement and motion perception is configured to implement the steps of the method as described in any of the above embodiments.

[0153] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A multi-target tracking method based on memory-augmented and motion-aware, characterized in that, include: Receive consecutive frames of a target video sequence containing multiple targets, input each frame in the consecutive frames into a hierarchical image encoder, and extract multi-scale visual feature embeddings shared by multiple targets; To share the embedded features of each target, target memory features and object pointers pre-stored in a preset memory bank are obtained in parallel and cross-attention calculation is performed with the feature embedding of the current frame to generate conditional features that incorporate temporal context information. The memory bank is configured to store the target memory features and object pointers corresponding to each target in the most recent N frames. The mask decoder receives the conditional features corresponding to each target, and through the lightweight Transformer decoding structure, outputs the prediction results corresponding to each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction. The memory encoder fuses the prediction results of the current frame corresponding to each target with the corresponding feature embedding to generate new target memory features and object pointers and updates them to the memory bank.

2. The multi-target tracking method based on memory enhancement and motion perception as described in claim 1, characterized in that: Before the method of receiving the conditional features corresponding to each target by the mask decoder, and outputting the prediction results for each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure: If the target user provides interactive prompts in the current frame or a historical frame, the interactive prompt data, including point operations, box operations, or mask prompts, is encoded into a prompt vector through the prompt encoder. The mask decoder is further configured to receive the conditional features and cue vectors corresponding to each target, and output the prediction results corresponding to each target, including the target segmentation mask, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure.

3. The multi-target tracking method based on memory enhancement and motion perception as described in claim 2, characterized in that: The mask decoder is configured to support multiple mask outputs; The target visibility state prediction is configured to determine whether the target is completely occluded or does not exist in the current frame.

4. The multi-target tracking method based on memory enhancement and motion perception as described in claim 3, characterized in that, During the process of the mask decoder outputting the prediction result through the lightweight Transformer decoding structure: The decoded features are gradually restored to the input image resolution using a 3-layer bilinear upsampling network. By fusing high-resolution shallow features from the output of the hierarchical image encoder through skip connections, the edge spatial details of the target are supplemented. When the target scenario is a scenario where the interactive prompts are semantically ambiguous, output 3 sets of candidate segmentation masks and their corresponding confidence scores, and select the mask with the highest confidence score as the final output result.

5. The multi-target tracking method based on memory enhancement and motion perception as described in claim 2, characterized in that: The memory bank is configured as a first-in, first-out queue; The cross-attention calculation is performed using a cross-attention mechanism with two-dimensional rotational position encoding.

6. The multi-target tracking method based on memory enhancement and motion perception as described in claim 5, characterized in that: Before generating conditional features that incorporate temporal context information, the following steps are also included: Extract the optical flow motion features of the current frame and historical frames, combine them with the historical motion state parameters of the target in the memory bank, and predict the spatial position and motion trajectory of the target in the current frame through Kalman filtering to generate prior information of target motion; When performing the cross-attention calculation, the attention weights are weighted and guided by prior information about the target motion in order to focus on the target motion region and suppress background interference.

7. The multi-target tracking method based on memory enhancement and motion perception as described in claim 1, characterized in that, The method for generating new target memory features and object pointers and updating them in the memory bank includes: Based on the target visibility state prediction results and confidence scores, a differentiated memory update strategy is implemented, including: If the target visibility status is visible and the confidence score is greater than or equal to the preset threshold, the newly generated target memory features and object pointer are stored in the memory bank of the corresponding target. If the memory bank is full, the oldest frame of memory data is removed. If the target's visibility status is invisible or the confidence score is less than a preset threshold, the memory bank for the corresponding target is not updated, and the historical memory data is retained for target recapture.

8. The multi-target tracking method based on memory enhancement and motion perception as described in claim 1, characterized in that: The layered image encoder uses a Hiera model pre-trained based on MAE; The hierarchical image encoder is configured to receive preprocessed video frames, perform multi-scale feature extraction processing through a four-stage hierarchical Transformer block, and generate visual feature maps at four scales. The four scale visual feature maps are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively. The deep semantic features at the 1 / 16 scale are used as the feature embedding input to subsequent modules, and the high-resolution shallow features at the 1 / 4 scale are output to the mask decoder to optimize the boundary details of the segmentation mask.

9. A multi-target tracking system based on memory enhancement and motion perception, characterized in that, include: The video stream access module is used to receive consecutive frames of a target video sequence containing multiple targets, and input each frame in the consecutive frames into a hierarchical image encoder; A hierarchical image coding module is used to extract multi-scale visual feature embeddings shared by multiple targets; The memory attention fusion module is used to acquire target memory features and object pointers pre-stored in a preset memory bank in parallel for each target with shared feature embedding, and to perform cross-attention calculation with the feature embedding of the current frame to generate conditional features that fuse temporal context information. The memory bank is configured to store target memory features and object pointers corresponding to each target in the most recent N frames. The mask decoding module is used to receive the conditional features corresponding to each target from the mask decoder, and output the prediction results corresponding to each target, including the target segmentation mask of the current frame, confidence score, and target visibility state prediction, through a lightweight Transformer decoding structure. The memory update module is used to fuse the prediction results of the current frame corresponding to each target with the corresponding feature embedding through the memory encoder, generate new target memory features and object pointers, and update them to the memory bank.

10. A multi-target tracking device based on memory enhancement and motion perception, characterized in that, The apparatus includes: a memory, a processor, and a memory-enhanced and motion-aware multi-target tracking program stored in the memory and executable on the processor, the memory-enhanced and motion-aware multi-target tracking program being configured to implement the steps of the method as described in any one of claims 1 to 8.

Images