Video instance segmentation method and system based on trajectory guided memory network

Abstract

The invention discloses a video instance segmentation method and system based on a track guide memory network, and relates to the technical field of computer vision. According to the method, a trajectory-appearance joint modeling module is designed, a learnable threshold gating layer is introduced into a time sequence state modeling branch to filter a noise trajectory, and trajectory features and appearance features are fused through a mask self-attention mechanism, so that robust instance query is generated. On the basis, a time sequence perception memory updating and management module is designed, and historical features are denoised through a channel-space collaborative filtering mechanism. And dynamically updating a memory pool with a fixed size based on the appearance similarity, maintaining the feature quality of the memory pool, and introducing a time sequence position code to endow historical features with a time sequence perception capability. According to the method, high-precision segmentation and cross-frame identity association of the video instance in a complex scene are realized, and the problem of instance identity association error caused by lack of motion trail modeling and memory noise accumulation is effectively relieved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video analysis technology, and more specifically, to a method and system for segmenting video instances in complex scenes through trajectory modeling and memory management. Background Technology

[0002] Video instance segmentation (VIS) is a fundamental and core task in computer vision and video analytics. Its goal is to simultaneously achieve pixel-level detection, segmentation, and cross-frame identity association of instances of interest within a video sequence. Unlike static image segmentation, VIS must maintain temporal consistency across complex spatiotemporal dynamics. This requires algorithms not only to possess fine-grained segmentation capabilities within a single frame but also to effectively address temporal challenges such as motion blur, deformation, occlusion, and target re-identification within videos. VIS has broad application prospects in video understanding, autonomous driving, video editing, and augmented reality. In video post-processing and visual content generation technologies, video instance segmentation technology, by generating temporally consistent pixel-level instance masks, can accurately define the boundaries of dynamic foreground targets, thereby achieving refined foreground-background separation and high-quality video effects compositing. For example, in interactive streaming applications, real-time pixel-level segmentation and trajectory tracking of foreground characters enables content-aware rendering, such as anti-occlusion bullet screen technology, which automatically renders the text layer after the foreground instance layer, thereby enhancing the interactive experience without interfering with the main visual information. In special effects production, VIS supports fine-grained operations on individual instances in the video, such as video inpainting, object removal, or virtual environment compositing, significantly improving the automation level and quality of video content production.

[0003] In recent years, with the development of deep learning technology, significant progress has been made in online video instance segmentation methods based on Transformer. Among them, query propagation-based methods attempt to maintain temporal consistency by enhancing query features. For example, Lee S et al. proposed the CAVIS method (Lee, S., Seo, J., Han, K., Choi, M., Im, S. Cavis: Context-aware video instance segmentation. arXiv: 2407.03010, 2025), which combines object boundary context to generate perceptual representations. However, the above techniques mainly rely on the spatial appearance features of instances for cross-frame matching. In complex scenes where targets move rapidly, deform, or occlude each other, appearance features alone are insufficient to maintain identity consistency. Existing techniques neglect the crucial role of temporal motion trajectories (trajectory / motion patterns) in distinguishing similar targets, and the lack of effective modeling of motion state information leads to insufficient robustness of identity association in complex scenes.

[0004] To address the temporal dependencies in long video sequences, memory-based methods have been introduced to store historical information. For example, Heo M et al. proposed the GenVIS method (Heo, M., Hwang, S., Hyun, J., Kim, H., Oh, SW, Lee, JY, Kim, SJ A generalized framework for video instancesegmentation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 14623-14632), which utilizes a memory pool to store historical instance features to enhance the temporal representation of queries. Kim H et al. proposed the VISAGE method (Kim, H., Kang, J., Heo, M., Hwang, S., Oh, SW, Kim, SJ VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement. In Computer Vision – ECCV 2024, SpringerNature Switzerland, Cham, 2025: 93-109), which utilizes two independent memory pools to store instance-level appearance features and object embeddings. Although these memory-based methods improve long-term associations to some extent, their memory update and management mechanisms are inefficient. These methods typically employ simple storage strategies and assign the same weight to all historical frames when retrieving memory. This approach not only easily introduces invalid features from background noise or occlusion, leading to a decrease in feature discrimination ability, but also lacks temporal awareness of historical information, failing to adaptively adjust the contribution of historical information according to the time span, resulting in limited performance when processing videos containing complex occlusions and appearance changes. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a video instance segmentation method and system based on a trajectory-guided memory network. Using video sequences as input, the backbone network in the feature extraction module performs multi-scale feature encoding on each frame, and a pixel decoder generates frame queries containing rich semantic information. A Trajectory-Appearance Joint Modeling Module (TAM) is designed and implemented, comprising a Temporal State Modeling (TSM) branch and a Spatial Appearance Modeling (SAM) branch. The TSM branch dynamically models and denoises the spatiotemporal motion trajectory of instances through a Learnable Threshold Gating (LTG) layer, while the SAM branch aggregates the spatial appearance features of instances. The two branches fuse trajectory and appearance features through a masked self-attention mechanism, thereby improving the robustness of instance identity association in complex scenes. A Temporal-aware Memory Update and Management Module (TMUM) is also designed and implemented, which utilizes channel-spatial collaborative filtering... The Collaborative Filtering (CSCF) mechanism performs multi-dimensional denoising on instance features. It dynamically maintains the memory pool through the Appearance Similarity-based Memory Update (ASMU) strategy and enhances the temporal awareness of historical features through the Temporal Position Encoding (TPE) mechanism. The prediction head outputs the category prediction and pixel-level segmentation mask of the current frame based on the enhanced instance query.

[0006] To achieve the above objectives, the present invention provides a video instance segmentation method and system based on a trajectory-guided memory network, comprising the following:

[0007] S1. Obtain the video instance segmentation dataset and the static image dataset, and combine each image in the dataset with its corresponding segmentation label to form an image-segmentation label pair;

[0008] S2. Sample the video dataset to generate real video clips, and copy a single static image into an image sequence and apply random data augmentation to each frame of the sequence independently to simulate inter-frame motion and appearance changes, thereby temporally expanding the static image dataset into pseudo video clips, thus constructing a hybrid training dataset containing real video clips and pseudo video clips.

[0009] S3. Design a segmentation model, which includes a feature extraction module, a trajectory-appearance joint modeling module (TAM), a time-aware memory update and management module (TMUM), and a prediction head;

[0010] S4. Design a joint loss function that includes classification loss, mask prediction loss, and feature similarity loss;

[0011] S5. Using the mixed training dataset constructed in S2, calculate the loss function designed in S4, and train the segmentation model constructed in S3 using the backpropagation algorithm.

[0012] S6. Using the segmentation model trained in S5, output a target mask with cross-frame identity association.

[0013] This invention provides an end-to-end trainable neural network architecture and system for video instance segmentation, including a video sequence input module for acquiring the video sequence to be segmented. The data is then transmitted to a segmentation model, which, without requiring pre-defined instance segmentation labels, processes the video sequence using the segmentation model. The model detects and segments targets of interest. The model training module constructs a hybrid training dataset. By minimizing the classification loss Masking prediction loss and feature similarity loss joint objective function The segmentation model is then optimized and trained. The video object segmentation module, during the model inference phase, employs an online processing mode for video sequences. The first frame Perform instance detection and segmentation and initialize the memory pool. Starting from the second frame, input the current frame into the trained segmentation model sequentially, and perform video instance segmentation and identity association on subsequent frames.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] This invention addresses the aforementioned problems by employing a Trajectory-Appearance Joint Modeling (TAM) module in the video instance segmentation network to jointly model the spatiotemporal motion trajectory and spatial appearance features of instances. The TSM branch uses a learnable threshold gating layer (LTG) to adaptively denoise the trajectory features, effectively suppressing noise interference caused by rapid motion and occlusion. The SAM branch aggregates spatial appearance features and effectively fuses them with trajectory features through a mask self-attention mechanism, improving the model's ability to distinguish similar instances and maintain identity consistency in complex scenes. This enables the network to effectively solve the problem of identity association errors caused by rapid movement, deformation, and mutual occlusion of targets in video instance segmentation tasks. Building upon this foundation, a Temporally Aware Memory Update and Management (TMUM) module was further designed. This module employs a channel-spatial collaborative filtering (CSCF) mechanism to perform multi-dimensional denoising on stored features and utilizes multi-scale depthwise separable convolution to capture spatial detail information within different receptive fields, effectively removing background noise and invalid features introduced by occlusion. Furthermore, an appearance similarity-based memory update (ASMU) strategy dynamically replaces low-relevance features in the memory pool, ensuring that the memory pool always retains high-quality historical information. Finally, a temporal position encoding (TPE) mechanism adaptively adjusts the contribution of historical features based on the time span, enhancing the model's ability to perceive temporal information. This, in turn, improves the model's segmentation performance while enhancing the robustness of instance identity association in long-term temporal videos. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 Flowchart of the video instance segmentation method based on trajectory-guided memory network in this invention patent;

[0018] Figure 2 Overall structure diagram of the video instance segmentation network in this invention patent;

[0019] Figure 3 Structural diagram of the trajectory-appearance joint modeling module (TAM) in this invention patent;

[0020] Figure 4 The structure diagram of the Time-Aware Memory Update and Management Module (TMUM) in this invention patent;

[0021] Figure 5 A schematic diagram of the video instance segmentation results by the network in this invention patent. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. The invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] Figure 1 The following is a flowchart of the video instance segmentation method and system based on trajectory-guided memory network according to this invention patent:

[0024] S1, Obtain the video instance segmentation dataset and the static image dataset, along with their segmentation labels, as detailed below:

[0025] S101. Obtain a still image dataset. and its corresponding segmentation labels The static image dataset contains multiple target instances and their pixel-level segmentation mask annotations. Each static image in the dataset and its corresponding segmentation label are combined to form an image-segmentation label pair set. .

[0026] S102. Obtain the video segmentation dataset. and its corresponding segmentation labels The video segmentation labels contain pixel-level segmentation masks and cross-frame identity association information. Each video sequence and its corresponding segmentation label in the dataset are combined to form a video-segmentation label pair set. .

[0027] S2 involves sampling the video dataset to generate real video clips, and then copying a single static image into an image sequence and independently applying random data augmentation to each frame of the sequence to simulate inter-frame motion and appearance changes. This time-series augmentation of the static image dataset into pseudo-video clips creates a hybrid training dataset containing real and pseudo-video clips, specifically including the following:

[0028] S201, The set of static image-segmentation label pairs obtained in S101 Each image-segmentation label pair The process involves generating a pseudo-video sequence. Specifically, a single static image is copied into a continuous image sequence, and random data augmentation operations are independently applied to each frame of the sequence. These random data augmentation operations include random scaling, random horizontal flipping, random color jitter, and random grayscale changes. This generates a pseudo-video dataset containing inter-frame motion and appearance changes from a single static image. ,in The length of the generated sequence;

[0029] S202, The set of video-segmentation label pairs obtained in S102 Video sampling processing is performed. Specifically, consecutive frames are sampled from the same video sequence at preset sampling intervals. Each sampled frame and its corresponding segmentation label are then normalized, cropped, and subjected to affine transformation to obtain a real video dataset. ;

[0030] S203, Take the pseudo-video dataset obtained in S201 The real video dataset obtained with S202 Mix them according to a preset ratio to construct a hybrid training dataset for training the segmentation model. .

[0031] S3, design a segmentation model that includes a feature extraction module, a trajectory-appearance joint modeling module (TAM), a time-aware memory update and management module (TMUM), and a prediction head. Figure 2 The diagram shown is a structural diagram of the segmentation model system in this invention patent, as detailed below:

[0032] S301. Construct a feature extraction module, which includes a backbone network and a pixel decoder. The backbone network uses the current video frame to be segmented. As input, each frame is feature-encoded to generate a multi-scale feature map set. ,in , These represent the height and width of the image, respectively. Indicates channel dimension, Indicates the downsampling rate; the pixel decoder queries the initialization object. and multi-scale feature maps As input, it is decoded by the Transformer decoder to generate a frame query containing rich semantic information. ,in The number of queries;

[0033] S302, Construct Trajectory-Appearance Joint Modeling (TAM) module, Figure 3 The diagram shown is a TAM network structure diagram from this invention patent. This module includes a Temporal State Modeling (TSM) branch and a Spatial Appearance Modeling (SAM) branch. The TSM branch is composed of... It consists of stacked attention blocks, each of which is internally connected to a cross-attention layer, a learnable threshold gating (LTG) layer, and a self-attention layer. This branch is based on the trajectory token of the previous frame. As input, a soft mask is generated using an LTG layer to filter noisy trajectory features, thereby dynamically modeling the spatiotemporal motion pattern of the instance; the SAM branch is also composed of... It consists of stacked attention blocks, each of which is internally connected to a cross-attention layer and a masked self-attention layer. This branch queries the appearance of the previous frame. The input is the trajectory feature output from the TSM branch, which is then concatenated with the instance's own appearance features. This concatenation is performed along the channel dimension using a masked self-attention mechanism, ultimately generating an enhanced instance feature representation. The specific implementation is as follows:

[0034] S3021, Initialize track token and appearance query and construct by The structure consists of a two-branch structure composed of stacked attention blocks, where the temporal state modeling branch is used to update the trajectory token and the spatial appearance modeling branch is used to update the appearance query.

[0035] S3022, Trajectory features obtained from S3021 As a query Query the frame of the current frame

[0036] As a key Sum By aggregating the motion features of instances in the spatiotemporal dimension of the current frame through a cross-attention layer, intermediate trajectory tokens are obtained. The specific calculation process is as follows:

[0037]

[0038] in, Represents the cross-attention function. It is a learnable weight matrix.

[0039] S3023, intermediate trajectory token obtained from S3022 Input a learnable threshold gating layer (LTG), and obtain a soft mask by calculating the spatiotemporal correlation matrix between trajectory tokens and frame queries. And combined with the trajectory token from the previous moment Denoising is performed to obtain the denoised trajectory token. The specific calculation process is as follows:

[0040]

[0041]

[0042]

[0043]

[0044] Where σ is the Sigmoid function, θ is the temperature coefficient, and θ is the learnable threshold.

[0045] S3024, the denoised trajectory token obtained from S3023 Enhancement is achieved by using a self-attention layer to model the feature associations of motion trajectories between different instances, resulting in enhanced trajectory tokens. The specific calculation process is as follows:

[0046]

[0047] in, This represents the self-attention function.

[0048] S3025, Appearance query obtained from S3021 As a query Query the frame of the current frame

[0049] As a key Sum By aggregating the appearance features of instances in the spatial dimension through a cross-attention layer, appearance queries are obtained. .

[0050] S3026, Enhanced trajectory token obtained from S3024 Appearance query obtained from S3025 The layers are concatenated using a self-attention mechanism in the soft mask. Feature interaction fusion is performed within the guided high-confidence associated region to obtain the trajectory token output by this layer. and appearance query The specific calculation process is as follows:

[0051]

[0052]

[0053] S3027, Track token obtained from S3026 and appearance query Repeat steps S3022 to S3026. The next iteration will output the enhanced appearance features. Enter TMUM for storage and management.

[0054] S303. Construct a Time-Aware Memory Update and Management (TMUM) module. Figure 4 The diagram shows the TMUM module structure of this invention. This module includes a Channel-Spatial Collaborative Filtering (CSCF) mechanism, an Appearance Similarity-Based Memory Update (ASMU) strategy, and a Temporal Position Encoding (TPE) mechanism. The CSCF mechanism first uses a channel attention submodule containing max-pooling layers, average-pooling layers, and a multilayer perceptron to extract cross-channel semantic distribution. Then, it uses a spatial attention submodule containing 3×3, 5×5, and 7×7 multi-scale depth-separable convolutional layers to extract spatial details, thus performing collaborative noise reduction on the input instance features in both channel and spatial dimensions. The ASMU strategy calculates the cosine similarity between the current frame features and the memory pool features, dynamically replacing low-relevance features in a fixed-size K memory pool. The TPE mechanism maps the feature offset between the current frame and the memory frame to a position encoding using a multilayer perceptron and superimposes it onto the memory features, outputting enhanced memory features with temporal awareness capabilities. Specific implementation details are as follows:

[0055] S3031. Obtain the instance features output by the iteration of S3027. As input features, multi-dimensional feature denoising is performed through the channel-space collaborative filtering mechanism (CSCF) to achieve robust suppression of complex background interference and occlusion noise.

[0056] S3032, Input features obtained from S3031 Max pooling and global average pooling (AvgPool) are performed separately to extract the maximum response feature across channels. and global context features The concatenated features are then input into a multilayer perceptron (MLP) and a sigmoid processing layer to generate channel weight vectors. and compare it with the original features Perform Hadamard product operations to obtain channel enhancement features. The specific calculation process is as follows:

[0057]

[0058]

[0059]

[0060] in, This represents the Hadamard product.

[0061] S3033, The channel enhancement features obtained from S3032 Input is fed into a multi-branch structure containing mapping branches of the same scale and three parallel depthwise separable convolution (DWConv) branches; utilizing , , Convolutional kernels of different scales capture the relationship between local details and the global space of the target; the outputs of each branch are summed and then... Convolutional fusion yields spatial weight vectors to achieve Spatial weighting to generate denoised instance features The specific calculation process is as follows:

[0062]

[0063]

[0064] S3034, Denoising instance features obtained from S3033 Based on the appearance similarity-based update strategy (ASMU), the data is stored in a memory pool of fixed size K: if the memory pool is not full, it is updated directly; if the memory pool is full, the data is calculated... With all historical features of the library The cosine similarity is calculated, and the feature at the lowest similarity position j is replaced. The specific calculation process is as follows:

[0065]

[0066]

[0067] in, This represents the function for calculating cosine similarity. The index of the determined replacement position; This refers to the updated memory features.

[0068] S3035. The memory features updated from S3034 are enhanced through a temporal position coding (TPE) mechanism. The weights are adaptively adjusted according to the feature offset between the current frame and historical features to obtain the temporally enhanced memory features. The specific calculation process is as follows:

[0069]

[0070]

[0071] in, Indicates the current feature and the first Temporal offset between memory features; Encode the generated temporal position; This is the final output temporal enhancement feature used to guide subsequent video frame segmentation.

[0072] S304. Construct a prediction head. The prediction head takes the enhanced instance query output by the TAM module and the enhanced memory features provided by the TMUM module as input, and predicts the instance category and segmentation mask through a multilayer perceptron and convolutional layers, respectively, and finally outputs the instance segmentation mask of the current frame. and category prediction The specific implementation method is as follows:

[0073] S3041, Obtain the appearance query from the final iteration output of S3027. This is used as the input query for the prediction head. The prediction head constructs a classification branch through cascaded linear layers, performs linear transformation and normalization on the fused instance features, and calculates the probability distribution of each instance query on the preset semantic category. The specific calculation process is as follows:

[0074]

[0075] in, and Learnable weights and biases for classification branches; This represents the output category probability distribution.

[0076] S3042. Construct a mask prediction branch, utilizing the current frame pixel feature map obtained from S301. Appearance query output by S3027 Interacting with the target, the prediction head maps query features to a mask embedding space using a multilayer perceptron (MLP) and performs a dot product with the high-resolution feature map. This generates a pixel-level instance segmentation mask through dynamic convolution. The specific calculation process is as follows:

[0077]

[0078] in, For mask feature mapping layer; Use the Sigmoid activation function; This is the pixel-level mask prediction result for the current frame.

[0079] S4. Design a joint loss function that includes classification loss, mask prediction loss, and feature similarity loss to train the segmentation model, as follows:

[0080] S401. Constructing a mixed training dataset This dataset is a pseudo-video dataset obtained from S201. The real video dataset obtained with S202 Together they form; The video sequence is input into the segmentation model constructed by S3 for forward propagation, and the output contains... Prediction set of query tokens Each prediction Includes classification probability Predictive mask and the corresponding feature vectors ;

[0081] S402, Obtain the real tag set (in (Construct a prediction set for the actual number of targets).

[0082] With real tag set The binary matching cost matrix between the given elements is used; the Hungarian algorithm is then used to search for the optimal permutation that minimizes the total matching cost. Determine each real target Corresponding optimal prediction ;

[0083] S403. Based on the optimal matching pair determined in S402, calculate the global optimization objective function. The objective function consists of classification loss. Masking prediction loss and feature similarity loss The weighted summation is calculated as follows:

[0084]

[0085] in, and To balance the weighting coefficients of each loss term;

[0086] S404, Classification loss obtained from S403 The cross-entropy loss function is used to calculate and measure the predicted classification probability. With real category labels The differences between them; the mask segmentation loss obtained from S403 Binary cross-entropy loss With Dice loss Together, they form a constrained prediction mask. With the real mask The pixel-level consistency is calculated as follows:

[0087]

[0088]

[0089] in, , Indicates pixel index, For smoothing terms;

[0090] S405, Feature similarity loss obtained from S403 A contrastive learning mechanism is adopted to constrain the temporal consistency of features by calculating the feature similarity between features of the same instance at different time steps. The specific calculation process is as follows:

[0091]

[0092] in, and Querying the same instance in different frames and eigenvectors; This is an indicator function; it is 1 when two features belong to the same real instance ID, and 0 otherwise. Use the Sigmoid activation function;

[0093] S5, training the segmentation model. Using the hybrid training dataset constructed using S403. The segmentation model constructed using S3 is trained based on the classification loss constructed using S403. Masking prediction loss and feature similarity loss The global optimization objective function is used to obtain the loss value, and the stochastic gradient descent method is used to update the parameters in the segmentation model until convergence, finally obtaining the trained segmentation model.

[0094] S6 uses the segmentation model trained in S5 to output a target mask with cross-frame identity association. During the model inference phase, an online processing mode is employed. For the first frame of the video sequence, the model directly performs instance detection and segmentation and initializes the memory pool. Starting from the second frame, the model inputs the current frame's features into the trained segmentation model, uses the TAM module to propagate the trajectory tokens from the previous frame to the current frame to guide feature aggregation, and uses the TMUM module to retrieve spatiotemporal prior information from the memory pool to assist in instance segmentation and identity association in the current frame. Finally, it outputs the segmentation mask and trajectory ID for all instances in the entire video sequence. Figure 5 This is a schematic diagram of the video segmentation results of the segmentation model invented in this patent.

[0095] According to another aspect of this application, a video instance segmentation system using a trajectory-guided memory network is also provided, comprising the following modules:

[0096] Video sequence input module: Given a video sequence The input is fed into the segmentation model, which does not require any pre-defined segmentation labels for any instances. The model then detects and segments objects of interest in the video.

[0097] Model Training Module: Used to train a video instance segmentation model based on a trajectory-guided memory network. This model consists of a feature extraction module, a trajectory-appearance joint modeling module (TAM), a temporal-aware memory update and management module (TMUM), and a prediction head. A hybrid training dataset containing pseudo-video clips and real video clips is constructed. During training, the feature extraction module first extracts multi-scale features of the current frame and initializes the query; the TAM module uses the Temporal State Modeling (TSM) branch to capture the motion trajectory of instances, uses the Spatial Appearance Modeling (SAM) branch to aggregate appearance features, and generates enhanced instance queries through a mask self-attention mechanism; the TMUM module maintains a fixed-size memory pool, removes noise from stored features through the Channel-Spatial Collaborative Filtering (CSCF) mechanism, and uses Temporal Position Encoding (TPE) to enhance the temporal correlation of historical features; finally, the prediction head outputs the class and mask prediction of the current frame based on the enhanced query, and optimizes the model parameters by minimizing a joint objective function that includes classification loss, mask loss, and feature similarity loss.

[0098] Video object segmentation module: During the model inference phase, an online processing mode is adopted. For the first frame of the video sequence, the model directly performs instance detection and segmentation and initializes the memory pool. Starting from the second frame, the model inputs the features of the current frame into the trained segmentation model, uses the TAM module to propagate the trajectory tokens of the previous frame to the current frame to guide feature aggregation, and uses the TMUM module to retrieve spatiotemporal prior information in the memory pool to assist in instance segmentation and identity association of the current frame. Finally, it automatically outputs the segmentation mask and trajectory ID of all instances in the entire video sequence.

[0099] This system is used to implement the functions of the methods in the above embodiments. The specific implementation of the methods involved in the system module has been described in the methods and will not be repeated here.

[0100] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A video instance segmentation method and system based on trajectory-guided memory networks, characterized in that, The method is executed by a computer and includes the following: S1. Obtain the video instance segmentation dataset and the static image dataset, and combine each image in the dataset with its corresponding segmentation label to form an image segmentation label pair; S2. Sample the video dataset to generate real video clips, and simulate inter-frame motion and appearance changes by copying a single static image into an image sequence and applying random data augmentation to each frame of the sequence independently. This time-series expansion of the static image dataset into pseudo video clips will construct a hybrid training dataset containing real video clips and pseudo video clips. S3. Construct a segmentation model, which includes a feature extraction module, a trajectory appearance joint modeling module, a time-aware memory update and management module, and a prediction head; S4. Design a joint loss function that includes classification loss, mask prediction loss, and feature similarity loss; S5. Using the hybrid training dataset constructed in S2, calculate the loss function designed in S4, and train the segmentation model constructed in S3 using the backpropagation algorithm; S6. Using the segmentation model obtained after training in S5, output the target mask with cross-frame identity association.

2. The video instance segmentation method based on trajectory-guided memory network as described in claim 1, characterized in that, The specific implementation process of S2 is as follows: S201. Perform pseudo-video sequence generation processing on each image-segmentation label pair in the static image-segmentation label pair set, copy a single static image into a continuous image sequence, and apply random data augmentation operations independently to each frame in the sequence. The random data augmentation operations include random scaling, random horizontal flipping, random color jittering, and random grayscale changes, generating a pseudo-video dataset containing inter-frame motion and appearance changes. S202. Perform video sampling processing on the video-segmentation label pair set. Sample consecutive frames from the same video sequence according to a preset sampling interval. Normalize, crop and perform affine transformation processing on each sampled frame image and its corresponding segmentation label to obtain the real video dataset. S203. Mix the pseudo video dataset with the real video dataset according to a preset ratio to construct a hybrid training dataset.

3. The video instance segmentation method based on trajectory-guided memory network as described in claim 1, characterized in that, The trajectory-appearance joint modeling module described in S3 includes a temporal state modeling branch and a spatial appearance modeling branch, specifically including: The temporal state modeling branch consists of multiple stacked attention blocks. Each attention block is internally connected to a cross-attention layer, a learnable threshold gating layer, and a self-attention layer. This branch takes the trajectory token from the previous frame as input, aggregates the motion features of the current frame through the cross-attention layer, generates a soft mask to filter noisy trajectory features through the learnable threshold gating layer, and models the feature associations of motion trajectories between different instances through the self-attention layer. The spatial appearance modeling branch consists of multiple stacked attention blocks. Each attention block is internally connected to a cross-attention layer and a mask self-attention layer. This branch takes the appearance query from the previous frame as input and aggregates the appearance features of the current frame through the cross-attention layer. The trajectory features output by the temporal state modeling branch are concatenated with the appearance features output by the spatial appearance modeling branch. Feature fusion is performed in the high-confidence association region guided by the soft mask through the mask self-attention mechanism to generate an enhanced instance feature representation.

4. The video instance segmentation method based on trajectory-guided memory network as described in claim 1, characterized in that, The specific processing steps of the learnable threshold gating layer in the trajectory-appearance joint modeling module include: Calculate the spatiotemporal correlation matrix between intermediate trajectory tokens and frame queries, and average and aggregate the spatiotemporal correlation matrix to obtain the confidence score of each trajectory token; scale the difference between the confidence score and the learnable threshold with a temperature coefficient and map it to a soft mask using the Sigmoid function; use the soft mask to perform weighted fusion of the intermediate trajectory tokens of the current frame and the trajectory tokens of the previous frame to obtain the denoised trajectory tokens.

5. The video instance segmentation method based on trajectory-guided memory network as described in claim 1, characterized in that, The time-aware memory update and management module described in S3 includes a channel-space collaborative filtering mechanism, an appearance similarity-based memory update strategy, and time-series position encoding, specifically including: The channel-space collaborative filtering mechanism includes a channel attention submodule and a spatial attention submodule. The channel attention submodule uses max pooling layers, average pooling layers, and a multilayer perceptron to extract cross-channel semantic distribution and generate channel weight vectors. The spatial attention submodule uses multi-scale depth-separable convolutional layers to extract spatial details and generate spatial weight vectors. The instance features are weighted sequentially by the channel weight vector and the spatial weight vector to achieve collaborative denoising of the channel and spatial dimensions. The appearance similarity-based memory update strategy maintains a fixed-size memory pool. When the memory pool is not full, the denoised instance features are directly stored. When the memory pool is full, the cosine similarity between the current frame feature and all historical features in the memory pool is calculated, and the feature with the lowest similarity position is replaced. The temporal position encoding mechanism generates temporal position codes through a multilayer perceptron based on the feature offset between the current frame feature and the historical features in the memory pool, and superimposes them onto the memory features to enhance temporal perception capabilities.

6. The video instance segmentation method based on trajectory-guided memory network as described in claim 1, characterized in that, The process of constructing the joint loss function described in S4 includes: The Hungarian algorithm is used to perform binary matching between the prediction set and the true label set to determine the optimal matching pair. Based on the optimal matching pair, a global optimization objective function is calculated, consisting of a weighted sum of classification loss, feature similarity loss, and mask segmentation loss. The classification loss is calculated using the cross-entropy loss function. The mask segmentation loss is composed of binary cross-entropy loss and Dice loss. The feature similarity loss adopts a contrastive learning mechanism, which constrains the temporal consistency of features by calculating the feature similarity of homologous instances at different time steps.

7. A video instance segmentation system based on a trajectory-guided memory network, characterized in that, It includes the following modules: The video sequence input module is used to acquire the video sequence to be segmented and transmit the video frame sequence to the segmentation model. The segmentation model detects and segments the target of interest in the video without needing to pre-define instance segmentation labels. The model training module is used to construct a mixed training dataset containing pseudo video clips and real video clips, and to train a segmentation model using the mixed training dataset. The segmentation model includes a feature extraction module, a trajectory-appearance joint modeling module, a time-aware memory update and management module, and a prediction head. The training process optimizes the model parameters by minimizing a joint objective function that includes classification loss, mask prediction loss, and feature similarity loss. The video instance segmentation module is used to perform online processing during the model inference stage. For the first frame of the video sequence, it performs instance detection and segmentation and initializes the memory pool. Starting from the second frame, it inputs the current frame features into the trained segmentation model. The trajectory-appearance joint modeling module propagates the trajectory tokens of the previous frame to the current frame to guide feature aggregation. The temporal-aware memory update and management module retrieves the spatiotemporal prior information in the memory pool to assist in instance segmentation and identity association of the current frame. Finally, it outputs the segmentation mask and trajectory ID of all instances in the entire video sequence.

Images