Video object segmentation method based on query adaptive attention and discriminative memory

Abstract

The invention belongs to the technical field of computer vision and digital video processing, and discloses a video object segmentation method based on query adaptive attention and discriminative memory, which comprises the following steps: step 1, constructing a video data set and preprocessing data; step 2, constructing a video object segmentation model combining multi-scale semantic feature integration and query adaptive discriminant enhancement, and training the video object segmentation model; step 3, video object segmentation reasoning; performing target segmentation reasoning on the preprocessed video sequence, outputting a frame-by-frame target segmentation mask, and generating a time sequence tracking result; 4, exporting, deploying and applying the model; through lightweight network design and a discriminative memory optimization strategy, the model is deployed to edge equipment, and real-time segmentation and visualization are realized. According to the method, on one hand, comprehensive target representation is provided on multi-scale feature extraction, and on the basis of the characteristic that only high-confidence target features are stored, error propagation is avoided, and the long-term segmentation stability of the method is ensured.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and digital video processing technology, specifically relating to a video object segmentation method based on query adaptive attention and discriminative memory. Background Technology

[0002] In existing technologies, early VOS methods heavily relied on online learning, using the first frame's annotations to fine-tune the model during the inference phase. While achieving some accuracy, this resulted in high computational costs and slow inference speeds, hindering practical applications. Subsequent propagation-based methods utilized attention mechanisms to establish temporal associations between frames, transmitting segmentation information offline. While improving efficiency, this led to a decline in segmentation quality due to error accumulation. In recent years, matching-based methods have become mainstream. These methods identify targets through feature matching, typically using an external memory to store historical frame information and retrieving it using attention mechanisms. Representative models like XMem retrieve relevant information from the memory through global cross-attention, achieving leading performance on multiple benchmarks. Simultaneously, inspired by DETR, query vector-based Transformer methods have been introduced into VOS, encoding targets as learnable queries and achieving accurate decoding through query-pixel interactions. Methods like Cutie significantly improve segmentation accuracy. Furthermore, the SegmentAnything Model (SAM) and its video versions, such as SAM2, have further propelled the development of cue-enabled segmentation, utilizing streaming memory Transformers to achieve efficient and high-quality video segmentation. Despite significant advancements in existing technologies, Transformer methods based on global memory matching and target queries still have obvious limitations, such as low memory retrieval efficiency and limited query representation capabilities, which restrict their practical application in highly dynamic, multi-target complex scenarios.

[0003] Global memory-based matching methods suffer from high computational complexity and are susceptible to background noise. Their global attention mechanism requires global similarity calculations for all features in the memory, causing computational costs to increase linearly or even quadratically with the memory size, becoming an efficiency bottleneck when processing long videos or high-resolution data. More importantly, global matching easily introduces significant background noise. When the target's proportion is small, the attention map is easily contaminated by the background, and the model often incorrectly focuses on distractors that resemble the target, leading to missegmentation. Furthermore, historical erroneous predictions stored in the memory are retrieved in subsequent frames, causing errors to accumulate and propagate over time, severely compromising long-term segmentation stability.

[0004] The Transformer method based on target queries mainly faces the problems of query drift and attention dispersion. During inter-frame transmission, the query vector is prone to semantic drift when encountering severe occlusion, rapid deformation, or drastic changes in appearance, gradually deviating from the initial target representation and ultimately leading to tracking loss. Simultaneously, its cross-attention mechanism remains content-oriented; when target-like interference exists, the query may incorrectly assign high attention weights, diverting focus from the true target. Although local window attention can alleviate computational burden and background interference, fixed windows also limit the receptive field, making it difficult to capture global context and long-range motion dependencies, easily leading to missing target parts and poor segmentation performance for fast-moving or large-scale objects. Summary of the Invention

[0005] The purpose of this invention is to provide a video object segmentation method based on query adaptive attention and discriminative memory, so as to solve the technical problems of high computational complexity, decreased segmentation accuracy over long time and susceptibility to similar interference objects in the prior art.

[0006] The video object segmentation method based on query adaptive attention and discriminative memory includes the following steps: Step 1: Video dataset construction and data preprocessing; Step 2: Construct and train a video object segmentation model that combines multi-scale semantic feature integration with query adaptive discriminative enhancement; Step 3: Video object segmentation inference; Perform target segmentation inference on the preprocessed video sequence, output frame-by-frame target segmentation mask, and generate temporal tracking results; Step 4: Model export, deployment and application; through lightweight network design and discriminative memory optimization strategy, the model is deployed to edge devices to achieve real-time segmentation and visualization.

[0007] Preferably, step 2 includes the following steps: Step 2.1: Multi-scale feature extraction; specifically, this involves associating pre-trained VIT features with global semantic features and local spatial features to provide a comprehensive target representation; Step 2.2: The query initialization mechanism based on discriminative memory update maintains a feature memory and a query memory, storing only high-confidence target features to avoid error propagation; the initial query of the current frame is generated by fusing historical queries and current features. Step 2.3: Query adaptive deformable attention mechanism; using the refined query as the global semantic vector, guide the offset prediction network to adaptively generate sampling point positions, and perform content-aware sampling and aggregation on the relevance feature map to obtain the enhanced output features; Step 2.4: Lightweight Transformer encoder and classification / decoding head process the enhanced output features to finally obtain feature maps for mask decoding and classification labels for target / object states; Step 2.5: Set the model training parameters and complete the training.

[0008] Preferably, step 2.1 includes the following steps: Step 2.1.1: Extract three-level multi-scale features using an encoder; Step 2.1.2: Reduce the number of channels in each layer to a uniform dimension using 1×1 convolution, and perform bilinear upsampling and element-wise summation to obtain multi-scale features for downstream applications. ; Step 2.1.3: Global semantic prior injection, with foreground prototype and background prototype For querying, with Using keys and values, a semantically aware feature map is generated through a cross-attention mechanism. ; Step 2.1.4: Adaptive Spatial Refinement: To handle complex structural targets, deformable convolutions are used on the feature maps. Further refinement is needed; for each spatial location Output features It is obtained by weighted summation of features at a set of adaptively predicted offset points around it; Step 2.1.5: Obtain the final features that integrate multi-scale contextual information through residual connections. .

[0009] Preferably, step 2.2 includes the following steps: Step 2.2.1: Maintain a dual-memory structure: The dual-memory structure includes a feature memory and a query memory. The feature memory stores the pixel features of historical frames, and the query memory stores historical query vectors. Step 2.2.2: Discriminative Query Initialization: The initial query vector for the current frame is formed by fusing two parts: one part is generated by matching the features of the current frame with the memory database, and the other part is the query updated in the previous frame.

[0010] Step 2.2.3: Memory Retrieval and Relevance Calculation: Given the multi-scale features of the current frame Memory feature set M, calculate correlation feature map: , in, This is a correlation feature map. , and These represent the linear projection matrix of the query vector, the linear projection matrix of the key vector, and the vector dimension, respectively. Step 2.2.4: Query Refinement: Based on Relevance Feature Map The refined global query is obtained by aggregating information from memory and updating the query. .

[0011] Preferably, in step 2.2.4, under the constraint of the prediction mask m, the correlation feature map is used as the basis for... Find and Extract the most relevant feature regions Subsequently, the query vector is refined using a residual update formula; where, This indicates element-wise multiplication. It is a learnable scaling factor used to balance update intensity. It is a learnable projection matrix. Global query. The residual update formula is as follows:

[0012] Preferably, step 2.3 includes the following steps: Step 2.3.1: Offset Prediction: The refined global query... After pooling, a global query embedding is obtained. For correlation feature maps Each spatial location on , and its feature vector and Concatenate the data, input a lightweight offset prediction network, and predict a set of... Each sampling offset ; Step 2.3.2: Feature Sampling and Weighted Aggregation: Features are sampled at the predicted offset position using bilinear interpolation, and attention weights are calculated based on the query conditions to output position-enhanced features. ; Step 2.3.3: Residuals and Normalization: ... The residuals are connected to the original features and processed by LayerNorm / Conv before being fed into the downstream encoder to maintain information flow and gradient stability.

[0013] Preferably, in step 2.3.2, bilinear interpolation is used from... From the sampling features, a set of sampling features is obtained. Use the refined query As the query vector, the sampled features are subjected to attention-weighted aggregation to obtain the final enhanced output features. The calculation formula is as follows:

[0014] Where p represents the semantically aware feature map p position on and This is the linear projection matrix corresponding to the query vector and the value vector, where d is the dimension of the vector; each spatial position... eigenvectors Forming the final enhanced output features .

[0015] Preferably, step 2.4 includes the following steps: Step 2.4.1: Input the output features enhanced by adaptive attention into the lightweight encoder after spatial location or tokenization, and add positional encoding; Step 2.4.2: Use grouped multi-head self-attention, divide the multi-head into 4 groups and share partial projection between groups to reduce computational and memory overhead; Step 2.4.3: The encoder layer uses residual connections with LayerNorm to ultimately output feature maps for mask decoding and classification labels for target / object states. ; The mask decoding head consists of several upsampled convolutional layers and pixel-by-pixel classification layers, with classification labels. Used for global target confidence estimation or categorical regression.

[0016] Preferably, step 2.5 includes: setting hyperparameters, setting the optimizer and decay strategy, setting the loss function, and training, wherein the learning rate adopts a multinomial or piecewise decay strategy, and the loss function adopts a weighted binary cross-entropy loss. Weighted IoU (or Dice) loss The weighted sum; training employs online sampling and data augmentation, evaluating and saving the best model snapshot on the validation set every few iterations.

[0017] Preferably, step 3 includes the following steps: Step 3.1: Input preprocessing: Scale and normalize the video frames to be segmented by the shorter side, and form a frame sequence according to the sampling strategy during training for model input; Step 3.2: Memory Initialization: Initialize the memory bank using the actual mask of the first frame; Step 3.3: Forward inference: Perform feature extraction frame by frame, then memory retrieval and query refinement, enhance features through query adaptive deformable attention and output features, and finally perform mask decoding to output frame-by-frame segmentation mask and target confidence.

[0018] Step 3.4: Post-processing and visualization: Binarize the prediction mask by threshold; apply temporal smoothing or small connected component filtering to low-confidence predictions or discontinuous regions; generate a target temporal heatmap or overlay visualization video; update features / queries into the memory bank only when the confidence of the current frame reaches the threshold to maintain memory quality.

[0019] The technical advantages of this invention are as follows: This invention combines multi-scale semantic feature integration with query adaptive discriminative enhancement. On the one hand, it provides a comprehensive target representation in multi-scale feature extraction. On the other hand, based on the characteristic of storing only high-confidence target features, it avoids error propagation and ensures the long-term segmentation stability of this method.

[0020] The query-adaptive deformable attention mechanism employed in this invention uses the refined query as a global semantic vector to guide the offset prediction network in adaptively generating sampling point positions. Content-aware sampling and aggregation are then performed on the relevance feature map to obtain enhanced target features. This improves query representation capabilities while minimizing computational overhead, balancing efficiency and segmentation performance. Furthermore, this scheme ensures minimal impact from semantic drift, avoiding tracking loss, while adaptively deforming the attention mechanism to effectively capture global context and long-range motion dependencies, preventing target fragment loss.

[0021] This invention also performs global semantic prior injection, introduces foreground and background prototypes, uses the foreground and background prototypes as queries, and uses fused multi-scale features as keys and values ​​to generate semantic awareness through a cross-attention mechanism, reducing the influence of background noise and improving the accuracy of target detection and segmentation, and reducing the probability of missegmenting interference objects in the background that are similar to the target. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a video object segmentation method based on query adaptive attention and discriminative memory according to the present invention.

[0023] Figure 2 This is a flowchart illustrating the implementation of step 2.1 of the semantic space dependency module in this invention.

[0024] Figure 3 This is a flowchart of the STCN algorithm in the prior art.

[0025] Figure 4 This is a flowchart of the XMem algorithm in the prior art.

[0026] Figure 5 A graph showing the performance degradation of long video sequences compared to the solution of this invention and other existing technologies.

[0027] Figure 6This is a comparison chart showing the average performance of the solution of this invention and other existing technologies at different sequence lengths. Detailed Implementation

[0028] The following detailed description of the embodiments, with reference to the accompanying drawings, will further illustrate the specific implementation of the present invention, in order to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0029] like Figures 1-6 As shown, the present invention provides a video object segmentation method based on query adaptive attention and discriminative memory, which includes the following steps.

[0030] Step 1: Video dataset construction and data preprocessing; including video frame image acquisition, image scaling and normalization, target mask annotation, and partitioning of training, validation, and test sets. This step includes the following steps: Step 1.1: Video Acquisition and Formatting: Acquire a video sequence of the target scene (resolution based on 480 pixels on the shorter side), and divide the original video into frames and save them as an RGB image sequence.

[0031] Step 1.2: Image scaling and normalization: Scale the shorter side of each frame to the set pixel size (480 pixels), and scale the longer side proportionally; then normalize the pixels according to the mean and variance of ImageNet.

[0032] Step 1.3: Labeling and Truth Generation: Provide an accurate segmentation mask for the target in the first frame and store it as a binary mask file or RLE encoded file for each frame.

[0033] Step 1.4: Training Sample Construction: Construct training samples according to a frame sampling strategy. For example, each sample contains T=5 frames, selecting 3 frames as reference memory and 2 frames as query / prediction. Then, divide the training samples into training set, validation set, and test set in a 7:2:1 ratio to ensure a balanced distribution of sequences and target categories.

[0034] Step 2: Construct and train a video object segmentation model that combines multi-scale semantic feature integration with query adaptive discriminative enhancement; including the following steps: Step 2.1: Multi-scale feature extraction (direct extraction using a regular encoder); specifically, this involves associating pre-trained VIT features with global semantic features and local spatial features to provide a comprehensive target representation. This step is implemented by the semantic space dependency module and includes the following steps: Step 2.1.1: Use ResNet-50 (or equivalent lightweight backbone) pre-trained on ImageNet as the encoder to extract three levels of multi-scale features (corresponding to downsampled 8×, 16×, and 32× outputs respectively).

[0035] Step 2.1.2: Reduce the number of channels in each layer to a uniform dimension (e.g., 128) using 1×1 convolution, and perform bilinear upsampling and element-wise summation to obtain multi-scale features for downstream applications. .

[0036] Step 2.1.3: Global semantic prior injection. This step introduces two learnable global prototype vectors, namely the foreground prototype. and background prototype ; with foreground prototype and background prototype For querying, with Using keys and values, a semantically aware feature map is generated through a cross-attention mechanism. ;in , , Generate feature maps from learnable linear projection matrices. The formula is as follows: , in, Indicates the dimension of the vector.

[0037] Step 2.1.4: Adaptive Spatial Refinement: To handle complex structural targets, deformable convolutions are used on the feature maps. Further refinement is needed. For each spatial location... Output features It is obtained by weighted summation of features at a set of adaptively predicted offset points around it. Wherein, It is the sampling offset predicted from local features by a lightweight network. These are the corresponding attention weights, used to generate output features. The formula is as follows: .

[0038] Step 2.1.5: Obtain the final features that integrate multi-scale contextual information through residual connections. The corresponding calculation formula is as follows: , in, Feature maps are formed from the output features at each spatial location. This indicates processing via convolutional blocks.

[0039] Step 2.2: Discriminative Memory and Query Initialization; including the following steps: Step 2.2.1: Maintain a dual-memory structure: The dual-memory structure includes a feature memory and a query memory. The feature memory stores the pixel features of historical frames, and the query memory stores historical query vectors. Both use a fixed-capacity FIFO (e.g., capacity of 10). Features / queries are only added to the memory when the prediction confidence of the current frame is ≥ ρ_update (e.g., ρ_update = 0.4) to avoid error propagation.

[0040] Step 2.2.2: Discriminative Query Initialization: Initial Query Vector for the Current Frame It is formed by fusing two parts. One part is generated by matching the features of the current frame with the memory database, denoted as... The other part is the query updated from the previous frame. ,Right now .

[0041] Step 2.2.3: Memory Retrieval and Relevance Calculation: Given the multi-scale features of the current frame Memory feature set M, calculate correlation feature map: , in, This is a correlation feature map. , and These represent the linear projection matrix of the query vector, the linear projection matrix of the key vector, and the vector dimension, respectively.

[0042] Step 2.2.4: Query Refinement: Based on Relevance Feature Map The refined global query is obtained by aggregating information from memory and updating the query. .

[0043] Under the constraint of the prediction mask m, based on the correlation feature map Find and Extract the most relevant feature regions Subsequently, the query vector is refined using a residual update formula. Wherein, This indicates element-wise multiplication. It is a learnable scaling factor used to balance update intensity. It is a learnable projection matrix. Global query. The residual update formula is as follows:

[0044] Step 2.3: The Query-Adaptive DeformableAttention mechanism includes the following steps: Step 2.3.1: Offset Prediction: The refined global query... After pooling, a global query embedding is obtained. For correlation feature maps Each spatial location on , and its feature vector and Concatenate the inputs to a lightweight offset prediction network (such as a small convolutional layer) to predict a set of... Each sampling offset The corresponding calculation formula is as follows: , in, For convolutions in offset prediction networks, is the activation function, and s is the scaling factor.

[0045] Step 2.3.2: Feature Sampling and Weighted Aggregation: Features are sampled at the predicted offset position using bilinear interpolation, and attention weights are calculated based on the query conditions to output position-enhanced features. .

[0046] Specifically: using bilinear interpolation from From the sampling features, a set of sampling features is obtained. Finally, use the refined query. As the query vector, the sampled features are subjected to attention-weighted aggregation to obtain the final enhanced output features. The calculation formula is as follows:

[0047] Where p represents the semantically aware feature map p position on and This is the linear projection matrix corresponding to the query vector and value vector, where d is the dimension of the vector. (Spatial locations) eigenvectors Forming the final enhanced output features .

[0048] Step 2.3.3: Residuals and Normalization: ... The residuals are connected to the original features and processed by LayerNorm / Conv before being fed into the downstream encoder to maintain information flow and gradient stability.

[0049] Step 2.4: Lightweight Transformer Encoder and Classification / Decoding Head; including the following steps: Step 2.4.1: Input the output features (spatial features) after adaptive attention enhancement into the lightweight encoder after spatial location or tokenization, and add positional encoding.

[0050] Step 2.4.2: Grouped Multi-Head Attention (GMHA) is used to divide the multi-head into 4 groups and share partial projections between groups to reduce computational and memory overhead.

[0051] Step 2.4.3: The encoder layer uses residual connections with LayerNorm to ultimately output feature maps for mask decoding and classification labels for target / object states. (For example, D=256).

[0052] The mask decoding head consists of several upsampled convolutional layers and pixel-by-pixel classification layers, with classification labels. Used for global target confidence estimation or categorical regression.

[0053] Step 2.5: Model training parameter settings; including the following steps: Step 2.5.1: Hyperparameter settings: Time window T=5, memory capacity=10, batch size=8 (can be increased as needed when two GPUs are running in parallel), initial learning rate lr=8e-5.

[0054] Step 2.5.2: Set the optimizer and decay strategy: Use the AdamW optimizer (with the corresponding parameters set as: β1=0.9, β2=0.999, weight_decay=1e-4), and use a polynomial or piecewise decay strategy for the learning rate (e.g., decay every N rounds or by step size).

[0055] Step 2.5.3: Set the loss function: Use weighted binary cross-entropy loss. Weighted IoU (or Dice) loss Weighted sum:

[0056] Step 2.5.4: Training process: Online sampling and data augmentation (random cropping, horizontal flipping, color jittering) are used. Every few iterations, the best model snapshot is evaluated on the validation set and saved.

[0057] Step 3: Video Object Segmentation Inference; Perform object segmentation inference on the preprocessed video sequence, output frame-by-frame object segmentation masks, and generate temporal tracking results. This includes the following steps: Step 3.1: Input preprocessing: Scale and normalize the video frames to be segmented by 480 degrees along the shorter side, and form a frame sequence according to the sampling strategy during training for model input.

[0058] Step 3.2: Memory initialization: Initialize the memory bank with the real mask of the first frame.

[0059] Step 3.3: Forward reasoning: Perform feature extraction frame by frame → memory retrieval and query refinement → query adaptive deformable attention → mask decoding, and output frame-by-frame segmentation mask and target confidence.

[0060] Step 3.4: Post-processing and visualization: Binarize the prediction mask by a threshold τ_mask (e.g., 0.5); apply temporal smoothing or small connected component filtering to low-confidence predictions or discontinuous regions; generate a target temporal heatmap or overlay visualization video; update the features / queries into the memory bank only when the confidence of the current frame is ≥ ρ_update (e.g., 0.8) to maintain memory quality.

[0061] Step 4: Model Export, Deployment, and Application; Through lightweight network design and discriminative memory optimization strategies, the model is deployed to edge devices to achieve real-time segmentation and visualization. This includes the following steps: Step 4.1: Model Export: After training, save the model weights as a .pth file and support ONNX export for deployment and cross-platform acceleration.

[0062] Step 4.2: Model compression and acceleration: Reduce the number of parameters and FLOPs through techniques such as pruning, channel pruning and mixed precision quantization (target reduction of about 20-40%), and accelerate inference by using TensorRT or TensorRT / ONNXRuntime.

[0063] Step 4.3: Edge device adaptation: Deploy the optimized model to edge computing devices (such as Jetson AGX Xavier) and combine asynchronous image acquisition with pipelined inference to achieve real-time segmentation; under the condition of input resolution of 480p, the optimized model can reach ≥20FPS on two RTX3060s and can reach near-practical frame rate on Jetson AGX Xavier (depending on the degree of quantization / acceleration).

[0064] Step 4.4: System Integration: Integrate camera, preprocessing, inference, postprocessing and front-end visualization components to provide real-time mask display, inter-frame history tracking and exportable segmentation reports.

[0065] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A video object segmentation method based on query adaptive attention and discriminative memory, characterized in that: Includes the following steps: Step 1: Video dataset construction and data preprocessing; Step 2: Construct and train a video object segmentation model that combines multi-scale semantic feature integration with query adaptive discriminative enhancement; Step 3: Video object segmentation inference; The preprocessed video sequence is subjected to target segmentation inference, and the frame-by-frame target segmentation mask is output, and the temporal tracking results are generated. Step 4: Model export, deployment, and application; By employing lightweight network design and discriminative memory optimization strategies, the model is deployed to edge devices to achieve real-time segmentation and visualization.

2. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 1, characterized in that: Step 2 includes the following steps: Step 2.1: Multi-scale feature extraction; specifically, this involves associating pre-trained VIT features with global semantic features and local spatial features to provide a comprehensive target representation; Step 2.2: The query initialization mechanism based on discriminative memory update maintains a feature memory and a query memory, storing only high-confidence target features to avoid error propagation; the initial query of the current frame is generated by fusing historical queries and current features. Step 2.3: Query the adaptive deformable attention mechanism; The refined query is used as the global semantic vector to guide the offset prediction network to adaptively generate sampling point positions, and content-aware sampling and aggregation are performed on the relevance feature map to obtain the enhanced output features. Step 2.4: Lightweight Transformer encoder and classification / decoding head process the enhanced output features to finally obtain feature maps for mask decoding and classification labels for target / object states; Step 2.5: Set the model training parameters and complete the training.

3. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 2, characterized in that: Step 2.1 includes the following steps: Step 2.1.1: Extract three-level multi-scale features using an encoder; Step 2.1.2: Reduce the number of channels in each layer to a uniform dimension using 1×1 convolution, and perform bilinear upsampling and element-wise summation to obtain multi-scale features for downstream applications. ; Step 2.1.3: Global semantic prior injection, with foreground prototype and background prototype For querying, with Using keys and values, a semantically aware feature map is generated through a cross-attention mechanism. ; Step 2.1.4: Adaptive Spatial Refinement: To handle complex structural targets, deformable convolutions are used on the feature maps. Further refinement is needed; for each spatial location Output features It is obtained by weighted summation of features at a set of adaptively predicted offset points around it; Step 2.1.5: Obtain the final features that integrate multi-scale contextual information through residual connections. .

4. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 3, characterized in that: Step 2.2 includes the following steps: Step 2.2.1: Maintain a dual-memory structure: The dual-memory structure includes a feature memory and a query memory. The feature memory stores the pixel features of historical frames, and the query memory stores historical query vectors. Step 2.2.2: Discriminative Query Initialization: The initial query vector for the current frame is formed by fusing two parts: one part is generated by matching the features of the current frame with the memory database, and the other part is the query updated in the previous frame. Step 2.2.3: Memory Retrieval and Relevance Calculation: Given the multi-scale features of the current frame Memory feature set M, calculate correlation feature map: ， in, This is a correlation feature map. , and These represent the linear projection matrix of the query vector, the linear projection matrix of the key vector, and the vector dimension, respectively. Step 2.2.4: Query Refinement: Based on Relevance Feature Map The refined global query is obtained by aggregating information from memory and updating the query. .

5. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 4, characterized in that: In step 2.2.4, under the constraint of the prediction mask m, based on the correlation feature map... Find and Extract the most relevant feature regions ; Subsequently, the query vector is refined using a residual update formula; where, This indicates element-wise multiplication. It is a learnable scaling factor used to balance update intensity. It is a learnable projection matrix; global query The residual update formula is as follows: 。 6. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 4, characterized in that: Step 2.3 includes the following steps: Step 2.3.1: Offset Prediction: The refined global query... After pooling, a global query embedding is obtained. For correlation feature maps Each spatial location on , and its feature vector and Concatenate the data, input a lightweight offset prediction network, and predict a set of... Each sampling offset ; Step 2.3.2: Feature Sampling and Weighted Aggregation: Features are sampled at the predicted offset position using bilinear interpolation, and attention weights are calculated based on the query conditions to output position-enhanced features. ; Step 2.3.3: Residuals and Normalization: ... The residuals are connected to the original features and processed by LayerNorm / Conv before being fed into the downstream encoder to maintain information flow and gradient stability.

7. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 6, characterized in that: In step 2.3.2, bilinear interpolation is used from... From the sampling features, a set of sampling features is obtained. Use the refined query As the query vector, the sampled features are subjected to attention-weighted aggregation to obtain the final enhanced output features. The calculation formula is as follows: Where p represents the semantically aware feature map p position on and This is the linear projection matrix corresponding to the query vector and the value vector, where d is the dimension of the vector; each spatial position... eigenvectors Forming the final enhanced output features .

8. The video object segmentation method based on query adaptive attention and discriminative memory according to claim 4, characterized in that: Step 2.4 includes the following steps: Step 2.4.1: Input the output features enhanced by adaptive attention into the lightweight encoder after spatial location or tokenization, and add positional encoding; Step 2.4.2: Use grouped multi-head self-attention, divide the multi-head into 4 groups and share partial projection between groups to reduce computational and memory overhead; Step 2.4.3: The encoder layer uses residual connections with LayerNorm to ultimately output feature maps for mask decoding and classification labels for target / object states. ; The mask decoding head consists of several upsampled convolutional layers and pixel-by-pixel classification layers, with classification labels. Used for global target confidence estimation or categorical regression.

9. A video object segmentation method based on query adaptive attention and discriminative memory according to claim 4, characterized in that: Step 2.5 includes: setting hyperparameters, setting the optimizer and decay strategy, setting the loss function, and training. The learning rate uses a multinomial or piecewise decay strategy, and the loss function uses weighted binary cross-entropy loss. Weighted IoU (or Dice) loss The weighted sum; training employs online sampling and data augmentation, evaluating and saving the best model snapshot on the validation set every few iterations.

10. A video object segmentation method based on query adaptive attention and discriminative memory according to claim 1, characterized in that: Step 3 includes the following steps: Step 3.1: Input preprocessing: Scale and normalize the video frames to be segmented by the shorter side, and form a frame sequence according to the sampling strategy during training for model input; Step 3.2: Memory Initialization: Initialize the memory bank using the actual mask of the first frame; Step 3.3: Forward reasoning: Perform feature extraction frame by frame, then memory retrieval and query refinement, enhance features through query adaptive deformable attention and output features, and finally perform mask decoding to output frame-by-frame segmentation mask and target confidence. Step 3.4: Post-processing and visualization: Binarize the prediction mask by threshold; apply temporal smoothing or small connected component filtering to low-confidence predictions or discontinuous regions; generate a target temporal heatmap or overlay visualization video; update features / queries into the memory bank only when the confidence of the current frame reaches the threshold to maintain memory quality.

Images