Lightweight video object segmentation method based on sparsification memory and recurrent dynamic feature compression

Abstract

The present application relates to a kind of light-weight video target segmentation method based on sparsification memory and cyclic dynamic feature compression, belong to video target segmentation technical field.The method is first fused by encoder to current video frame image and its preliminary mask;The preliminary fusion feature is adaptively reduced dimension in memory feature generation link by light-weight memory generation method based on cyclic dynamic hybrid expert, and then local information density is judged by adaptive memory storage compression strategy based on information density, and memory feature is stored in memory bank;Local diversity screening is carried out based on the sparsity of the maximum information diversity retrieval strategy of memory, and sparse memory retrieval space is constructed according to the screened information;High representation information is read from sparse memory retrieval space using decoder, combined with the shallow features of backbone network, reconstruct and output target pixel-level mask.The present application can effectively alleviate the problem that the calculation and storage overhead increases with the length of video.

Description

Technical Field

[0001] This invention belongs to the field of video target segmentation technology, and relates to a lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression. Background Technology

[0002] For video object segmentation (VOS) tasks, existing methods can be mainly categorized into three types. Online fine-tuning-based methods adapt the network online using first-frame annotations, enabling the model to memorize the appearance and semantic attributes of the target. However, due to the need for repeated fine-tuning during the inference phase, they suffer from high computational overhead, slow inference speed, and sensitivity to input changes. Mask propagation-based methods rely on the predicted mask of the previous or first frame for frame-by-frame propagation. While simple in structure and efficient, they lack modeling of long-term spatiotemporal context, making them prone to error accumulation in cases of target disappearance, occlusion, or rapid movement, resulting in limited robustness. In contrast, feature matching-based methods have become the mainstream approach. These methods achieve frame-by-frame segmentation by establishing feature correspondences between target frames and reference frames. Among them, spatiotemporal memory networks (such as STM) and their improved methods (such as XMem) effectively enhance the modeling ability of short- and long-term temporal information by introducing multi-level memory mechanisms, achieving excellent performance while maintaining low computational cost. However, the high dependence of these methods on large-scale feature memorization and high-frequency matching operations also puts them under significant computational and storage pressure in long videos and high-resolution scenarios.

[0003] Against this backdrop, lightweight research on video object segmentation has gradually become an important research direction. The core issue is how to reduce the overhead of spatiotemporal memory networks in terms of storage scale and computational complexity while maintaining the advantages of memory matching. Existing lightweight methods mainly focus on the structure and inference process of spatiotemporal memory networks, but they still face the bottleneck of difficulty in balancing efficiency and accuracy. As a foundational work in this field, STM achieved effective temporal inference by constructing an external feature memory bank, but its memory usage increases linearly with the video duration, making it difficult to apply to long video sequences. Although subsequent STCN and XMem alleviated the memory expansion problem by improving memory retrieval strategies and introducing hierarchical storage mechanisms, the complex feature matching process still brings high computational overhead, and the solidification of long-term memory features weakens the model's adaptability to changes in the appearance of the target to some extent. In terms of real-time performance, SwiftNet improves inference speed through a pixel-level adaptive update strategy, but its change-aware mechanism is prone to error accumulation in motion-blurred or occluded scenarios. While MobileVOS, designed for mobile deployment, significantly compresses model size through knowledge distillation, its segmentation accuracy in complex backgrounds still suffers a noticeable decline due to the limited representational capabilities of its lightweight backbone network. Overall, existing methods still struggle to achieve an ideal balance between memory efficiency, computational complexity, and segmentation accuracy, hindering the practical deployment of video object segmentation algorithms in resource-constrained environments.

[0004] In long-term video object segmentation tasks, memory-based networks achieve cross-temporal correlation modeling by accumulating historical features. However, their resource consumption increases rapidly with the sequence length, becoming a core bottleneck restricting practical deployment. As the number of video frames increases, multi-frame object features are continuously written into the memory, causing the number of memory tokens to expand linearly over time. This not only rapidly consumes a large amount of GPU memory resources but also significantly increases the complexity of subsequent similarity matching and attention calculations. When processing long video sequences containing thousands of frames, the system is prone to accumulated computational latency in the later stages of inference, making it difficult to meet the application requirements of real-time and low-power scenarios.

[0005] From the perspective of feature representation, existing memory storage mechanisms generally suffer from significant redundancy issues. Spatially, background regions and smooth areas within the target in video frames occupy a large number of pixels, and their corresponding features are highly similar in semantic space. Repeatedly storing this information not only wastes storage resources but also introduces interference during global matching, weakening the discriminative power of effective features. Temporally, the limited appearance changes between adjacent frames lead to a high degree of semantic overlap in feature descriptions within the memory bank; in reality, only a small number of representative key features are needed to maintain a stable target representation. Furthermore, the redundancy of high-dimensional feature representations in the channel dimension increases the parameter scale of subsequent decoding modules, further amplifying the overall computational burden.

[0006] Based on the above problems, how to effectively compress redundant spatiotemporal information and reduce feature representation complexity while ensuring segmentation accuracy has become a key technical challenge for improving the efficiency of video target segmentation. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a lightweight video object segmentation method based on sparse memory and cyclic dynamic feature compression, which is synergistically optimized from three levels: the storage method of memory features, the retrieval strategy, and the representation dimension. By introducing an adaptive spatial feature compression mechanism, a sparse filtering strategy that emphasizes information diversity, and a dynamic dimension mapping model that combines temporal semantic constraints, this method significantly reduces memory usage and computational overhead while achieving stable and efficient inference for long video sequences, thereby alleviating the strong coupling between high-precision segmentation performance and high computational cost.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A lightweight video object segmentation method based on sparse memory and cyclic dynamic feature compression, the method comprising: S1. Obtain the current video frame image and its prior mask, and perform preliminary fusion through the encoder; S2. The preliminary fusion features are adaptively reduced in the memory feature generation stage using a lightweight memory generation method based on cyclic dynamic hybrid experts. S3. The generated preliminary memory features are used to determine the local information density through an adaptive memory storage compression strategy based on information density, and the memory features are stored in the memory bank. S4. Based on the sparse memory retrieval strategy of maximizing information diversity, perform local diversity filtering and construct a sparse memory retrieval space based on the filtered information. S5. Then, the decoder is used to read highly representative information from the sparse memory retrieval space, and combined with the shallow features of the backbone network, the target pixel-level mask is reconstructed and output.

[0009] Furthermore, in step S1, the value encoder adopts a lightweight design based on ResNet-18, which realizes the synchronous encoding of appearance features and location priors by concatenating the original image with the target mask as input; and generates a highly discriminative Value Token by fusing the deep semantic features of the key encoding.

[0010] Furthermore, in step S2, a content-aware dynamic projection strategy is first introduced. By explicitly modeling the global semantic information of the current frame, it guides expert selection and weight allocation during feature compression. The process is as follows: The encoder is set at time... t The output fusion features are First, a global average pooling operation is performed on the spatial dimension to compress the features into channel-level descriptions; then, a low-dimensional semantic embedding vector is generated through linear mapping. :

[0011] in, This represents the weight matrix of the linear projection layer. Represents the ReLU activation function. These represent the batch size, height, and width of the input feature map, respectively. This represents the bias vector of the linear projection layer; A cyclic gating unit is introduced for decision-making, which involves storing the hidden state from the previous time step. As a decision memory, it is introduced into the expert selection process of the current frame to carry the constraints of historical semantic distribution on the compression strategy; the gating logic of the cyclic gating unit and its hidden state update equation are as follows:

[0012]

[0013]

[0014]

[0015] in, , These represent resetting the door and updating the door, respectively. The state is hidden for the decision at the current moment; For activation function, To reset the input weight matrix of the gate, To reset the cyclic weight matrix of the gate, To reset the gate's bias vector; To update the input weight matrix of the gate, To update the cyclic weight matrix of the gate, To update the bias vector of the gate; This represents the current candidate hidden state. The hyperbolic tangent activation function is used. The input weight matrix represents the candidate states. Here is the cyclic weight matrix for the candidate states. The bias vector for the candidate state; Finally, the hidden state is processed using the Softmax function. After normalization, the contribution weights of each expert are obtained. :

[0016] In the formula, the subscript " "Indicates time" Next An index corresponding to each expert.

[0017] Furthermore, step S2 also includes constructing a hybrid expert pool and providing online experience feedback, the process of which is as follows: The expert pool consists of three static projection experts And a dynamic caching expert The system is composed of static experts, which are obtained through offline end-to-end training and are responsible for characterizing stable and universally applicable dimensionality reduction mapping relationships in the feature space; while dynamic cache experts serve as online updatable adaptation units, used to continuously absorb and characterize the personalized appearance features of the target in the current video. For the online inference phase, based on the obtained routing weights At the parameter level, multiple experts are weighted and fused; the fusion process is as follows: The optimal projection operator for the current frame is generated in real time using tensor shrinking. :

[0018] in, For the fused dynamic projection kernel, For the first The convolution weight matrix of a static expert, The convolution weight matrix for dynamic caching experts; Subsequently, the dynamic projection kernel is applied to the output features of the value encoder in the form of grouped convolutions. Complete the mapping from high dimension to low dimension:

[0019] in, The low-dimensional features after mapping Represents the convolution operation. The bias terms of each expert are determined based on the routing weights. The dynamically shifted bias vector synthesized by linear weighting; After each frame of inference is completed, the projection kernel synthesized in the current frame is used. right Perform smooth updates to gradually incorporate recent observational experience:

[0020] in, The inertia factor is used to control the fusion ratio between new and old experiences; it is increased in the early stages of the video sequence. Values ​​are used to quickly capture target attributes; after the model has formed a stable representation of the target, the value is reduced. To filter environmental noise.

[0021] Furthermore, in step S3, an adaptive storage compression method for segmented memory is provided, which dynamically adjusts the storage strategy based on the information distribution of local features. Specifically, for regions with gentle feature changes and high semantic consistency, features are compressed through aggregation operations; for regions with large feature differences and clear semantic boundaries, representative feature responses are selectively retained. The adjustment process first defines a metric that reflects the degree of local feature change, and then executes a key-value synchronous hybrid compression strategy based on a decision threshold. The Key feature is used as the basis for calculating local information density, and the process is as follows: The key feature map is divided into several non-overlapping parts in spatial dimension. Local windows, each window corresponds to a spatial location index. For any local window Its interior contains Key feature vectors ; The mean of the eigenvectors within the window is calculated as a local reference center:

[0022] Then, the average Euclidean dispersion is introduced as an indicator to measure the information complexity within the window, which is calculated by the degree of deviation of each key feature within the window from the mean vector:

[0023] The decision threshold is set using a dynamic quantile thresholding method, and the dispersion distribution of the key features across the entire image is statistically analyzed. Using the dynamic quantile threshold method and according to a preset ratio Calculate the decision threshold :

[0024] in, Quantity ratio; Construct the corresponding binary decision mask This is used to indicate the compression strategy used by different windows:

[0025] in, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise.

[0026] Furthermore, step S3 also includes an information density mask. The controlled dual-branch hybrid compression structure uses key features as a reference for spatial sampling, determines the compression method and sampling location based on the local statistical information of the key features, and synchronously applies the corresponding rules to the value features, wherein: for In the region, a pooling-based aggregation-based compression method is used to simplify the features, resulting in the compressed feature representation. Obtained from the mean of features within a local window:

[0027] for In the region, a selective sampling strategy based on maximum deviation is adopted to preserve key semantic information by retaining the feature vectors that differ most from the local mean.

[0028] The compression characteristics of highly discrete regions are then expressed as: ; Finally, through information density masking By selectively combining the two compression paths described above, the final compressed feature representation is obtained. .

[0029] Furthermore, in step S4, memory features are quickly filtered before global attention computation is performed based on a sparsity strategy for the memory retrieval stage. First, a sparsity reduction method based on group processing is performed: the flattened long-term memory key sequence is... Divide the sequence along its dimensions into several local subsequences, each subsequence containing A series of consecutive tokens, and select from them. A representative token is proportionally compressed; let the first... The local subsequences are represented as L2 normalization is performed on all tokens within the group to obtain a set of unit vectors. Subsequently, using the linear superposition property of vector algebra, the eigenvalues ​​and vectors of this set are calculated. :

[0030] The relevance score of each token is calculated using a single dot product operation. :

[0031] Perform a score-based selection operation on each subsequence to obtain the corresponding set of locally retained indices:

[0032] To avoid disrupting the sequence structure and maintain temporal consistency, selected local indices are sorted in ascending order and mapped back to their original sequence positions, thereby generating a global sparse mask. For sequences ending with less than a complete group of tokens, all are retained without further filtering. Based on the generated sparse mask, the key sequences, value sequences, and related auxiliary items in the memory are simultaneously pruned to construct a sparsified memory. , .

[0033] Furthermore, in step S5, the decoder adopts a progressive upsampling architecture based on feature pyramid logic to restore the high-dimensional readout features retrieved from the memory space into an accurate pixel-level target mask; through the skip connection organic fusion key encoder, the multi-scale shallow spatial features retained in the downsampling stage are integrated with the readout semantic information and underlying texture details using a series of feature fusion operators.

[0034] The beneficial effects of this invention are as follows: This invention employs a value encoder incorporating a cyclic dynamic hybrid expert mechanism to encode the target features of the current frame. Unlike traditional fixed-dimensional mapping methods, this encoder can incorporate the temporal variation characteristics of the target, achieving adaptive compression of the channel dimension while generating memory features. By mapping the original high-dimensional features to a more compact representation space, the system reduces the representation size of a single memory unit while maintaining semantic expressiveness, thereby reducing the computational burden on subsequent storage and decoding modules.

[0035] To reduce redundant information in the spatial dimension, this invention introduces an adaptive compression strategy based on local information distribution. During the feature writing process, the appropriate storage method is selected according to the information complexity of different regions of the feature map: regions with small semantic changes are aggregated and compressed, while regions containing structural boundaries or significant changes are preferentially retained for representative feature responses. Through this method, the number of features written to the memory bank per frame is effectively reduced without affecting the expression of key target information, thus lowering the GPU memory consumption during long-term operation.

[0036] To address the computational overhead associated with the continuous expansion of the similarity memory, this invention employs a sparse feature selection mechanism. Before performing similarity calculations, the memory features are grouped and evaluated, and only a subset of features with strong informational capabilities are selected for subsequent matching. This strategy ensures matching stability while reducing the scale of attention computation, thereby improving overall inference efficiency.

[0037] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0038] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall architecture of the lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a lightweight memory representation generation method based on a cyclic dynamic hybrid expert according to an embodiment of the present invention; Figure 3 This is a schematic diagram of adaptive memory storage compression based on information density according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a memory sparse retrieval strategy based on maximum information diversity according to an embodiment of the present invention. Detailed Implementation

[0039] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0040] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0041] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0042] Please see Figures 1-4 This is a lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression.

[0043] Example This embodiment details a lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression, such as... Figure 1 As shown, it specifically includes the following steps: S1. Obtain the current video frame image and its prior mask, and perform preliminary fusion through the encoder; S2. The preliminary fusion features are adaptively reduced in the memory feature generation stage using a lightweight memory generation method based on cyclic dynamic hybrid experts. S3. The generated preliminary memory features are used to determine the local information density through an adaptive memory storage compression strategy based on information density, and the memory features are stored in the memory bank. S4. Based on the sparse memory retrieval strategy of maximizing information diversity, perform local diversity filtering and construct a sparse memory retrieval space based on the filtered information. S5. Then, the decoder is used to read highly representative information from the sparse memory retrieval space, and combined with the shallow features of the backbone network, the target pixel-level mask is reconstructed and output.

[0044] In step S1 of this embodiment, the value encoding module uses a lightweight ResNet-18 as its backbone network. By concatenating the original image and the target mask as inputs along the channel dimension, it achieves deep coupling between visual appearance representation and spatial location prior. During feature extraction, this module further integrates global semantic cues provided by the key encoder, thereby producing value vectors with strong discriminative power. As key carriers of target information, these high-dimensional embeddings are persistently stored in the memory bank, providing core data support for identity reconstruction and spatiotemporal consistency maintenance when the model processes long-term video sequences.

[0045] In step S2 of this embodiment, in existing video object segmentation systems, value encoders typically output high-dimensional feature representations to support semantic modeling in complex scenarios. However, in long-term video applications, such high-dimensional memory features can lead to significant resource pressure. On the one hand, the memory usage of the memory module is linearly related to the feature channel dimension. When memory features are continuously written to the memory bank at a high dimension, memory consumption accumulates rapidly with the video length, making it susceptible to hardware resource limitations when processing large-scale video sequences. On the other hand, high-dimensional memory features also increase the computational burden of subsequent decoding stages. During multi-scale decoding, features from the memory bank need to be repeatedly fused with multi-layer features output by the encoder. The larger the channel size, the more convolutional operations and intermediate caching overhead are involved, thus affecting overall inference efficiency, especially unfavorable for deployment in resource-constrained environments.

[0046] To alleviate the aforementioned problems, this invention proposes a recurrent dynamic hybrid expert module for performing pre-mapping of dimensions during the memory feature generation stage. This module, without relying on fixed mapping rules, reorganizes and compresses input features through a recurrent dynamic expert selection mechanism, reducing channel dimensions while generating memory representations. Specifically, the original high-dimensional features are mapped to a more compact representation space in this module, thereby reducing the representation size of individual memory units before being written to the memory bank. Through this design, the channel dimensions of the memory features are compressed at the source, significantly reducing the storage load on the memory bank. Simultaneously, the feature size required for processing during the decoding stage is reduced, resulting in a cascaded computational and memory optimization effect throughout the inference chain. The overall structure is as follows: Figure 2 As shown.

[0047] The proposed cyclic dynamic hybrid expert module stems from the observation of the insufficient adaptability of traditional static linear projection in the feature compression stage. In video object segmentation tasks, the appearance features of objects change continuously over time, significantly affected by factors such as lighting conditions, occlusion relationships, and non-rigid deformation. In this context, using fixed mapping parameters to reduce the dimensionality of high-dimensional features often fails to simultaneously achieve information preservation and compression efficiency at different time steps, easily leading to the loss of key semantic components. Based on the above analysis, this embodiment introduces a content-aware dynamic projection strategy. By explicitly modeling the global semantic information of the current frame, it guides expert selection and weight allocation during the feature compression process, enabling the dimensionality reduction operation to adaptively adjust with changes in video content, thereby improving the stability and robustness of the memory representation in the temporal dimension.

[0048] The encoder is set at time... t The output fusion features are To extract global contextual information while ensuring computational efficiency, the module first performs global average pooling on the spatial dimension, compressing the features into channel-level descriptions. Then, a low-dimensional semantic embedding vector is generated through linear mapping. This serves as the input for subsequent decision-making modules. Its calculation form is shown below:

[0049] in, This represents the weight matrix of the linear projection layer. Represents the ReLU activation function. These represent the batch size, height, and width of the input feature map, respectively. This represents the bias vector of the linear projection layer. A 128-dimensional semantic embedding space is used to minimize the computational and parameter overhead of the recurrent decision unit while expressing global semantic features.

[0050] In designing the routing decision structure, this invention does not follow the independent routing approach based on multilayer perceptrons commonly found in traditional hybrid expert models. Instead, it introduces a recurrent gated unit (GRU) as the core decision module. Existing methods typically assign expert weights based solely on the features of the current frame, lacking the ability to model historical states. In cases of rapid target movement or brief absence, this can easily lead to drastic changes in the projection strategy, thus affecting the temporal consistency of the segmentation results. Therefore, the proposed recurrent routing mechanism incorporates the hidden state from the previous time step... As a decision memory, it is introduced into the expert selection process of the current frame to carry the constraints of historical semantic distribution on the compression strategy. The gating logic of the GRU unit and its hidden state update equation are as follows:

[0051]

[0052]

[0053]

[0054] in, , These are respectively referred to as the reset door and the update door. The hidden state is used for the decision at the current moment; S For activation function, To reset the input weight matrix of the gate, To reset the cyclic weight matrix of the gate, To reset the gate's bias vector; To update the input weight matrix of the gate, To update the cyclic weight matrix of the gate, To update the bias vector of the gate; This represents the current candidate hidden state. The hyperbolic tangent activation function is used. The input weight matrix represents the candidate states. Here is the cyclic weight matrix for the candidate states. The bias vector for the candidate state; Finally, the hidden state is processed using the Softmax function. After normalization, the contribution weights of each expert are obtained. :

[0055] subscript " "Indicates time" Next The index corresponds to each expert. This cyclic decision routing mechanism provides the hybrid expert module with continuous constraints in the time dimension, thereby avoiding unnecessary oscillations in the compression strategy due to inter-frame noise. It ensures that the feature dimension compression process follows a smooth and consistent semantic evolution trajectory, which is particularly suitable for long-term video object segmentation scenarios.

[0056] Hybrid Expert Pool Construction and Online Experience Feedback Mechanism: Regarding the construction of the expert pool, this invention employs an asymmetric "3+1" hybrid expert structure to simultaneously consider general modeling capabilities and target-specific adaptability. Specifically, the expert pool consists of three statically projected experts. And a dynamic caching expert The system is composed of static experts, which are obtained through offline end-to-end training and are mainly responsible for characterizing stable and universally applicable dimensionality reduction mapping relationships in the feature space; while dynamic cache experts serve as online updatable adaptation units, used to continuously absorb and characterize the personalized appearance features of the target in the current video.

[0057] During the online inference phase, the system first bases its decisions on the obtained routing weights. At the parameter level, multiple experts are weighted and fused. To achieve frame-by-frame adaptive projection kernel construction, this invention employs tensor shrinking to generate the optimal projection operator for the current frame in real time without introducing additional memory copying. The process is defined as follows:

[0058] in, This represents the fused dynamic projection kernel. For the first The convolution weight matrix of a static expert, This is the convolutional weight matrix for the dynamic caching expert. Subsequently, this projection kernel acts on the value encoder output features in the form of grouped convolutions. The mapping from high dimension to low dimension is completed, and the calculation process is shown in the following formula:

[0059] in The low-dimensional features after mapping Represents the convolution operation. The bias terms of each expert are determined based on the routing weights. A dynamically weighted bias vector is synthesized using linear weighting. This dynamic projection process avoids the limitations of "uniform compression" in traditional fixed dimensionality reduction methods, transforming feature compression into a selective mapping mechanism that preserves features on demand. Through frame-level adaptive parameter adjustment, the model can continuously perceive changes in the appearance of the target in the video sequence and dynamically enhance the preservation of discriminative semantic components, thereby significantly reducing channel dimensions while effectively suppressing the loss of key information.

[0060] To enable the dynamic caching expert to self-update over time, this invention further introduces an online experience feedback mechanism. After each frame inference is completed, the system utilizes the projection kernel synthesized in the current frame. right A smooth update is performed to gradually incorporate recent observational experience, as shown in the following equation:

[0061] in, The inertia factor is used to control the fusion ratio between new and old experiences. To achieve a balance between rapidly adapting to changes in the target and suppressing short-term noise, this embodiment designs a phased adaptive inertia factor strategy. In the initial stage of the video sequence, a larger inertia factor is set. The value is used to quickly capture the target attribute; however, after the model has formed a stable representation of the target, the value is reduced. To filter environmental noise, as shown in the following formula:

[0062] In step S3 of this embodiment, in the original VOS framework, the construction of segmentation memory features relies on joint representations from multiple visual information streams. The system first performs feature extraction processing on the current video frame, encoding the original image information and the corresponding segmentation prediction result to obtain an initial feature representation that simultaneously includes semantic description and spatial prior. Subsequently, this representation is fused with the deep visual features output by the backbone feature extraction network to generate a segmentation feature map for long-term memory modeling. While these features possess strong abstraction capabilities at the semantic level, they maintain a consistent layout with the input image in spatial structure. Due to the significant spatial imbalance of visual content, feature representations within semantically consistent regions (e.g., background or target interior regions) are often highly similar. This makes storing all memory units point-by-point significantly redundant in practical applications, increasing memory usage and imposing an additional burden on subsequent similarity-based matching processes.

[0063] To address this, this invention proposes an adaptive storage compression method for segmentation memory. This method dynamically adjusts the storage strategy based on the information distribution of local features, rather than using a fixed ratio or uniform rules for downsampling. Specifically, for regions with gentle feature changes and high semantic consistency, the system compresses features through aggregation operations to eliminate redundant information; while in regions with significant feature differences and clear semantic boundaries, representative feature responses are selectively retained to maintain key structural information. Through this content-difference-based processing method, the system effectively reduces the number of features written to the memory bank without weakening the segmentation expressive power, thereby reducing the storage and computational overhead during long-term inference. An overall illustration is shown below. Figure 3 As shown.

[0064] Mechanism for measuring local information density: To achieve adaptive compression based on content differences, it is first necessary to define a measurement method that can reflect the degree of change in local features. Given the key features... Carrying sufficient discriminative information in the spatial dimension, this invention uses Key features as the basis for calculating local information density. Specifically, the Key feature map is divided into several non-overlapping regions in the spatial dimension. Local windows, each window corresponds to a spatial location index. For any local window Its interior contains Key feature vectors To characterize the central tendency of the feature distribution within this region, the mean of the feature vectors within the window is first calculated as a local reference center, defined as follows:

[0065] in, For the window size, in this embodiment, we take... .

[0066] After obtaining the local mean, the average Euclidean dispersion is introduced as an indicator to measure the information complexity within the window. This indicator is calculated by the degree of deviation of each key feature within the statistical window from the mean vector, and its expression is shown in the following formula:

[0067] The above discrete measure reflects the magnitude of change in the Key feature within a local region. When When the value is small, it indicates that the feature distribution within the window is concentrated and the semantic expression is consistent, usually corresponding to a smooth region inside the background or target; when... When the value is large, it indicates that there are obvious differences in characteristics within the region, which often occur at the edge of the target or in locations with significant structural changes.

[0068] To adaptively segment high and low information density regions, this embodiment sets a decision threshold. Considering the differences in the dynamic range of feature distributions across different video frames, a fixed threshold lacks generalization ability. Therefore, this invention employs a dynamic quantile thresholding method. Specifically, it statistically analyzes the dispersion distribution of the key features across the entire image. Using the dynamic quantile threshold method and according to a preset ratio Calculate the decision threshold :

[0069] in, As a quantile ratio, this embodiment uses 0.5, representing that high information density and low information density each account for half.

[0070] Based on this, construct the corresponding binary decision mask. This is used to indicate the compression strategy adopted by different windows, and its definition is shown in the following formula:

[0071] in, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise.

[0072] Key-value synchronous hybrid compression strategy: During the memory retrieval phase, attention calculations rely on key features to generate spatial matching weights, and these weights are then used to weight and converge the corresponding value features. This process requires a one-to-one correspondence between key and value features in spatial location; otherwise, the matching results will be inconsistent with the actual semantic locations, thus affecting the accuracy of the segmentation results. Therefore, when compressing memory features, it is essential to avoid using inconsistent sampling strategies for keys and values.

[0073] Therefore, this invention proposes a synchronous compression mechanism guided by key features. This mechanism uses key features as a reference for spatial sampling. First, it determines the compression method and sampling position based on the local statistical information of the key features, and then synchronously applies the corresponding rules to the value features. This reduces the size of the memory features while ensuring strict alignment of the compressed key-value pairs in the spatial dimension. Based on the above design, a mechanism using information density masks is constructed. Controlled dual-branch hybrid compression structure.

[0074] (1) For regions with low information dispersion Key features within a local window typically exhibit strong consistency, with subtle differences contributing little to semantic representation. For such regions, a pooling-based aggregation-based compression method is used to simplify the features. Specifically, the compressed feature representation... It is obtained from the mean of features within a local window, as shown in the following formula:

[0075] This processing method can reduce the number of features while smoothing out random fluctuations, making the generated memory features more stable.

[0076] (2) For regions with high information dispersion Local windows typically contain significant semantic changes or structural boundary information. Directly aggregating such regions can easily lead to the mixing of different semantic features, thereby weakening the feature discrimination ability. Therefore, this embodiment employs a selective sampling strategy based on maximum deviation, retaining the feature vector with the largest difference from the local mean to maintain key semantic information. Specifically, the following indices are determined within the window:

[0077] Since the distance term is already obtained when calculating the local dispersion index, this sampling process does not introduce additional computational overhead. Finally, the compressed features of the highly dispersed region are represented as follows: .

[0078] Information density mask By selectively combining the two compression paths described above, the final compressed feature representation can be obtained. :

[0079] In practical implementation, since the key features directly determine the accuracy of the matching, their spatial structure is the most critical. Therefore, this method first calculates the information density mask based on the key features. and the corresponding optimal sampling index The result is then directly applied to the value feature compression process. The value feature compression method is defined as follows:

[0080] This design ensures that value features strictly adhere to the spatial sampling positions of key features during compression, thus avoiding feature misalignment issues caused by inconsistent sampling. Through this synchronous compression mechanism, the number of key-value features written to the memory module is significantly reduced, lowering memory usage while improving the execution efficiency of subsequent segmentation calculations. This provides an efficient and stable implementation method for long-term video object segmentation tasks.

[0081] In step S4 of this embodiment, during the inference process of video target segmentation, the memory retrieval module is one of the links with the highest consumption of video memory and computing resources. This module needs to retrieve relevant information from historical memory using the encoded features of the current frame. Its implementation is usually based on an attention computing framework: the features of the current frame are used as query vectors for matching, the historical features stored in the memory are used to provide matching references, and a context-enhanced feature representation is generated through a weighted method.

[0082] As the length of video sequences increases, the number of feature units accumulated in the memory continues to expand, causing the size of the intermediate matrix required for matching calculations to grow rapidly. This not only significantly increases the GPU memory usage during the inference stage but also increases the overall computational overhead, thus limiting the real-time processing capabilities of long temporal videos. Especially in ultra-long video scenarios, directly performing global matching on all memory features leads to a significant decrease in efficiency. From the perspective of feature distribution, memory features in long videos often exhibit strong repetition. Whether between consecutive frames in the temporal dimension or between adjacent regions in the spatial dimension, their corresponding feature representations show high similarity in the embedding space. This redundancy means that a large number of matching calculations do not bring effective information gain but instead consume unnecessary computational resources.

[0083] To address the aforementioned issues, this invention proposes a sparsity strategy for the memory retrieval stage, used to quickly filter memory features before performing global attention computation. This method introduces a low-complexity preprocessing step to group and evaluate the key features in the memory. Within each group, a subset with the lowest similarity to other tokens is retained, thereby ensuring that the retained token subset contains the maximum information richness, thus enhancing the overall information richness of the memory features and improving the efficiency of subsequent attention computation. Figure 4 As shown.

[0084] To reduce computational burden while ensuring screening efficiency, this invention employs a sparsification method based on grouping processing. Specifically, the flattened long-term memory key sequence... The sequence is divided into several local subsequences along the sequence dimension, and each subsequence contains a fixed number of consecutive tokens. In this method, each group contains... Each token, and select from them. Using representative tokens, a sequence compression ratio of approximately 50% is achieved. Let the first... The local subsequences are represented as The core criterion for selection is to maximize the richness of local information contained in the retained tokens. To quantify this metric, all tokens within the group are first L2 normalized to obtain a set of unit vectors. Subsequently, using the linear superposition property of vector algebra, the eigenvalues ​​and vectors of this set are calculated. It represents the average characteristic direction within this local region, as shown in the following formula:

[0085] Based on this, the relevance score of each token is calculated directly through a single dot product operation. :

[0086] The score is mathematically equivalent to the sum of the similarities between the token and all other vectors in the group, but with significantly reduced computational cost.

[0087] Based on the principle of maximizing information diversity, the subset of tokens with lower relevance scores often provides the most discriminative and richest semantic information. Therefore, a score-based selection operation is performed on each subsequence to obtain the corresponding locally retained index set, as shown in the following equation:

[0088] in, .

[0089] To avoid disrupting the sequence structure and maintain temporal consistency, selected local indices are sorted in ascending order and mapped back to their original sequence positions, thereby generating a global sparse mask. For sequences ending with less than a complete group of tokens, considering they typically contain the latest temporal information, they are all retained without further filtering. Based on the generated sparse mask, the system performs synchronous pruning on the key sequences, value sequences, and related auxiliary items in the memory, constructing a sparsified memory. , .

[0090] In step S5 of this embodiment, to address the geometric deformation and edge blurring issues that easily occur during resolution restoration of deep semantic features, the decoder adopts a coarse-to-fine progressive restoration scheme. The module performs cross-layer mapping between the memory readout features with strong discriminative power and the detailed features retained by the encoder by concatenating a series of feature fusion operators. This design essentially utilizes the local topological information of the shallow network to perform "pixel-level correction" on the deep global information, significantly improving the model's accuracy in replicating complex contours and minute structures. The resulting mask exhibits excellent spatiotemporal coherence and edge sharpness.

[0091] In summary, the lightweight memory representation generation method based on cyclic dynamic hybrid experts proposed in this invention alleviates the contradiction between storage density and computational load in long-duration video segmentation tasks by implementing deep feature dimension compression at the front end. This design, combining dynamic perception and temporal constraints, achieves a lightweight effect throughout the entire process while maintaining almost all segmentation accuracy. By introducing compressed mapping in the memory feature representation stage and employing a sparsity strategy in memory storage and matching, the problem of computational and storage overhead increasing with video length is effectively mitigated.

[0092] (1) Front-end feature representation generation stage: The system uses a value encoder with a recurrent dynamic hybrid expert mechanism to encode the target features of the current frame. Unlike the traditional fixed-dimensional mapping method, this encoder can combine the temporal variation characteristics of the target and perform adaptive compression of the channel dimension while generating memory features. By mapping the original high-dimensional features to a more compact representation space, the system reduces the representation size of a single memory unit while ensuring semantic expressiveness, thereby reducing the computational burden of subsequent storage and decoding modules.

[0093] (2) Mid-range memory storage stage: To reduce redundant information in the spatial dimension, the system introduces an adaptive compression strategy based on local information distribution. During the memory feature writing process, this module selects the appropriate storage method according to the information complexity of different regions of the feature map: regions with small semantic changes are aggregated and compressed, while regions containing structural boundaries or significant changes are preferentially retained with representative feature responses. Through the above method, without affecting the expression of key target information, the number of features written to the memory bank per frame is effectively reduced, and the memory consumption during long-term operation is reduced.

[0094] (3) Backend matching and retrieval stage: To address the computational overhead caused by the continuous expansion of the memory database, the system employs a sparse feature selection mechanism. Before performing similarity calculations, the memory features are grouped and evaluated, and only a subset of features with strong information expression capabilities are selected to participate in the subsequent matching process. This strategy ensures matching stability while reducing the scale of attention computation, thereby improving overall reasoning efficiency.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression, characterized in that: The method includes: S1. Obtain the current video frame image and its prior mask, and perform preliminary fusion through the encoder; S2. The preliminary fusion features are adaptively reduced in the memory feature generation stage using a lightweight memory generation method based on cyclic dynamic hybrid experts. S3. The generated preliminary memory features are used to determine the local information density through an adaptive memory storage compression strategy based on information density, and the memory features are stored in the memory bank. S4. Based on the sparse memory retrieval strategy of maximizing information diversity, perform local diversity filtering and construct a sparse memory retrieval space based on the filtered information. S5. Then, the decoder is used to read highly representative information from the sparse memory retrieval space, and combined with the shallow features of the backbone network, the target pixel-level mask is reconstructed and output.

2. The lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 1, characterized in that: In step S1, the value encoder adopts a lightweight design based on ResNet-18. By concatenating the original image with the target mask as input, it achieves simultaneous encoding of appearance features and location priors. By fusing the deep semantic features encoded by the key, it generates a highly discriminative Value Token.

3. The lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 2, characterized in that: In step S2, a content-aware dynamic projection strategy is first introduced. By explicitly modeling the global semantic information of the current frame, it guides expert selection and weight allocation during feature compression. The process is as follows: The encoder is set at time... t The output fusion features are First, a global average pooling operation is performed on the spatial dimension to compress the features into channel-level descriptions; then, a low-dimensional semantic embedding vector is generated through linear mapping. : in, This represents the weight matrix of the linear projection layer. Represents the ReLU activation function. These represent the batch size, height, and width of the input feature map, respectively. Represents the bias vector of the linear projection layer; A cyclic gating unit is introduced for decision-making, which involves storing the hidden state from the previous time step. As a decision memory, it is introduced into the expert selection process of the current frame to carry the constraints of historical semantic distribution on the compression strategy; the gating logic of the cyclic gating unit and its hidden state update equation are as follows: in, , These represent resetting the door and updating the door, respectively. The state is hidden for the decision at the current moment; For activation function, To reset the input weight matrix of the gate, To reset the cyclic weight matrix of the gate, To reset the gate's bias vector; To update the input weight matrix of the gate, To update the cyclic weight matrix of the gate, To update the bias vector of the gate; This represents the current candidate hidden state. The hyperbolic tangent activation function is used. The input weight matrix represents the candidate states. Here is the cyclic weight matrix for the candidate states. The bias vector for the candidate state; Finally, the hidden state is processed using the Softmax function. After normalization, the contribution weights of each expert are obtained. : In the formula, the subscript " "Indicates time" Next An index corresponding to each expert.

4. The lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 3, characterized in that: Step S2 also includes building a hybrid expert pool and providing online experience feedback, the process of which is as follows: The expert pool consists of three static projection experts And a dynamic caching expert The system is composed of static experts, which are obtained through offline end-to-end training and are responsible for characterizing stable and universally applicable dimensionality reduction mapping relationships in the feature space; while dynamic cache experts serve as online updatable adaptation units, used to continuously absorb and characterize the personalized appearance features of the target in the current video. For the online inference phase, based on the obtained routing weights At the parameter level, multiple experts are weighted and fused; the fusion process is as follows: The optimal projection operator for the current frame is generated in real time using tensor shrinking. : in, For the fused dynamic projection kernel, For the first The convolution weight matrix of a static expert, The convolution weight matrix for dynamic caching experts; Subsequently, the dynamic projection kernel is applied to the output features of the value encoder in the form of grouped convolutions. Complete the mapping from high dimension to low dimension: in, The low-dimensional features after mapping Represents the convolution operation. The bias terms of each expert are determined based on the routing weights. The dynamically shifted bias vector synthesized by linear weighting; After inference is completed in each frame, the projection kernel synthesized in the current frame is used. right Smooth updates are performed to gradually incorporate recent observational experience: in, The inertia factor is used to control the fusion ratio between new and old experiences; it is increased in the early stages of the video sequence. Values ​​are used to quickly capture target attributes; after the model has formed a stable representation of the target, the value is reduced. To filter environmental noise.

5. A lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 4, characterized in that: In step S3, an adaptive storage compression method for segmented memory is provided, which dynamically adjusts the storage strategy based on the information distribution of local features. Specifically, for regions with gentle feature changes and high semantic consistency, features are compressed through aggregation operations; for regions with large feature differences and clear semantic boundaries, representative feature responses are selectively retained. The adjustment process first defines a metric that reflects the degree of local feature change, and then executes a key-value synchronous hybrid compression strategy based on a decision threshold. The Key feature is used as the basis for calculating local information density, and the process is as follows: The key feature map is divided into several non-overlapping parts in spatial dimension. Local windows, each window corresponds to a spatial location index. For any local window Its interior contains Key feature vectors ; The mean of the eigenvectors within the window is calculated as a local reference center: Then, the average Euclidean dispersion is introduced as an indicator to measure the information complexity within the window, which is calculated by the degree of deviation of each key feature within the window from the mean vector: The decision threshold is set using a dynamic quantile thresholding method, and the dispersion distribution of the key features across the entire image is statistically analyzed. Using the dynamic quantile threshold method and according to a preset ratio Calculate the decision threshold : in, Quantity ratio; Construct the corresponding binary decision mask This is used to indicate the compression strategy used by different windows: in, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise.

6. A lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 5, characterized in that: Step S3 also includes an information density mask. The controlled dual-branch hybrid compression structure uses key features as a reference for spatial sampling, determines the compression method and sampling location based on the local statistical information of the key features, and synchronously applies the corresponding rules to the value features, wherein: for In the region, a pooling-based aggregation-based compression method is used to simplify the features, resulting in the compressed feature representation. Obtained from the mean of features within a local window: for In the region, a selective sampling strategy based on maximum deviation is adopted to preserve key semantic information by retaining the feature vectors that differ most from the local mean. The compression characteristics of highly discrete regions are then expressed as: ; Finally, through information density masking By selectively combining the two compression paths described above, the final compressed feature representation is obtained. .

7. A lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 6, characterized in that: In step S4, memory features are quickly filtered before global attention computation is performed based on a sparsity strategy for the memory retrieval stage. First, a sparsity reduction method based on group processing is implemented: the flattened long-term memory key sequence is... Divide the sequence along its dimensions into several local subsequences, each subsequence containing A series of consecutive tokens, and select from them. A representative token is proportionally compressed; let the first... The local subsequences are represented as L2 normalization is performed on all tokens within the group to obtain a set of unit vectors. ; Subsequently, using the linear superposition property of vector algebra, the eigenvalues ​​and vectors of this set are calculated. : The relevance score of each token is calculated using a single dot product operation. : Perform a score-based selection operation on each subsequence to obtain the corresponding set of locally retained indices: To avoid disrupting the sequence structure and maintain temporal consistency, selected local indices are sorted in ascending order and mapped back to their original sequence positions, thereby generating a global sparse mask. For sequences ending with less than a complete group of tokens, all are retained without further filtering. Based on the generated sparse mask, the key sequences, value sequences, and related auxiliary items in the memory are simultaneously pruned to construct a sparsified memory. , .

8. A lightweight video target segmentation method based on sparse memory and cyclic dynamic feature compression according to claim 7, characterized in that: In step S5, the decoder adopts a progressive upsampling architecture based on feature pyramid logic to restore the high-dimensional readout features retrieved from the memory space into an accurate pixel-level target mask; through skip connections, the organic fusion key encoder retains multi-scale shallow spatial features in the downsampling stage, and uses a series of feature fusion operators to integrate the readout semantic information with the underlying texture details.

Images