Referential image segmentation method and system based on sparse bidirectional state space adapter
By setting a state space adaptation module and a cross-modal attention mechanism in the visual coding network, the problem of freezing the complementary context of image patch feature propagation in the visual coding network is solved, achieving efficient indexed expression image segmentation and improving segmentation accuracy and parameter utilization efficiency.
Patent Information
- Application Number
- CN202611585783.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
In image segmentation with efficient parameter representation of frozen visual coding networks and text coding networks, how can we preserve the adaptability of local spatial structure, enable spatially separated image patch features to propagate complementary context, utilize natural language representation to filter the propagated information, and avoid new adaptation branches perturbing the frozen visual representation in the initial training stage?
By setting up a state space adaptation module in some feature processing stages of the visual encoding network and retaining a local space feature adaptation module in other stages, state space feature propagation is performed in the opposite sequence direction, bidirectional contextual visual features are fused, and residual features are modulated using learnable gating parameters. Information filtering is then performed in conjunction with a cross-modal attention mechanism.
It achieves the complementary effect of cross-image patch context propagation and local spatial structure adaptation, reduces the loss of orientation information, avoids the perturbation of frozen visual representation by new adaptation branches, reduces the proportion of training parameters, and improves the efficiency and accuracy of image segmentation.
Smart Images

Figure CN123071663A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image segmentation technology, and particularly relates to a method and system for reference image segmentation based on a sparse bidirectional state space adapter. Background Technology
[0002] Naming-based image segmentation is used to identify the target object indicated by a natural language narration in an image and output a pixel-level segmentation mask for that target object. Unlike semantic segmentation based solely on preset categories, naming-based image segmentation also needs to consider object attributes, spatial relationships, and scene context to establish a pixel-level correspondence between visual features and textual features.
[0003] To reduce the training cost of large-scale pre-trained models on specific segmentation tasks, the visual encoding network and text encoding network can be frozen, with a small number of trainable adaptation modules placed between them. Existing parameter-efficient schemes typically utilize convolutional operations to preserve local structure and multi-scale spatial information, and employ attention mechanisms to establish cross-modal connections. However, in frozen visual backbones that primarily use convolutional visual adaptation for spatial feature adjustment, there is still room for improvement in long-distance context propagation between intermediate image patch features. When the target object needs to be determined using spatially separated reference objects, scene-level locations, or relationships between multiple similar objects, relying solely on local adaptation may be insufficient to adequately convey the relevant context.
[0004] In recent years, schemes based on State Space Models (SSMs) have begun to be introduced into vision tasks. Some literature proposes using bidirectional state space models as a general visual backbone to replace traditional Transformer backbone networks. However, the architecture design of using bidirectional state space models as the backbone network itself requires replacing the entire visual encoder network, making it impossible to embed existing frozen models as lightweight adaptable modules. Other literature proposes introducing window Mamba fusion into image segmentation tasks, building a neck module after the visual encoder for intermediate fusion of multi-scale features. However, this approach places the state space model after the encoder for fusion, without supplementing cross-image patch context propagation within the frozen visual backbone for each feature processing stage.
[0005] Replacing all existing local convolutional adaptation modules with global sequence modeling modules might weaken the spatial priors useful for boundaries and local structures. Furthermore, if the newly configured adaptation branch directly injects non-zero residuals into the frozen visual features at the start of training, it might alter the initial mapping of the pre-trained representation. Therefore, a technical solution is needed that can preserve local spatial adaptation capabilities while supplementing cross-image patch context propagation within the backbone with fewer trainable parameters, combining index expressions to filter relevant information, and controlling residual injection. Summary of the Invention
[0006] The technical problem this invention aims to solve is: in image segmentation with efficient parameter representation of frozen visual coding networks and text coding networks, how to preserve the adaptability of local spatial structure while enabling spatially separated image patch features to propagate complementary context, and how to use natural language representation to filter the propagated information, and how to avoid new adaptation branches perturbing the frozen visual representation in the initial training stage.
[0007] To address the aforementioned technical problems, this invention provides a method for referential image segmentation based on a sparse bidirectional state space adapter.
[0008] S1. Obtain the image to be processed and a natural language expression for indicating the target object in the image to be processed.
[0009] S2. Input the image to be processed into a visual coding network for visual feature extraction to obtain a visual feature sequence including multiple image block features, and input the natural language index into a text coding network for text feature extraction to obtain a text feature sequence.
[0010] S3. Set a state space adaptation module in some feature processing stages of the multiple feature processing stages of the visual coding network, and retain a visual adaptation module for local spatial feature adaptation in at least some feature processing stages where the state space adaptation module is not set.
[0011] S4. In the feature processing stage where the state space adaptation module is set, the image block features are mapped to the adaptation feature space, and state space feature propagation is performed along the first sequence direction and the second sequence direction opposite to the first sequence direction to obtain the first direction state space features and the second direction state space features. The first direction state space features and the second direction state space features are then fused to obtain bidirectional contextual visual features.
[0012] S5. Based on the text feature sequence, perform cross-modal feature interaction on the bidirectional contextual visual features to obtain text conditional visual features related to the expression of the natural language index.
[0013] S6. Based on the text conditional visual features, residual features are formed, and the residual features are modulated using learnable gating parameters. The modulated residual features are then added to the corresponding visual feature sequence to obtain the adapted visual features.
[0014] S7. Generate a target object segmentation mask based on the adapted visual features, representing the target object in natural language.
[0015] Further preferably, the multiple feature processing stages are provided with multiple preset visual adaptation positions; the state space adaptation module is set in some of the preset visual adaptation positions from the middle to the rear of the visual encoding network, and the number of state space adaptation modules is less than the total number of preset visual adaptation positions; the remaining at least one preset visual adaptation position is provided with the visual adaptation module, and the visual adaptation module includes a convolutional visual adaptation module based on multi-scale convolution operations.
[0016] Further preferably, the visual feature sequence includes non-image patch marker features and image patch features; the non-image patch marker features bypass the state space propagation branch of the state space adaptation module and remain unchanged, while the state space adaptation module generates text conditional residual features only for the image patch features; the unchanged non-image patch marker features and the adapted image patch features together form the adapted visual feature sequence. Since non-image patch marker features (such as category markers and register markers) carry global semantic information in the visual Transformer, bypassing the state space propagation branch avoids the loss of global semantic priors during the adaptation process.
[0017] Further preferably, the state space feature propagation performed along the first sequence direction and the second sequence direction respectively includes: The image patch features are subjected to layer normalization, and then the learnable dimensionality reduction matrix is used to map the layer normalized image patch features from the feature channel dimension of the visual coding network to the adaptive feature space dimension to obtain compact visual features. The adaptive feature space dimension is lower than the feature channel dimension of the visual coding network. Maintaining the spatial raster scanning order corresponding to the image block features, the compact visual features are input into the first state space model and propagated along the first sequence direction to obtain the first direction state space features. The sequence order of the compact visual features is reversed, the reversed compact visual features are input into the second state space model for state propagation, and the output of the second state space model is restored to the original sequence order to obtain the second direction state space features.
[0018] Further optimized, the first state space model and the second state space model have independent model parameters; the state space features of the first direction and the state space features of the second direction are concatenated along the feature channel dimension, and the concatenated features are mapped back from twice the adaptive feature space dimension using a learnable mapping matrix to obtain fused bidirectional features; the fused bidirectional features are residually added to the compact visual features before propagation of the state space features, and then the residual addition result is subjected to layer normalization to obtain the bidirectional contextual visual features. Using two Mamba blocks with independent parameters to process the state propagation in the two directions can avoid directional aliasing of the same set of parameters in bidirectional propagation; residual connection between the bidirectional features and the input before propagation can preserve the original local information.
[0019] Further preferably, the cross-modal feature interaction includes: mapping the text feature sequence to key features and value features in the same adapted feature space as the bidirectional contextual visual features using a learnable key projection matrix and a learnable value projection matrix, respectively; using the bidirectional contextual visual features as query features, and the key features and value features as key and value inputs for multi-head cross-modal attention operations, performing multi-head cross-modal attention operations to obtain attention output features; adding the attention output features and the bidirectional contextual visual features by residual addition, and then performing layer normalization on the residual addition result to obtain the text conditional visual features. Through cross-modal feature interaction, the attention mechanism allows the context propagated by each image patch to be filtered according to natural language indices, retaining only information related to the indicative target.
[0020] More preferably, step S6 includes: By using a learnable upprojection matrix, the text conditional visual features corresponding to the image patch features are mapped from the adaptive feature space to the feature channel dimension of the visual coding network to obtain the image patch residual features. Set learnable channel gating parameters that correspond one-to-one with each feature channel of the visual coding network, and use the learnable channel gating parameters to modulate the residual features of the image block channel by channel; Construct a residual sequence in which the residual values at non-image block marker feature positions are zero, and the residual values at image block feature positions are the image block residual features after channel-by-channel modulation; The residual sequence is added element by element to the input visual feature sequence of the state space adaptation module to obtain the adapted visual features.
[0021] Further preferably, the learnable channel gating parameter is initialized to zero at the start of model training, so that the residual sequence is zero in the initial state and the adapted visual feature sequence output by the state space adaptation module is strictly equal to its input visual feature sequence; during model training, the learnable channel gating parameter is updated as a parameter to be trained, so that task-related image patch residual features are injected into the visual feature sequence.
[0022] Further preferably, the visual encoding network includes twelve visual feature processing blocks. The state space adaptation module is set in the sixth, eighth, and tenth visual feature processing blocks, and a visual adaptation module based on multi-scale convolution operations is set in the second, fourth, and twelfth visual feature processing blocks. The adaptation feature space dimension of the state space adaptation module is 128. The state space feature propagation in the first sequence direction and the second sequence direction is each executed by a state space model block. The model parameters of the two state space model blocks are independent of each other, with a state dimension of 16 and a local convolution width of 4. The output of the text encoding network is processed by a trainable text adaptation module to form the text feature sequence. The target object segmentation mask is generated by a segmentation decoding module. During model training, the parameters of the visual encoding network and the text encoding network are kept frozen, and the visual adaptation module, the state space adaptation module, the text adaptation module, and the segmentation decoding module are trained.
[0023] The present invention also provides a referential image segmentation system based on a sparse bidirectional state-space adapter, comprising: The input acquisition module is used to acquire the image to be processed and a natural language expression for indicating the target object in the image to be processed. The feature encoding module includes a visual encoding network and a text encoding network. The visual encoding network is used to extract visual features from the image to be processed to obtain a visual feature sequence including features of multiple image patches. The text encoding network is used to extract text features from the natural language index to obtain a text feature sequence. A state space adaptation module is set in a partial feature processing stage of the visual coding network. It is used to map the image block features to the adaptation feature space, perform state space feature propagation along a first sequence direction and a second sequence direction opposite to the first sequence direction, and fuse the obtained first direction state space features and second direction state space features to obtain bidirectional contextual visual features. A visual adaptation module is set in at least a portion of the feature processing stage where the state space adaptation module is not set, and is used to perform local spatial feature adaptation. A cross-modal fusion module is used to perform cross-modal feature interaction on the bidirectional contextual visual features based on the text feature sequence to obtain text conditional visual features; The gated residual update module is used to form residual features based on the text conditional visual features, modulate the residual features using learnable gating parameters, and add the modulated residual features to the corresponding visual feature sequence to obtain the adapted visual features. The segmentation prediction module is used to generate a target object segmentation mask based on the adapted visual features, which represents the natural language instruction to indicate the target object.
[0024] The present invention also provides an electronic device, including a processor and a memory communicatively connected to the processor; the memory stores a computer program executable by the processor, which, when executed by the processor, causes the processor to perform the above-described method.
[0025] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.
[0026] The present invention has the following advantages: 1. By setting the state space adaptation module only in some visual feature processing stages and retaining the convolutional visual adaptation module in other stages, cross-image patch context propagation and local spatial structure adaptation can complement each other in the same frozen visual backbone, avoiding the complete removal of local multi-scale adaptation.
[0027] 2. Performing state space propagation along the opposite sequence direction and fusing features from both directions allows each image block to obtain complementary context from the two scanning directions before entering text condition filtering, reducing the loss of directional information caused by a single scanning direction.
[0028] 3. By using bidirectional contextual visual features as queries and text features as keys and values to perform cross-modal attention, the propagated visual context can be filtered according to natural language indication, reducing the direct injection of context unrelated to the target into the visual sequence.
[0029] 4. Modulate the image patch residuals using zero-initialized channel-level gating parameters and pad with zeros at non-image patch marker positions to ensure that the state space adaptation module maintains an accurate identity mapping during initialization; learn the gating parameters again during training to allow task-related corrections to be injected.
[0030] 5. During model training, only the visual adaptation module, state space adaptation module, text adaptation module, and segmentation / decoding module are trained, while the visual encoding network and text encoding network remain frozen. The trainable parameters account for a small proportion of the entire model, achieving efficient parameter-based image segmentation. Attached Figure Description
[0031] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.
[0032] Figure 2 This is a diagram illustrating the overall structure of the Dyda image segmentation system of this invention.
[0033] Figure 3 This is a structural diagram of the sparse bidirectional state space adaptation module of the present invention.
[0034] Figure 4 This is a flowchart of the bidirectional state-space feature propagation process of the present invention.
[0035] Figure 5 This is a flowchart of the text-guided cross-modal fusion and gated residual update process of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0037] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.
[0038] Example 1 This embodiment provides a method for referential image segmentation based on a sparse bidirectional state-space adapter, such as... Figure 1 As shown, the method includes steps S1 to S7.
[0039] S1. Obtain the image to be processed and a natural language expression for indicating the target object in the image to be processed.
[0040] The image to be processed is the image from which pixel-level segmentation masks need to be generated. Its format can be common image formats such as JPEG (Joint Image Experts Group image format) or PNG (Portable Network Graphics format). The natural language indicator is a text description containing information used to distinguish the category, attributes, location, or object relationships of the target object, such as "the yellow umbrella on the right" or "the little brown dog near the bench." The image to be processed and the natural language indicator constitute an input pair, provided by the user or an upstream system.
[0041] S2. Input the image to be processed into a visual coding network for visual feature extraction to obtain a visual feature sequence including multiple image block features, and input the natural language index into a text coding network for text feature extraction to obtain a text feature sequence.
[0042] Visual encoding networks can employ pre-trained models based on the Vision Transformer (ViT) architecture, such as DINOv2 (a self-supervised pre-trained visual Transformer model) or the visual branch of CLIP (a contrastive language-image pre-trained model). The visual encoding network divides the image to be processed into multiple image patches. Each image patch is linearly projected to form image patch features, and all image patch features are arranged according to their spatial positions to form a visual feature sequence. In addition to image patch features, the visual feature sequence also includes non-image patch labeled features, such as category labels and register labels, which carry global semantic information.
[0043] Text encoding networks can employ pre-trained language models based on the Transformer architecture (a neural network architecture based on self-attention mechanisms), such as the text branch of CLIP. Text encoding networks encode natural language tokens into a sequence of text features, where each text token corresponds to a feature vector.
[0044] In one specific implementation, the visual encoding network uses a frozen DINOv2ViT-B / 14, containing twelve visual feature processing blocks with a feature channel dimension C of 768; where ViT represents the visual Transformer, and B / 14 represents the base size and image patch size of 14×14. The text encoding network uses a frozen CLIP ViT-B / 16 text branch with a text feature channel dimension of 512, where B / 16 represents the base size and image patch size of 16×16. The above model is a reproducible experimental configuration, and other pre-trained visual encoding networks and text encoding networks are also applicable.
[0045] The stage-by-stage transmission of visual features in the frozen skeleton is represented by formula (1): (1) in, The first character of the visual coding network One frozen feature processing block, This represents the sequence of adaptive visual features from the previous stage. This indicates the output of the currently frozen block. Indicates the length of the visual sequence. Indicates the visual feature channel dimension. For feature dimensions.
[0046] The text feature sequence is obtained according to formula (2): (2) in, Natural language refers to the expression of meaning. This indicates a frozen text encoding network. This indicates a text adaptation module. Indicates the length of the text sequence. Indicates the channel dimension of text features. This represents the text feature sequence after text adaptation.
[0047] S3. Set a state space adaptation module in some feature processing stages of the multiple feature processing stages of the visual coding network, and retain a visual adaptation module for local spatial feature adaptation in at least some feature processing stages where the state space adaptation module is not set.
[0048] The multiple feature processing stages in a visual coding network refer to the positions between layers or feature processing blocks within the network. For example, in a visual coding network containing twelve visual feature processing blocks, each block can serve as a candidate location for setting an adaptation module. This step does not set the same adaptation module at all candidate locations, but rather sets the state-space adaptation module only at some of the candidate locations, while retaining the visual adaptation module at at least some of the remaining candidate locations.
[0049] The visual adaptation module is used for local spatial feature adaptation and can employ a convolutional visual adaptation module (i.e., a visual adaptation module based on multi-scale convolution operations) to preserve local spatial structure and multi-scale spatial information. The state-space adaptation module is used to supplement long-distance context propagation across image patches at selected stages.
[0050] No. The adaptation operation used in this stage is represented by formula (3): (3) in, Indicates the first The adaptation operation used in each stage depends on the stage position: it is either a convolutional visual adaptation module, a state space adaptation module, or an identity mapping. Represents a sequence of text features; This represents the adapted visual feature sequence.
[0051] In one specific implementation, multiple preset visual adaptation positions are provided in multiple feature processing stages, numbered starting from 1. State space adaptation modules are located at some of the preset visual adaptation positions from the middle to the end of the visual encoding network, such as positions 6, 8, and 10, and the number of such modules is less than the total number of preset visual adaptation positions. At least one other preset visual adaptation position, such as positions 2, 4, and 12, is equipped with a convolutional visual adaptation module. This sparse hybrid configuration has been verified in controlled ablation experiments to be superior to either complete replacement or uniformly sparse configurations.
[0052] The specific insertion set is represented by formula (4): (4) in, This indicates the set of visual feature processing block numbers for the state space adaptation module in a specific embodiment. This layer is an experimental implementation under the DINOv2 ViT-B / 14 and DETRIS-B (a reference to an image segmentation baseline model) configuration, and does not constitute a limitation on the insertion position of other visual backbones.
[0053] S4. In the feature processing stage where the state space adaptation module is set, the image block features are mapped to the adaptation feature space, and state space feature propagation is performed along the first sequence direction and the second sequence direction opposite to the first sequence direction to obtain the first direction state space features and the second direction state space features. The first direction state space features and the second direction state space features are then fused to obtain bidirectional contextual visual features.
[0054] Let the input visual feature sequence of the current state space adaptation module be... The visual feature sequence includes non-image patch marker features and image patch features. The splitting of the visual feature sequence is represented by formula (5): (5) in, Indicates the first A sequence of visual features for each state-space adaptation position express Non-image patch labeling features, express Each image patch feature is represented by a semicolon indicating a combination along the sequence dimension. Non-image patch marker features are propagated around the state space branch and remain unchanged; the state space adaptation module processes only image patch features.
[0055] The serial operations of compact mapping, bidirectional propagation, text-guided fusion, and gated residual injection performed by the entire state space adaptation module are represented by formula (6): (6) in, For the adapted visual features, This represents the serial operations performed by the state space adaptation module, including compact mapping, bidirectional propagation, text-guided fusion, and gated residual injection.
[0056] like Figure 4 As shown, the specific processing procedure for step S4 is as follows: S4-1, Dimensional reduction mapping to obtain compact visual features.
[0057] Image patch features The input state space adaptation module first performs layer normalization on the image patch features to stabilize the feature distribution during training. Then, a learnable dimensionality reduction matrix is used to map the normalized features from the feature channel dimension C of the visual encoding network to the adaptation feature space dimension D, obtaining compact visual features. The feature space dimension D of the adaptive network is smaller than the feature channel dimension C of the visual encoding network, thereby reducing the computational complexity of subsequent state space propagation.
[0058] Specifically, the compact visual features are obtained according to formula (7): (7) in, Presentation layer normalization operation, This is a learnable dimensionality reduction matrix that maps from C-dimensional to D-dimensional. D can be 128. The compact visual features after dimensionality reduction preserve the spatial order of the image patches, i.e., arranged according to the raster scan order.
[0059] S4-2, Perform state space feature propagation along the first sequence direction.
[0060] Preserving the raster scan order corresponding to image patch features, compact visual features The first state-space model is input and propagated along the first sequence direction to obtain the state-space features in the first direction. Specifically, the state space characteristics in the first direction are obtained according to formula (8): (8) in, Mamba represents a state space model (SSM) that is the first state space model. It employs a selective scanning mechanism of the state space model, which can model long-distance dependencies between sequence elements within linear computational complexity.
[0061] S4-3, Perform state space feature propagation along the second sequence direction.
[0062] Compact visual features The sequence order is reversed. The reversed, compact visual features are then input into the second state space model. State propagation is performed. Since the input sequence has been reversed, this state propagation is equivalent to proceeding along a second sequence direction opposite to the first sequence direction. After state propagation is complete, the output of the second state-space model is reversed again to restore the original sequence order, thus obtaining the state-space features in the second direction. Specifically, the state space characteristics in the second direction are obtained according to formula (9): (9) in, This represents a sequence reversal operation. This is the second state space model. The first and second state space models have independent model parameters, meaning that propagation in two directions is executed by two independent state space model blocks respectively. This avoids directional aliasing caused by the same set of parameters during bidirectional propagation, allowing each direction to learn its own directional context pattern. In one specific implementation, the state dimension of each state space model block is 16, and the local convolution width is 4.
[0063] S4-4, Fusion of bidirectional state space features.
[0064] The first direction state space features Second-direction state space features The features are concatenated along the feature channel dimension to obtain the concatenated features. The concatenated features are 2D in dimension. A learnable mapping matrix is then used. The concatenated features are mapped from 2D dimension back to a D-dimensional adapted feature space to obtain the fused bidirectional features. Specifically, the fused bidirectional features are obtained according to formula (10): (10) The semicolon indicates a concatenation operation along the feature channel dimension. To map 2D concatenated features back to a D-dimensional learnable matrix.
[0065] The fused bidirectional features Compact visual features before propagation of state space features The residuals are summed, and then the summed results are subjected to layer normalization to obtain the bidirectional contextual visual features. Specifically, the bidirectional contextual visual features are obtained according to formula (11): (11) in, This represents the bidirectional features after fusion. This represents bidirectional contextual visual features. Residual connections are used to ensure that the output of state space propagation retains the original local information before propagation, avoiding the loss of spatial priors useful for boundaries and local structures during propagation; layer normalization is used to stabilize the feature distribution of subsequent cross-modal interactions.
[0066] S5. Based on the text feature sequence, perform cross-modal feature interaction on the bidirectional contextual visual features to obtain text conditional visual features related to the expression of the natural language index.
[0067] like Figure 5 As shown, the specific processing procedure for step S5 is as follows: S5-1. Map the text feature sequence to key features and value features.
[0068] Text feature sequence Each is projected into a key feature that resides in the same adapted feature space as the bidirectional contextual visual features. Sum value characteristics Specifically, using a learnable key projection matrix and learnable projection matrix Perform linear transformations on the text features respectively: (12) in, It is a text feature sequence.
[0069] S5-2, Perform multi-head cross-modal attention computation.
[0070] With bidirectional contextual visual features As query features, the mapped text key features Sum value characteristics Each as a key Sum Perform multi-head cross-modal attention operations. Attention output features. According to formula (13): (13) in, This refers to multi-head attention, a cross-modal attention mechanism that captures the alignment between visual and textual information from multiple subspaces, enhancing the expressive power of cross-modal interaction. In one specific implementation, eight-head attention is employed, with a dropout rate of 0.1 for the attention output projection.
[0071] S5-3, Residual summation and layer normalization.
[0072] Pay attention to output features With bidirectional contextual visual features The residuals are summed, and then the summed residuals are subjected to layer normalization to obtain the text conditional visual features. Specifically, the text conditional visual features are obtained according to formula (14): (14) in, This represents the textual conditional visual features after residual summation and layer normalization.
[0073] S6. Based on the text conditional visual features, residual features are formed, and the residual features are modulated using learnable gating parameters. The modulated residual features are then added to the corresponding visual feature sequence to obtain the adapted visual features.
[0074] The detailed process of step S6 in this embodiment is as follows: S6-1. Obtain image patch residual features by upprojection.
[0075] Using learnable upprojection matrices The textual conditional visual features corresponding to the image patch features are mapped from the adaptation feature space dimension D back to the feature channel dimension C of the visual encoding network to obtain the image patch residual features. Specifically, the image patch residual features are obtained according to formula (15): (15) in, The learnable upprojection matrix is used to map from the adaptive feature space dimension D back to the feature channel dimension C.
[0076] S6-2, Channel door control.
[0077] Set learnable channel gating parameters that correspond one-to-one with each feature channel of the visual encoding network. (Also referred to as learnable gating parameter), with dimension C. This gating parameter is used to analyze the residual features of image patches. Channel-by-channel modulation involves multiplying the features of each channel by a corresponding gate value. This modulation method applies different degrees of correction to different channels of the features, allowing the model to adaptively determine which information from each channel needs to be injected into the visual feature sequence. The channel gating parameters can be learned. Initialize it as a zero vector at the start of model training.
[0078] S6-3, Construct the residual sequence.
[0079] Construct a residual sequence Its sequence length is the same as the input visual feature sequence of the state space adaptation module. The same. Specifically, the residual value at non-image patch labeled feature locations in the residual sequence is zero, i.e., the size is... The zero matrix; the residual values at the feature locations of the image blocks are the residual features of the image blocks after channel-by-channel modulation. The residual sequence is obtained according to formula (16): (16) in, for Zero residual matrix at non-image patch marker locations, For learnable channel gating parameters, Indicates in Channel-by-channel multiplication broadcast for each image patch location The number of features labeled for non-image patches. This represents the number of image patch features. Non-image patch labeled features carry global semantic information in the visual Transformer; setting their residual positions to zero can prevent the loss of global semantic priors during the adaptation process.
[0080] S6-4, Residual Injection.
[0081] residual sequence Input visual feature sequence with state space adaptation module Add elements one by one to obtain the adapted visual features. Specifically, the adapted visual features are obtained according to formula (17): (17) when When initialized to a zero vector, the residual features of the image patch after channel-by-channel modulation are zero, and the residual sequence... The zero matrix represents the adapted visual features. Equal to the input visual feature sequence The state-space adaptation module maintains an exact identity mapping during initialization, without perturbing the output of the frozen visual encoding network. During model training, As trainable parameters, the residual correction of image patches related to the injection task is gradually learned from scratch and updated progressively.
[0082] S7. Generate a target object segmentation mask based on the adapted visual features, representing the target object in natural language.
[0083] The adapted multi-level visual features are collected and input into the segmentation and decoding module. The segmentation and decoding module typically adopts a Transformer-based decoder architecture, which can fuse multi-level visual features with text representations and output a pixel-level segmentation mask corresponding to the spatial resolution of the input image.
[0084] Specifically, the segmentation decoding module receives adapted visual features from multiple feature processing stages, such as features after the 5th and 9th visual feature processing blocks and the final 12th visual feature processing block. Internally, the segmentation decoding module includes multiple layers of cross-attention layers and a feedforward network. It restores the feature map to the original image resolution through upsampling operations and predicts whether each pixel belongs to the target object or the background, ultimately generating a binary segmentation mask. In one specific implementation, the segmentation decoding module includes three layers, eight attention heads, a 512-dimensional feedforward layer, and a dropout rate of 0.1. The insertion positions of the adaptation module (blocks 6, 8, and 10) and the feature acquisition positions of the decoding module (blocks 5, 9, and 12) represent different processing positions.
[0085] In one specific experimental configuration, the visual encoding network used was a frozen DINOv2ViT-B / 14, containing twelve visual feature processing blocks with a feature channel dimension C of 768. The images to be processed were scaled while maintaining aspect ratio and zero-padded to 448×448 pixels, and normalized using the mean and standard deviation of CLIP. The image block size was 14×14 pixels, thus forming 32×32 image blocks. The value is 1024; the visual sequence also includes a category label and four register labels. The value is 5, and the total sequence length is 1029. During training, one associated representation is randomly selected for each pointed object, and no geometric or photometric enhancements are added except for affine scaling and zero padding of the baseline.
[0086] The adaptation feature space dimension D of the state space adaptation module is 128, meaning the adaptation feature dimension is 128. State space feature propagation in the first and second sequence directions is performed by a separate state space model block. The model parameters of the two state space model blocks are independent, both with a state dimension of 16, a local convolution width of 4, a spread factor of 2, and an automatic dt rank of 8, where the automatic dt rank represents the rank of the automatic time step parameter in the state space model. The convolutional visual adaptation module uses standard convolution operations such as 3×3 depthwise separable convolution, and its adaptation feature dimension is consistent with that of the state space adaptation module.
[0087] The text encoding network uses a frozen CLIP ViT-B / 16 text branch, with a text feature channel dimension of [missing information]. The value is 512. Natural language refers to the truncation or padding of characters. Each text tag is used. The output of the text encoding network is processed by a trainable text adaptation module to form a text feature sequence. The text adaptation module adopts a structure similar to that of the visual adaptation module.
[0088] During model training, the parameters of the visual encoding network and the text encoding network are kept frozen, and only the convolutional visual adaptation module, the state space adaptation module, the text adaptation module, and the segmentation and decoding module are trained.
[0089] The training objective employs a text-to-pixel contrastive loss. The segmentation and decoding module generates cross-modal pixel features. Sentence-level text representation It is obtained after pooling and projection transformation. Let... This represents the set of positive pixels belonging to the target object. This represents the set of negative pixels belonging to the background. The overall training loss is obtained according to formula (18): (18) in, For the overall training loss, Indicates the first Cross-modal pixel features of individual pixels, This represents sentence-level text representation. For the first The pixel-by-pixel contrast loss for the nth pixel. The pixel-by-pixel contrast loss is obtained according to formula (19): (19) in, This represents the Sigmoid function. Represents cross-modal pixel features Sentence-level text representation The inner product of the two. The loss for positive pixels minimizes the negative logarithm of their similarity probability to the sentence-level text representation, and the loss for negative pixels minimizes the negative logarithm of their dissimilarity probability to the sentence-level text representation.
[0090] In one specific training configuration, the Adam (Adaptive Moment Estimator) optimizer is used, with an initial learning rate of... The weight decay was set to 0, the global training batch size to 32, and training was performed for 65 epochs. The learning rate was multiplied by 0.1 in epochs 35 and 55. Mixed precision training was enabled, the random seed was set to 0, and the validation batch size was 64. The experiment used two NVIDIA GeForce RTX 4090 GPUs, each with 24GB of VRAM. NVIDIA represents the graphics processor manufacturer, and GeForce RTX 4090 represents a graphics processor model. Distributed data parallelism was used. The above training parameters and hardware constitute a reproducible example.
[0091] During inference, a single image to be processed and a natural language instruction are input into the trained model. The visual encoding network generates adapted multi-level visual features, and the segmentation decoding module combines the text representation to output a target object segmentation mask that is spatially aligned with the input image.
[0092] This embodiment uses the publicly available RefCOCO, RefCOCO+, and G-Ref indexed representation image segmentation benchmark datasets for experiments. All three datasets are constructed based on MSCOCO (Microsoft Universal Object Context Dataset) images and are trained and evaluated according to the standard data partitioning provided by the publicly available benchmarks, without re-partitioning the datasets. Specifically, RefCOCO and RefCOCO+ are evaluated using val, testA, and testB partitions according to the standard protocol; G-Ref is evaluated using val(u) and test(u) partitions according to the UMD protocol (a dataset evaluation protocol), and val(g) partitions according to the Google protocol (a dataset evaluation protocol). val, testA, testB, val(u), test(u), and val(g) all represent dataset partitions. The evaluation metric is mIoU (mean Intersection over Union), which is the average of the intersection over union ratios calculated between the predicted mask and the ground truth mask for each indexed representation.
[0093] In controlled comparisons with the same training and evaluation configurations as the DETRIS-B baseline, the values of val, testA, and testB for RefCOCO improved from 75.52, 77.51, and 72.92 to 76.83, 78.56, and 74.68, respectively; the corresponding three splits for RefCOCO+ improved from 68.35, 73.42, and 60.93 to 69.91, 74.50, and 62.65, respectively; and the values of val(u), test(u), and val(g) for G-Ref improved from 67.35, 67.55, and 65.35 to 68.01, 68.66, and 66.47, respectively. The results of these controlled comparisons are shown in Table 1.
[0094] Table 1. Comparison of mIoU under controlled reproduction settings
[0095] This table only lists the results of controlled comparisons under the same training and evaluation configurations, with mIoU as the evaluation metric.
[0096] Under the same dual RTX4090, 448×448 input, precision settings, and inference protocol, the DETRIS-B reproduction baseline has 2.71 million trainable parameters, accounting for 1.81% of the entire model, with a computational cost of 124.6 GFLOPs, a peak VRAM of 7.85 GB, an inference speed of 15.30 FPS, and a RefCOCO-verified mIoU of 75.52. The sparse bidirectional state-space adaptation scheme has 3.46 million trainable parameters, accounting for 2.31% of the entire model, with a computational cost of 131.2 GFLOPs, a peak VRAM of 8.28 GB, an inference speed of 14.65 FPS, and a verified mIoU of 76.83. Therefore, this specific embodiment trades 0.75 million additional trainable parameters, 6.6 GFLOPs, 0.43 GB of additional peak VRAM, and a speed decrease of 0.65 FPS for a 1.31-point improvement in mIoU. The results show that the method of the present invention improves segmentation accuracy with an efficient parameter setting of 2.31% of trainable parameters, while introducing certain computational and storage overhead. A comparison of specific parameter efficiency and computational overhead is shown in Table 2.
[0097] Table 2 Comparison of parameter efficiency and computational cost
[0098] Measured under the same dual RTX 4090, input size, precision settings, and inference protocol.
[0099] In insertion position ablation, the mIoU on RefCOCO val for convolutional visual adaptation baseline, full replacement, early blocks 2 / 4 / 6, late blocks 8 / 10 / 12, uniformly sparse blocks 2 / 6 / 10, and mid-to-late blocks 6 / 8 / 10 are 75.52, 75.43, 75.12, 75.65, 75.30, and 76.83, respectively. In spatial modeling ablation, the mIoU on val for convolutional visual adaptation baseline, self-attention, forward Mamba only, backward Mamba only, and bidirectional Mamba are 75.52, 75.75, 74.85, 74.81, and 76.83, respectively. In residual update ablation, the mIoU on val for direct addition, constant scaling of 0.1, gate initialization to 1, and gate initialization to 0 are 73.50, 74.90, 73.65, and 76.83, respectively. In the label processing and fusion ablation, the mIoU on val for processing all labels, removing cross-modal attention, simple addition or multiplication fusion, and processing only image patches using cross-modal fusion are 75.20, 68.40, 72.10, and 76.83, respectively. These ablation results support the sparse insertion, bidirectional state space propagation, zero-initialization gating, and non-image patch label feature bypass and cross-modal fusion methods of this embodiment. Specific ablation experimental results are shown in Table 3.
[0100] Table 3 Ablation Experiment Results
[0101] Components not listed in each subgroup are fixed as the final configuration.
[0102] Qualitative tests include scene-level location selection for the "yellow umbrella on the right," the target-space separation reference relationship for the "small brown dog near the bench," and attribute filtering under similar interference for the "woman in the red coat." The sparse bidirectional state-space adaptation scheme can utilize direction, relationship, and attribute cues in the illustrated cases, while reducing boundary leakage or confusion with similar targets. The above qualitative cases illustrate that this embodiment can handle scene-level location cues, target-reference relationship, and similar object attribute filtering. The experimental results correspond to the dataset, hardware, input size, and training configuration given in this embodiment.
[0103] Example 2 like Figure 2 As shown, this embodiment provides a referential image segmentation system based on a sparse bidirectional state-space adapter, including: The input acquisition module is used to acquire the image to be processed and a natural language expression for indicating the target object in the image to be processed. The feature encoding module includes a visual encoding network and a text encoding network. The visual encoding network is used to extract visual features from the image to be processed to obtain a visual feature sequence including features of multiple image patches. The text encoding network is used to extract text features from the natural language index to obtain a text feature sequence. A state space adaptation module is set in a partial feature processing stage of the visual coding network. It is used to map the image block features to the adaptation feature space, perform state space feature propagation along a first sequence direction and a second sequence direction opposite to the first sequence direction, and fuse the obtained first direction state space features and second direction state space features to obtain bidirectional contextual visual features. A visual adaptation module is set in at least a portion of the feature processing stage where the state space adaptation module is not set, and is used to perform local spatial feature adaptation. A cross-modal fusion module is used to perform cross-modal feature interaction on the bidirectional contextual visual features based on the text feature sequence to obtain text conditional visual features; The gated residual update module is used to form residual features based on the text conditional visual features, modulate the residual features using learnable gating parameters, and add the modulated residual features to the corresponding visual feature sequence to obtain the adapted visual features. The segmentation prediction module is used to generate a target object segmentation mask based on the adapted visual features, representing the natural language instruction to indicate the target object. The segmentation prediction module corresponds to the segmentation decoding module in the method.
[0104] This embodiment also provides an electronic device, including a processor and a memory communicatively connected to the processor. The memory stores a computer program and model parameters executable by the processor. When the computer program is executed by the processor, it reads the image to be processed and a natural language index, calls a visual encoding network, a text encoding network, a state space adaptation module, and a segmentation decoding module to execute the aforementioned index image segmentation method based on a sparse bidirectional state space adapter, and outputs a target object segmentation mask. In one embodiment, the processor includes a GPU (Graphics Processing Unit) for performing model training or inference.
[0105] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the aforementioned referential image segmentation method based on a sparse bidirectional state-space adapter. The storage medium includes, but is not limited to, non-volatile storage media such as read-only memory, random access memory, magnetic disk, or optical disk.
[0106] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Those skilled in the art should understand that equivalent substitutions, modifications, or equivalent transformations can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications, substitutions, or transformations do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of the claims of the present invention and their equivalents.
Claims
1. A method for representing image segmentation based on a sparse bidirectional state-space adapter, characterized in that, include: S1. Acquire the image to be processed and a natural language expression for indicating the target object in the image to be processed; S2. Input the image to be processed into a visual coding network for visual feature extraction to obtain a visual feature sequence including features of multiple image blocks, and input the natural language index into a text coding network for text feature extraction to obtain a text feature sequence. S3. Set a state space adaptation module in some feature processing stages of the multiple feature processing stages of the visual coding network, and retain a visual adaptation module for local spatial feature adaptation in at least some feature processing stages where the state space adaptation module is not set. S4. In the feature processing stage where the state space adaptation module is set, the image block features are mapped to the adaptation feature space, and state space feature propagation is performed along the first sequence direction and the second sequence direction opposite to the first sequence direction to obtain the first direction state space features and the second direction state space features. The first direction state space features and the second direction state space features are then fused to obtain bidirectional contextual visual features. S5. Based on the text feature sequence, perform cross-modal feature interaction on the bidirectional contextual visual features to obtain text conditional visual features related to the natural language index expression; S6. Based on the text conditional visual features, a residual feature is formed. The residual feature is modulated using a learnable gating parameter. The modulated residual feature is then added to the corresponding visual feature sequence to obtain the adapted visual feature. S7. Generate a target object segmentation mask based on the adapted visual features, representing the target object in natural language.
2. The method for referential image segmentation based on a sparse bidirectional state-space adapter according to claim 1, characterized in that, The multiple feature processing stages are provided with multiple preset visual adaptation positions; the state space adaptation module is set in some of the preset visual adaptation positions from the middle to the rear of the visual encoding network, and the number of state space adaptation modules is less than the total number of preset visual adaptation positions; the remaining at least one preset visual adaptation position is provided with the visual adaptation module, and the visual adaptation module includes a convolutional visual adaptation module based on multi-scale convolution operations.
3. The method for referential image segmentation based on a sparse bidirectional state-space adapter according to claim 1, characterized in that, The visual feature sequence includes non-image patch marker features and image patch features; The non-image patch marker features bypass the state space propagation branch of the state space adaptation module and remain unchanged, while the state space adaptation module generates text conditional residual features only for the image patch features; The unchanged non-image patch marker features are combined with the adapted image patch features to form the adapted visual feature sequence.
4. The method for referential image segmentation based on a sparse bidirectional state-space adapter according to claim 1, characterized in that, The state space feature propagation is performed along the first sequence direction and the second sequence direction, respectively, including: The image patch features are subjected to layer normalization, and then the learnable dimensionality reduction matrix is used to map the layer normalized image patch features from the feature channel dimension of the visual coding network to the adaptive feature space dimension to obtain compact visual features. The adaptive feature space dimension is lower than the feature channel dimension of the visual coding network. Maintaining the spatial raster scanning order corresponding to the image block features, the compact visual features are input into the first state space model, and state propagation is performed along the first sequence direction to obtain the first direction state space features. The sequence order of the compact visual features is reversed, the reversed compact visual features are input into the second state space model for state propagation, and the output of the second state space model is restored to the original sequence order to obtain the second direction state space features.
5. The method for referential image segmentation based on a sparse bidirectional state-space adapter according to claim 4, characterized in that, The first state-space model and the second state-space model have mutually independent model parameters; The first direction state space features and the second direction state space features are concatenated along the feature channel dimension, and the concatenated features are mapped from twice the adaptive feature space dimension back to the adaptive feature space dimension using a learnable mapping matrix to obtain the fused bidirectional features. The fused bidirectional features are added to the compact visual features before propagation of the state space features by residual addition, and then the residual addition result is subjected to layer normalization to obtain the bidirectional contextual visual features.
6. The method for referential image segmentation based on a sparse bidirectional state-space adapter according to claim 1, characterized in that, The cross-modal feature interaction includes: The text feature sequence is mapped to key features and value features in the same adaptive feature space as the bidirectional contextual visual features using learnable key projection matrices and learnable value projection matrices, respectively. Using the bidirectional contextual visual features as query features, and the key features and value features as key and value inputs for multi-head cross-modal attention operations, multi-head cross-modal attention operations are performed to obtain attention output features; The attention output features and the bidirectional contextual visual features are added together by residual summation, and then the result of residual summation is subjected to layer normalization to obtain the text conditional visual features.
7. The method for referential image segmentation based on a sparse bidirectional state-space adapter according to claim 1, characterized in that, Step S6 includes: By using a learnable upprojection matrix, the text conditional visual features corresponding to the image patch features are mapped from the adaptive feature space to the feature channel dimension of the visual coding network to obtain the image patch residual features. Set learnable channel gating parameters that correspond one-to-one with each feature channel of the visual coding network, and use the learnable channel gating parameters to modulate the residual features of the image block channel by channel; Construct a residual sequence in which the residual values at non-image block marker feature positions are zero, and the residual values at image block feature positions are the image block residual features after channel-by-channel modulation; The residual sequence is added element by element to the input visual feature sequence of the state space adaptation module to obtain the adapted visual features.
8. A referential image segmentation system based on a sparse bidirectional state-space adapter, characterized in that, include: The input acquisition module is used to acquire the image to be processed and a natural language expression for indicating the target object in the image to be processed. The feature encoding module includes a visual encoding network and a text encoding network. The visual encoding network is used to extract visual features from the image to be processed to obtain a visual feature sequence including features of multiple image patches. The text encoding network is used to extract text features from the natural language index to obtain a text feature sequence. A state space adaptation module is set in a partial feature processing stage of the visual coding network. It is used to map the image block features to the adaptation feature space, perform state space feature propagation along a first sequence direction and a second sequence direction opposite to the first sequence direction, and fuse the obtained first direction state space features and second direction state space features to obtain bidirectional contextual visual features. A visual adaptation module is set in at least a portion of the feature processing stage where the state space adaptation module is not set, and is used to perform local spatial feature adaptation. A cross-modal fusion module is used to perform cross-modal feature interaction on the bidirectional contextual visual features based on the text feature sequence to obtain text conditional visual features; The gated residual update module is used to form residual features based on the text conditional visual features, modulate the residual features using learnable gating parameters, and add the modulated residual features to the corresponding visual feature sequence to obtain the adapted visual features. The segmentation prediction module is used to generate a target object segmentation mask based on the adapted visual features, which represents the natural language instruction to indicate the target object.
9. An electronic device, characterized in that, Includes a processor and a memory communicatively connected to the processor; The memory stores a computer program that can be executed by the processor, which, when executed by the processor, causes the processor to perform the referential image segmentation method based on a sparse bidirectional state space adapter as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the reference image segmentation method based on a sparse bidirectional state space adapter as described in any one of claims 1 to 7.
