Image segmentation method and system based on deterministic feature decoupling and latent space contrast
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-15
- Publication Date
- 2026-08-14
AI Technical Summary
[0003]针对现有技术的不足，本发明提出一种基于确定性特征解耦与隐空间对比的稳健指代图像分割方法，能够解决现有模型依赖嘈杂全局特征造成的误分割问题，并显著提升面对细粒度错位文本时的鲁棒防御能力
[0021]与现有技术相比，本发明的有益效果是：彻底消除“强制分割”陷阱：利用掩码引导图文匹配机制（MG-ITM），将网络初步生成的掩码充当物理门控，硬性提取纯净的前景特征。背景噪声被显式剥离，实现了完全基于纯净表征的图文匹配，大幅提高了存在性判别的准确率。提升鲁棒性并实现对错位文本的降维打击：独创的对比动态难负样本挖掘策略（CL-DNM），通过在迷你批次内部在线打乱文本，强制模型在多模态隐空间内学习对负向句子的显式语义排斥力，使模型对错误文本的绝对拒识能力获得断崖式提升。极高的推理计算能效：对比学习防御机制被设计为非对称架构，仅在训练阶段激活，在实际推理时为零额外计算开销；结合提出的轻量级空间门控（LSG）模块，模型在保持高精度的同时实现了极致的轻量化。
Smart Images
Figure CN122574408A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary field of computer vision and natural language processing, and in particular to a robust referential image segmentation method and system based on deterministic feature decoupling and latent space comparison under multimodal artificial intelligence technology, which is designed to address complex misaligned text interference. Background Technology
[0002] Reference Image Segmentation (RIS) aims to locate and output a pixel-level mask of a target object in an image based on a given natural language text description. Traditional RIS tasks rely on an idealized assumption: the object described in the text must exist in the image. However, in real-world applications, the text description may contain incorrect attributes, misaligned spatial relationships, or completely nonexistent targets (i.e., negative sentences or misaligned text). To address this issue, Robust Reference Image Segmentation (R-RIS) was proposed, requiring the model to "reject" negative text, i.e., output a blank mask of all zeros, such as... Figure 3 As shown. Existing R-RIS solutions typically introduce additional whitespace tokens and utilize global multimodal features for existence binary classification. However, existing techniques have two fatal limitations: unreliable global feature classification: global multimodal features inevitably contain background noise. When the object described by the negative sentence is extremely similar to the image content, the model is prone to misjudgment due to visual redundancy, leading to the "forced segmentation" problem of forcibly outputting a mask. Lack of latent space repulsion: existing binary classification methods process each image-text pair in isolation, and the model cannot learn the semantic distance between positive and negative text, resulting in extremely weak defense against "hard negatives" with local detail misalignment. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention proposes a robust referential image segmentation method based on deterministic feature decoupling and latent space comparison. This method can solve the missegmentation problem caused by existing models relying on noisy global features and significantly improve the robust defense capability against fine-grained misaligned text.
[0004] One objective of this invention is to provide an image segmentation method based on deterministic feature decoupling and latent space contrast, comprising the following steps:
[0005] Step 1: Obtain the input image and natural language description text, extract multi-scale visual features using a visual encoder, extract word-level text features and global text tokens using a language encoder, and obtain a multimodal fusion feature sequence through deep interaction using the Visual-Language Token Fusion (VLTF) module.
[0006] Step 2: Construct a feature decoder based on lightweight spatial gating (LSG), broadcast the global text token in the spatial dimension and concatenate it with multimodal fusion features, generate a two-dimensional spatial attention map through 1×1 convolution, use it as a physical gate to generate gated enhancement features, and finally output a fine image foreground segmentation mask.
[0007] Step 3: Perform mask-guided deterministic image-text matching (MG-ITM), extract the logarithm of the preliminary foreground mask generated in Step 2, convert it into spatial soft attention weights, dynamically pool it with the deepest multimodal fusion features, extract clean local foreground features and predict the matching probability;
[0008] Step 4, Model Inference and Output: During the inference stage, the mask is determined based on the matching probability output in Step 3. If the probability is greater than the set threshold, the fine segmentation mask from Step 2 is output; otherwise, it is judged as negative text and a zero-space mask is output.
[0009] Step 5: Compare the training of the Dynamic Hard Negative Sample Mining (CL-DNM) model. In the same mini-batch during the training phase, the text sequences of other images are randomly shuffled to construct dynamic hard negative samples. After mapping to a unified contrast quantity space, cross-modal alignment optimization is performed using the InfoNCE loss function.
[0010] As a further improvement of the present invention, the specific process of step 2 is as follows: broadcast the global text features in the spatial dimension to a resolution consistent with the multimodal fusion features; concatenate the broadcast text features with the visual features, and generate a two-dimensional spatial attention map through lightweight 1×1 convolution calculation; use the two-dimensional spatial attention map as a physical gating weight to perform element-wise multiplication weighting on the multimodal fusion features, and generate a fine mask after residual connection and upsampling.
[0011] As a further improvement of the present invention, the specific process of step 3 is as follows: the logits of the initial segmentation mask are converted into soft attention weights through the Sigmoid activation function, and the deepest multimodal features are multiplied element-wise with the weights to remove redundant visual information of the image background and extract clean features. These clean features are then concatenated with the global text token and input into a multilayer perceptron to predict the probability of existence.
[0012] As a further improvement of the present invention, the specific operation of step 5 is as follows: the optimization objective of the loss function is to make the clean local foreground features close to the matching real text description positive samples in the metric space, while moving away from the misaligned text negative samples generated by batch shuffling, so that the model can obtain explicit contrast capability. This mechanism is only activated during the training phase and turned off during the inference phase.
[0013] As a further improvement of the present invention, the threshold is 0.5. That is, if the probability in step 4 is... The goal is believed to exist. They believe the target does not exist.
[0014] A second objective of this invention is to provide an image segmentation system based on deterministic feature decoupling and latent space contrast, comprising: a multimodal encoding module (Language Encoder and Vision Encoder) for extracting features from the input image and text and performing multimodal fusion; a spatial gated decoding module (GFP) for generating a spatial attention map using text priors to guide the generation of a target segmentation mask; a mask-guided verification module (MG-ITM) for extracting clean foreground features using the segmentation mask and performing existence probability prediction based on the clean features; and a contrastive defense training module (CL-DNM) for constructing dynamically misaligned text negative samples during training and calculating contrastive loss to strengthen the latent space repulsion of the model.
[0015] The robust referential image segmentation method based on deterministic feature decoupling and latent space comparison of the present invention includes the following steps:
[0016] Step 1: Obtain the input image and natural language description text, perform multi-scale visual feature extraction and text feature extraction respectively, and perform multi-modal feature fusion through a cross-attention mechanism;
[0017] Step 2: Combine the multimodal fusion features with the global text features, and generate a preliminary segmentation mask containing the target space prior through a lightweight spatial gating mechanism; broadcast the global text token in the spatial dimension to match its resolution with the multimodal features in the current decoding stage; after concatenation in the channel dimension, generate a two-dimensional spatial attention map through 1×1 convolution and Sigmoid activation function, which serves as a physical gating feature for generating enhancement features;
[0018] Step 3: Extract the preliminary segmentation mask as spatial attention weights, and perform dynamic pooling on the multimodal fusion features to remove background noise and extract clean local foreground features;
[0019] Step 4, Model Inference Stage: The probability of target existence is predicted using the clean local foreground features and global text features. If the probability is higher than a set threshold, a fine mask is output; otherwise, a blank mask with all zeros is output.
[0020] Step 5, Model Training Phase: Text sequences within the same data batch are shuffled to construct dynamic hard-to-bear samples, and the contrastive learning loss function is used to optimize the alignment and exclusion of multimodal features in the latent space.
[0021] Compared with existing technologies, the beneficial effects of this invention are: Completely eliminating the "forced segmentation" trap: Utilizing the Mask-Guided Image-Text Matching Mechanism (MG-ITM), the mask initially generated by the network acts as a physical gate, forcibly extracting pure foreground features. Background noise is explicitly stripped away, achieving image-text matching entirely based on pure representations, significantly improving the accuracy of existence discrimination. Enhanced robustness and dimensionality reduction against misaligned text: The original Contrastive Dynamic Hard Negative Sample Mining (CL-DNM) strategy, by shuffling text online within mini-batches, forces the model to learn explicit semantic rejection of negative sentences in the multimodal latent space, resulting in a dramatic improvement in the model's absolute rejection capability for erroneous text. Extremely high inference computational efficiency: The contrastive learning defense mechanism is designed with an asymmetric architecture, activated only during the training phase, with zero additional computational overhead during actual inference; combined with the proposed Lightweight Spatial Gating (LSG) module, the model achieves extreme lightweighting while maintaining high accuracy. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the overall method flow provided in the embodiments of the present invention;
[0023] Figure 2 This is an example diagram of the model network architecture provided in an embodiment of the present invention;
[0024] Figure 3 This is a comparison chart of qualitative detection effects provided in an embodiment of the present invention;
[0025] Figure 4 This is an internal structure diagram for mask image matching provided in an embodiment of the present invention.
[0026] Image and Text are the two basic input sources for the model. The multimodal encoding module includes a Language Encoder and a Vision Encoder: the Language Encoder processes the input natural language description, extracting high-dimensional text feature representations; the Vision Encoder typically employs a hierarchical backbone network, shown in the diagram as having four stages (Stage 1 to Stage 4), used to extract image features at different resolutions from shallow to deep (V1, V2, V3, V4). The Spatial Gated Decoding (GFP) module receives features from the end of the Vision Encoder (F4) after sufficient multimodal interaction, performs further feature aggregation and decoding, and uses text priors to generate a spatial attention map to guide the generation of a target segmentation mask. The Mask-Guided Validation (MG-ITM) module uses the segmentation mask to extract clean foreground features and performs existence probability prediction based on these clean features. The Contrastive Defense Training Module (CL-DNM) is used to construct dynamically misaligned text negative samples during training, calculate the alignment similarity score between the image and the text, and calculate the contrastive loss based on this score to strengthen the model's latent space repulsion. Mask Logits are the raw numerical values without activation function processing. The Final Mask is the final weight / mask matrix obtained after applying an activation function (such as Sigmoid to map the values to 0-1) or binarization (converting to pure 0 and 1 by setting a threshold) to the Mask Logits. The Alignment Score (sim) represents the similarity between image features and text features in the multimodal feature space. Fi typically represents a feature in diagrams and formulas. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments and accompanying drawings. Obviously, the described embodiments are merely some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0029] This embodiment provides a robust referential image segmentation method based on deterministic feature decoupling and latent space comparison, which includes the following steps:
[0030] Step 1: Obtain the input image and natural language description text, extract multi-scale visual features using a visual encoder, extract word-level text features and global text tokens using a language encoder, and obtain a multimodal fusion feature sequence through deep interaction using the Visual-Language Token Fusion (VLTF) module.
[0031] Step 2 involves constructing a feature decoder based on Lightweight Spatial Gating (LSG). The global text token is broadcast spatially and concatenated with the multimodal fusion features. A two-dimensional spatial attention map is generated through 1×1 convolution, which is then used as a physical gate to generate gated enhancement features. Finally, a refined image foreground segmentation mask is output. The specific process of Step 2 is as follows: The global text features are broadcast spatially to a resolution consistent with the multimodal fusion features; the broadcast text features are concatenated with visual features and a two-dimensional spatial attention map is generated through lightweight 1×1 convolution; this two-dimensional spatial attention map is used as a physical gate weight and applied to the multimodal fusion features for element-wise multiplication and weighting. After residual connection and upsampling, a refined mask is generated.
[0032] Step 3: Perform Mask-Guided Deterministic Image-Text Matching (MG-ITM). Extract the logarithm of the initial foreground mask generated in Step 2 and convert it into spatial soft attention weights. Dynamically pool these weights with the deepest multimodal fusion features to extract clean local foreground features and predict the matching probability. The specific process of Step 3 is as follows: The logarithm (Logits) of the initial segmentation mask is converted into soft attention weights through a Sigmoid activation function. The deepest multimodal features are multiplied element-wise with these weights to remove redundant visual information from the image background and extract clean features. These clean features are then concatenated with the global text token and input into a multilayer perceptron to predict the existence probability.
[0033] Step 4, Model Inference and Output: During the inference phase, the mask is determined based on the matching probability output in Step 3. If the probability is greater than a set threshold, the fine-grained segmentation mask from Step 2 is output; otherwise, it is determined as negative text and a blank mask with zeros is output. The threshold is 0.5, meaning that if the probability in Step 4 is greater than a set threshold, the mask is output. The goal is believed to exist. The target is deemed not to exist;
[0034] Step 5: Compare the training of the Dynamic Hard Negative Sample Mining (CL-DNM) model. Within the same mini-batch during the training phase, text sequences from other images are randomly shuffled to construct dynamic hard negative samples. After mapping to a unified contrast metric space, cross-modal alignment optimization is performed using the InfoNCE loss function. The specific operation of Step 5 is as follows: The optimization objective of the loss function is to make the clean local foreground features close to the matching real text description positive samples in the metric space, while moving away from the misaligned text negative samples generated within the batch, so that the model can obtain explicit contrast capability. This mechanism is only activated during the training phase and turned off during the inference phase.
[0035] An image segmentation system based on deterministic feature decoupling and latent space contrast includes: a multimodal encoding module (Language Encoder and Vision Encoder) for extracting features from the input image and text and performing multimodal fusion; a spatial gated decoding module (GFP) for generating a spatial attention map using text priors to guide the generation of a target segmentation mask; a mask-guided verification module (MG-ITM) for extracting clean foreground features using the segmentation mask and performing existence probability prediction based on the clean features; and a contrastive defense training module (CL-DNM) for constructing dynamically misaligned text negative samples during training and calculating contrastive loss to strengthen the latent space repulsion of the model.
[0036] like Figure 1 The method flow shown is based on Figure 2 The model diagram shown includes the following steps:
[0037] Step 1: Multimodal feature extraction and preliminary fusion. Obtain the input image. and natural language description text Extracting text features using a language encoder and global text token Multi-stage visual features are extracted using a visual encoder. These visual features are then deeply interacted with textual features via a Visual-Language Token Fusion (VLTF) module. The feature extraction and fusion process can be formally represented as follows:
[0038] ;
[0039] ;
[0040] in, This represents the first stage of operation of the visual encoder. This indicates the absolute positional encoding of the injection. Indicates the first Stage-based multimodal fusion feature sequences Word-level text features extracted by the language encoder. It is the length of the words in the text description (number of tokens). It is the channel dimension of the feature. Represents the first in the text sequence A specific word or token This represents the length of the words in the input text description. This represents the raw visual feature map output by the visual encoder in the first stage. The second phase is for The results after further downsampling and feature extraction It is the third stage for The result after processing.
[0041] Step 2, Feature decoding based on Lightweight Spatial Gating (LSG). This involves decoding the global text token. Perform a broadcast operation (Expand) in the spatial dimension, making its resolution match the resolution of the current decoding stage. Multimodal fusion features Matching; concatenating the broadcast text features with the multimodal fusion features along the channel dimension, through... Convolution and Sigmoid activation function Generate a 2D spatial attention map The calculation formula is as follows:
[0042] ;
[0043] in, This represents the spatial resolution of the attention map. This represents a type of technology that is extremely commonly used in deep learning. Convolution operation, It represents a macro-level semantic summary of the entire input sentence. For the current decoding stage Multi-modal fusion features of layers The sigmoid activation function compresses and maps the output values of a convolution to... Within the range.
[0044] Subsequently, the two-dimensional spatial attention map is used as a physical gate and multiplied element-wise with the multimodal fusion feature. The weighted features are then processed and residually connected with the original features to generate gated enhanced features:
[0045] ;
[0046] The gated enhancement features are bilinearly upsampled and concatenated with shallow features to finally output a fine image foreground segmentation mask.
[0047] Step 3, Mask-Guided Deterministic Image-Text Matching (MG-ITM), such as Figure 4 Extract the initial foreground mask logits generated in step 2, and apply Sigmoid activation as spatial soft attention weights. ; fusing the deepest multimodal features Element-wise multiplication and dynamic pooling are performed with the spatial soft attention weights to extract clean local foreground features stripped of background noise. The formula for calculating the decoupling of this deterministic feature is:
[0048] ;
[0049] in, represents the real number field, and c represents the channel dimension of the feature. These are spatial pixel coordinates. The smallest constant used to prevent division by zero errors is usually set to 0. , The deepest multimodal fusion features at a specific pixel location eigenvectors, Spatial soft attention weights. These are derived from the initially generated mask logits through a sigmoid transformation, representing the probability that the pixel belongs to the foreground.
[0050] Subsequently, the pure local foreground features With global text token The images are concatenated and fed into a multilayer perceptron (MLP) to predict the matching probability of the text description object in the image.
[0051] Step 4: Compare the training of the Dynamic Hard Negative Sample Mining (CL-DNM) model. Within the same mini-batch during the training phase, maintain the current image and the matched real text description as positive sample pairs. Randomly shuffle the text sequences of other images within a batch online to construct a highly deceptive dynamic set of hard-to-bear samples. ; to clean local foreground features The text sequence and the projector are mapped to a unified contrast space, and cross-modal alignment optimization is performed using the InfoNCE loss function:
[0052] ;
[0053] in, To compare the loss values, the smaller the value, the closer the positive samples are pulled and the farther the negative samples are pushed. In mathematics, it is represented by the natural constant. An exponential function with base 0. Cosine similarity, which measures the distance between vectors, To control the learnable temperature coefficient of repulsion strength, This represents the number of difficult-to-bear samples dynamically generated within the batch.
[0054] Step 5, Joint Loss Optimization and Model Inference Output. During the model training phase, the mask segmentation loss is used... Existence prediction of binary cross-entropy loss and comparative loss Combined end-to-end optimization, total loss function Defined as:
[0055] ;
[0056] in, and To balance the weighting coefficients of each task, during the inference phase, the contrastive learning branch in step 4 is closed, and the final mask is determined based on the matching probability output in step 3: if the matching probability is greater than a set threshold, the fine segmentation mask generated in step 2 is output; otherwise, it is determined to be negative text, and a blank mask with all zeros is output.
[0057] Those skilled in the art will understand that the embodiments of this application can be provided as methods, systems, or computer program products, and can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code, and can include the processes of the embodiments of the methods described above. The embodiments described above are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. An image segmentation method based on deterministic feature decoupling and latent space comparison, characterized in that, Includes the following steps: Step 1: Obtain the input image and natural language description text, extract multi-scale visual features using a visual encoder, extract word-level text features and global text tokens using a language encoder, and obtain a multimodal fusion feature sequence through deep interaction using the Visual-Language Token Fusion Module (VLTF). Step 2: Construct a feature decoder based on lightweight spatial gated LSG, broadcast the global text token in the spatial dimension and concatenate it with multimodal fusion features, generate a two-dimensional spatial attention map through 1×1 convolution, use it as a physical gate to generate gated enhancement features, and finally output a fine image foreground segmentation mask. Step 3: Perform mask-guided deterministic image-text matching MG-ITM, extract the logarithm of the preliminary foreground mask generated in Step 2 and convert it into spatial soft attention weights, dynamically pool it with the deepest multimodal fusion features, extract clean local foreground features and predict the matching probability. Step 4, Model Inference and Output: During the inference stage, the mask is determined based on the matching probability output in Step 3. If the probability is greater than the set threshold, the fine segmentation mask from Step 2 is output; otherwise, it is judged as negative text and a zero-space mask is output. Step 5: Compare the training of the dynamic hard negative sample mining CL-DNM model. In the same mini-batch during the training phase, the text sequences of other images are randomly shuffled to construct dynamic hard negative samples. After mapping to a unified contrast quantity space, cross-modal alignment optimization is performed using the InfoNCE loss function.
2. The image segmentation method based on deterministic feature decoupling and latent space comparison according to claim 1, characterized in that, The specific process of step 2 is as follows: broadcast the global text features in the spatial dimension to a resolution consistent with the multimodal fusion features; concatenate the broadcast text features with the visual features, and generate a two-dimensional spatial attention map through lightweight 1×1 convolution calculation; use the two-dimensional spatial attention map as a physical gating weight to perform element-wise multiplication weighting on the multimodal fusion features, and generate a fine mask after residual connection and upsampling.
3. The image segmentation method based on deterministic feature decoupling and latent space comparison according to claim 1, characterized in that, The specific process of step 3 is as follows: the logits of the initial segmentation mask are transformed into soft attention weights through the Sigmoid activation function. The deepest multimodal features are multiplied element-wise with these weights to remove redundant visual information from the image background and extract clean features. These clean features are then concatenated with the global text token and input into a multilayer perceptron to predict the probability of their existence.
4. The image segmentation method based on deterministic feature decoupling and latent space comparison according to claim 1, characterized in that, The specific operation of step 5 is as follows: The optimization objective of the loss function is to make the clean local foreground features close to the matching real text description positive samples in the metric space, while staying away from the misaligned text negative samples generated by shuffling within the batch. This mechanism is only activated during the training phase and turned off during the inference phase.
5. The image segmentation method based on deterministic feature decoupling and latent space comparison according to claim 1, characterized in that, The threshold is 0.
5.
6. An image segmentation system employing the method described in claims 1-5, characterized in that, include: The multimodal coding module is used to extract features from the input image and text and perform multimodal fusion. The Spatial Gated Decoding Module (GFP) is used to generate a spatial attention map based on text priors to guide the generation of a target segmentation mask. The Mask-Guided Verification Module (MG-ITM) is used to extract clean foreground features using the segmentation mask and perform existence probability prediction based on the clean features. The contrastive defense training module CL-DNM is used to construct dynamically misaligned text negative samples during training and calculate contrastive loss to enhance the latent space repulsion of the model.