Fine-grained segmentation method, system and device driven by multiple images and storage medium

Abstract

The invention discloses a fine-grained segmentation method, system and equipment driven by multiple images and a storage medium, which are corresponding schemes, in the scheme, object semantics, component forms and color attributes are decoupled and combined into a unified segmentation framework, and a complex target can be accurately described through flexible combination of simple visual prompts; according to the method, a two-stage decoding assembly line is designed, target categories are screened through semantic prompts, fine segmentation is carried out in combination with components specified by a user and color features, efficient fusion is achieved while decoupling representation is carried out, and the control precision of fine-grained attributes is remarkably improved; in addition, the advanced performance in the industry is obtained on the semantic-component-color driven combined segmentation task, and the false drop rate in fine-grained query is greatly reduced; meanwhile, the framework is flexible in design, seamless switching between standard prompt drive segmentation and combined prompt drive segmentation can be achieved, and high competitiveness on the general segmentation reference is kept.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a fine-grained segmentation method, system, device, and storage medium driven by multiple images. Background Technology

[0002] Image segmentation is a fundamental and crucial task in computer vision, aiming to assign a semantic or instance label to each pixel of an input image. Traditional image segmentation methods primarily rely on deep convolutional neural networks (CNNs). Early landmark work, such as the Fully Convolutional Network (FCN) proposed by Long et al. in their paper "Fully Convolutional Networks for Semantic Segmentation," replaced the fully connected layers in traditional classification networks with convolutional layers, enabling the network to accept input images of arbitrary sizes and generate corresponding pixel-level segmentation maps, thus establishing an end-to-end segmentation learning paradigm. Building on this, Ronneberger et al. proposed the U-Net architecture in their paper "U-Net: Convolutional Networks for Biomedical Image Segmentation." By introducing a symmetrical encoder-decoder structure and "skip connections," the shallow high-resolution feature maps retained in the encoder are passed to the decoder, significantly improving the model's ability to recover object edges and details and enhancing localization accuracy. Region proposal methods, exemplified by Mask R-CNN proposed by He et al., integrate object detection and pixel-level segmentation tasks into a unified framework. This method generates high-quality segmentation masks for each object instance while detecting it, thereby improving segmentation accuracy from the semantic level to the instance level. However, a common feature of these methods is that they require supervised training on large labeled datasets for specific segmentation categories. Once the model is trained, its functionality is fixed, making it difficult to generalize to object categories not seen during training.

[0003] To overcome this limitation, in-context learning (ICL)-based visual segmentation paradigms have become a new research hotspot in recent years. Originating in natural language processing, this paradigm's core idea is to allow the model to understand and perform new tasks during the inference phase by referencing one or more examples (i.e., "context" or "cues"), without updating the model weights. In image segmentation, this typically means using a reference image and its corresponding segmentation mask as cues to guide the model in segmenting any target in the query image.

[0004] In this new paradigm, early exploratory work attempted to unify different segmentation tasks into a general visual model. For example, Wang et al., in their paper "Painter: In-Context Learning for Generalist Models," treated segmentation tasks as a kind of mask filling or image drawing task; similarly, their subsequent paper "SegGPT: Segmenting Everything in Context" formalized this concept as "in-context coloring." These methods validated the feasibility of achieving general segmentation through context learning, but they mainly rely on matching high-level semantics or overall visual patterns, and their capabilities are limited when users require more refined and specific appearance constraints.

[0005] Subsequently, the development of large-scale pre-trained models greatly advanced this field. For example, the emergence of Segment Anything Model (SAM) demonstrated powerful zero-shot segmentation capabilities, capable of segmenting almost any object in an image based on simple cues such as points and bounding boxes. To further improve the generalization ability and robustness of context-learning segmentation, subsequent works, such as Liu et al.'s papers "Matcher: Segment Anything with One Shotusing All-Purpose Features Matching" and "A Simple Image Segmentation Framework via In-Context Examples," began to explore the use of powerful visual features extracted by pre-trained image pedestal models such as DINOv2, and to more accurately match user intent by calculating semantic correspondences or designing cue fusion mechanisms.

[0006] While the aforementioned methods have achieved significant progress in semantic or instance-level segmentation tasks, existing technologies still face significant bottlenecks when user needs evolve from "what objects to segment" to "what objects look like to segment." These common problems mainly manifest in: (1) Limited ability to model and control appearance attributes: Existing ICL models are good at understanding cues that specify semantic categories (such as “segment out a person”) or specific instances (such as “segment out this person”), but they are not good at responding to cues that contain fine-grained appearance attribute constraints (such as “segment out the person wearing a black coat” or “segment out the red back of the chair”). These models lack the mechanism to parse and prioritize such fine-grained visual constraints and can only perform a broader semantic matching.

[0007] (2) Severe coupling between semantic and appearance features: In the internal feature representation of many models, the high-level semantic information used to identify the object's identity is coupled with the fine-grained visual details describing its specific appearance, forming an inseparable feature vector. This coupling makes it impossible for the model to independently learn or apply appearance constraints without affecting semantic recognition, thereby reducing the accuracy and controllability of user prompts.

[0008] (3) Limited combinatorial cues: The feature coupling problem manifests at the input level as a lack of "compositionality" in the cues mechanism. Existing methods generally cannot logically "combine" the part shapes and color attributes from different examples to generate new descriptions that accurately describe the target object. The model can only process a single, holistic reference image. Furthermore, in practical applications, users often find it difficult to find a single reference image that perfectly matches all expected attributes.

[0009] (4) Low reliability of fine-grained queries: Due to the above limitations, when the segmentation prompts contain fine-grained appearance descriptions, the false detection rate and false negative rate of existing models will increase significantly. This unreliability makes them unsuitable for application scenarios with strict requirements for segmentation accuracy. In these scenarios, the model may incorrectly segment semantically similar but visually inconsistent regions, or directly ignore appearance constraints, resulting in unreliable segmentation results and reducing the practical value of the technology.

[0010] In summary, existing technologies still face challenges in achieving fine-grained, composable, and interpretable segmentation of targets. To overcome these limitations, the field urgently needs a feature representation that can decouple high-level semantics from fine-grained appearance at the model architecture level, supporting their independent processing and controllable fusion, and flexibly integrating guiding information from multiple, potentially unrelated, reference cues. This would enable a unified technical solution for accurate, attribute-driven segmentation without relying on a single perfect example.

[0011] In view of this, the present invention is hereby proposed. Summary of the Invention

[0012] The purpose of this invention is to provide a fine-grained segmentation method, system, device, and storage medium driven by multiple images, which can achieve fine-grained segmentation of targets in query images in a combinatorial and interpretable manner.

[0013] The objective of this invention is achieved through the following technical solution: A multi-image-driven fine-grained segmentation method includes: The input target image, semantic reference image, component reference image and color reference image are encoded to obtain the corresponding encoded features, which are called target image feature map, semantic prototype, component prototype and color feature map; In the first decoding stage, a set of learnable features are used to perform instance query. Combining the semantic prototype and the target image feature map, all object instances belonging to the specified semantic category are located in the target image. The first-stage instance features are output, and the instance mask is generated using the first-stage instance features. The semantic prototype processed in the first decoding stage is also output. In the second decoding stage, the color feature map and the component prototype are mutually enhanced, and then fused with the semantic prototype processed in the first decoding stage to obtain the combined prototype features. These features are then connected with the semantic prototype processed in the first decoding stage to obtain multi-category prototype features. A weighted target feature map is generated using an instance mask. The instance features from the first stage are refined by combining the multi-category prototype features and the weighted target feature map to obtain refined instance features. At the same time, the multi-category prototype features are transformed to obtain the multi-category prototype features processed in the second decoding stage. The multi-category prototype features processed in the second decoding stage are used to segment the refined instance features based on a dual matching mechanism to obtain the final segmentation mask.

[0014] A multi-image-driven fine-grained segmentation system for implementing the aforementioned method includes: The encoding module is used to encode the input target image, semantic reference image, component reference image and color reference image respectively to obtain the corresponding encoded features, which are called target image feature map, semantic prototype, component prototype and color feature map; The first decoding module, applied to the first decoding stage, is used to perform instance query using a set of learnable features, combine semantic prototypes with target image feature maps, locate all object instances belonging to the specified semantic category in the target image, output the first stage instance features, generate an instance mask using the first stage instance features, and output the semantic prototype processed in the first decoding stage. The second decoding module, applied in the second decoding stage, enhances the color feature map with the component prototype and fuses it with the semantic prototype processed in the first decoding stage to obtain combined prototype features. Then, it connects these combined features with the semantic prototype processed in the first decoding stage to obtain multi-category prototype features. It generates a weighted target feature map using an instance mask. It refines the instance features of the first stage by combining the multi-category prototype features and the weighted target feature map to obtain refined instance features. At the same time, it transforms the multi-category prototype features to obtain the multi-category prototype features processed in the second decoding stage. The segmentation module is used to segment the refined instance features based on the multi-category prototype features processed in the second decoding stage and based on a dual matching mechanism to obtain the final segmentation mask.

[0015] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0016] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0017] As can be seen from the technical solutions provided by the present invention: (1) The semantics of the object, the shape of the component and the color attribute are decoupled and combined into a unified segmentation framework, allowing users to accurately describe complex targets by flexibly combining simple visual cues; (2) A two-stage decoding pipeline of "coarse localization-fine optimization" is designed. This architecture first filters the target category through semantic cues, and then performs fine segmentation by combining the component and color features specified by the user. Thus, efficient fusion is achieved while decoupling the representation, which significantly improves the control accuracy of fine-grained attributes; (3) The present invention has achieved industry-leading performance in the combined segmentation task driven by "semantics-component-color", which greatly reduces the false detection rate in fine-grained queries. At the same time, the overall framework design is flexible and can seamlessly switch between standard prompt-driven segmentation and combined prompt-driven segmentation, maintaining high competitiveness on the general segmentation benchmark. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a multi-image-driven fine-grained segmentation method provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram comparing the present invention with existing methods, provided for an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the overall architecture of a multi-image-driven fine-grained segmentation method provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of a color fusion module provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the feature map weighting module provided in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of a color component feature fusion module provided in an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram illustrating an example of actual segmentation effect provided in an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of a fine-grained segmentation system driven by multiple images, provided in an embodiment of the present invention.

[0027] Figure 9 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0029] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0030] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0031] The following provides a detailed description of a multi-image-driven fine-grained segmentation method, system, device, and storage medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of the instruments used in the embodiments of the present invention are not specified, they are all conventional products that can be purchased commercially.

[0032] Example 1 This invention provides a fine-grained segmentation method driven by multiple images, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Encode the target image and the combined multimodal cues.

[0033] In this embodiment of the invention, the multimodal combined cues include: a semantic reference image, a component reference image, and a color reference image. By encoding the target image and the multimodal combined cues respectively, and converting them into high-dimensional feature representations, corresponding target image feature maps, semantic prototypes, component prototypes, and color feature maps are obtained.

[0034] The encoding process mainly includes: (1) Encoding the target image, semantic reference image and component reference image respectively using a pre-trained visual model to obtain the target image feature map. Semantic reference image features Features of component reference images Mask pooling is used to extract semantic reference image features. Features of component reference images Extract the corresponding semantic prototypes from each. With component prototype (2) Convert the color reference image and the target image to the same color space, and calculate the color difference pixel by pixel to obtain a color similarity map. The color similarity map is generated using a convolutional network. The affine transformation parameters, and the feature map of the target image. Modulation is performed to obtain a color feature map. .

[0035] Step 2: The first decoding stage obtains a rough semantic location.

[0036] In this embodiment of the invention, in the first decoding stage, a set of learnable features are used to perform instance query. By combining the semantic prototype and the target image feature map, all object instances belonging to the specified semantic category are located in the target image, and the instance features are output. At the same time, the semantic prototype processed in the first decoding stage is also output.

[0037] In this embodiment of the invention, the semantic prototype is processed through a Transformer block. and target image feature map Perform interactive processing to obtain the semantic prototype after processing by the Transformer block. and target image feature map The first decoder in the first decoding stage utilizes a set of learnable features. To perform instance queries, the target image feature map is processed by the Transformer block using self-attention and cross-attention mechanisms. To interact, and in the semantic prototype processed by the Transformer block. Guided by this process, instance queries gradually learn and focus on regions in the image that match the specified semantic category. This stage ultimately outputs the first-stage instance features. and semantic prototype features after the first decoding stage First-stage instance features It will also be used to generate the mask for the first decoding stage. And a rough mask (i.e., an instance mask) prepared for the second decoding stage. .

[0038] In this embodiment of the invention, the semantic prototype is processed through a Transformer block. and target image feature map To perform interactive processing, which is also about to In cross-attention, they serve as keys and values ​​to mutually reinforce each other. For details, please refer to conventional techniques. This invention will not elaborate further.

[0039] In this embodiment of the invention, instance features The context-aware instance features are projected onto the same dimensional space as the pixel features using an MLP (Multilayer Perceptron) to obtain the predicted instance mask; it also uses a classification head to predict the confidence score for each category corresponding to the instance.

[0040] Furthermore, the predicted instance mask and the confidence score for each category are also used to generate the mask for the first decoding stage. and a rough mask prepared for the second decoding stage. Specifically, for the same instance, the category with the highest confidence score is directly selected as the label, and then the categories are merged to obtain a coarse mask. Generate the mask for the first decoding stage. Similar to the operation of Mask2Former (an advanced deep learning model for general image segmentation) in existing technologies, the top k confidence scores are selected. After filtering, the mask corresponding to the scores is used to calculate the overall mask quality score (the average of the mask region scores after sigmoid transformation). The overall mask quality score is then multiplied by the corresponding confidence score to obtain the overall confidence score of the corresponding mask. Masks with a confidence score greater than a set threshold (e.g., 0.5) are used as the final mask for the first decoding stage. .

[0041] Step 3: The second stage of decoding obtains the refined instance features.

[0042] In this embodiment of the invention, in the second decoding stage, the color feature map and the component prototype are mutually enhanced, and then fused with the semantic prototype processed in the first decoding stage to obtain combined prototype features. These combined features are then concatenated with the semantic prototype processed in the first decoding stage to obtain multi-category prototype features; instance masks are then used. The generated weighted target feature map; the first-stage instance features are refined by combining the multi-class prototype features and the weighted target feature map to obtain refined instance features. At the same time, the multi-class prototype features are transformed to obtain the multi-class prototype features processed in the second decoding stage.

[0043] In this embodiment of the invention, the color feature map and the component prototype are mutually enhanced and compared with the processed semantic prototype obtained in the first decoding stage. Fusion is performed to obtain combined prototype features Including: utilizing component prototypes Guide and modulate color feature maps The semantically guided feature map is obtained. , is represented as: ;in, Cosine similarity calculation per patch in the representation space; utilizing semantically guided feature maps. Reverse reinforcement component prototype This generates a visually perceptible component feature. ; the perceived features of the component semantic prototype after the first decoding stage The final combined prototype features are obtained through fusion. .

[0044] In this embodiment of the invention, the final combined prototype features The processed semantic prototype obtained in the first decoding stage Perform connection operations as multi-class prototype features .

[0045] Preferably, the present invention also includes an instance mask. The foreground score map is merged into a comprehensive foreground score map, which is then combined with the target image feature map processed by the Transformer block in the first decoding stage. Perform dot product weighting to obtain the weighted target feature map. .

[0046] Similar to the first decoding stage, the second decoder in the second decoding stage uses a weighted target feature map. And through multi-category prototype features To guide the first phase of instance features The process of refining the data yields the refined instance features. Simultaneously, the multi-category prototype features are transformed to obtain the multi-category prototype features processed in the second decoding stage. .

[0047] Step 4: Generate the final segmentation mask based on the dual matching mechanism.

[0048] In this embodiment of the invention, the multi-category prototype features processed in the second decoding stage are utilized. The refined instance features are then segmented using a dual matching mechanism to obtain the final segmentation mask. Specifically, the features of each candidate instance in the refined instance features need to be simultaneously matched with... Components: Semantic prototypes processed in the second decoding stage Combined prototype features after the second decoding stage The comparison is performed. Only when a candidate instance highly matches both of these prototype features is it determined to be the final target. High matching means that the similarity between the instance features and both semantic prototypes is higher than a threshold. If the features of multiple candidate instances highly match both semantic prototypes, these candidate instances will all be considered as the target class. A segmentation mask is generated based on the above discrimination results. This part is mainly the inference stage. During the training process, the instance features obtained in the two decoding stages mentioned above are mainly used to obtain the predicted semantic category and the predicted mask, which will be explained in detail later.

[0049] In this embodiment of the invention, , It is the second decoder pair in the second decoding stage. , The information obtained after processing is similar to the first decoding stage. The transformation process, for example, the processing of the decoder in the existing SINE (Segmentation framework via IN-context Examples) model, will not be elaborated here, considering that this part can be implemented by conventional techniques.

[0050] In this embodiment of the invention, the first decoding stage is implemented by a first decoding module, and the second decoding stage is implemented by a second decoding module. The Hungarian algorithm is used to calculate the loss of the two decoding stages and optimize the two decoding modules.

[0051] Loss in two decoding stages for: ; in, The loss for the i-th decoding stage, This corresponds to the first decoding stage and the second decoding stage.

[0052] Loss in the i-th decoding stage , is represented as: ; in, It is the optimal match found by the Hungarian algorithm. It represents the number of instance pairs matched in the i-th decoding stage, that is, the number of instance features that match the real instance in the corresponding decoding stage. It is for predicting semantic categories The standard negative log-likelihood loss, used in the first decoding stage, employs instance features. To predict semantic categories, the refined instance features are used in the second decoding stage. To predict semantic categories; Used to evaluate the prediction mask With the real mask The differences between them; in the first decoding stage, the prediction mask This is the mask mentioned above. In the second decoding stage, the prediction mask For mask Utilizing instance features And use with a mask It was calculated in the same way.

[0053] Since the methods for predicting semantic categories using instance features and calculating the corresponding prediction masks discussed here can be referenced from conventional techniques (e.g., see the operation of Mask2Former), they will not be elaborated upon here.

[0054] The above-described solution provided by the embodiments of the present invention aims to achieve fine-grained segmentation of targets in query images that is combinable and interpretable. It primarily addresses the technical problems of existing methods being insufficiently flexible and accurate in describing complex targets, as well as the high false detection rate and insufficient generalization ability of the segmentation results. Figure 2 This visually demonstrates the difference between this invention and previous methods. The multi-image-driven fine-grained segmentation method provided by this invention is called the Compositional Prompts and Phased Decoupling to Empower In-Context Segmentation with Fine-Grained Attribute Control (CDICS). By introducing a combined prompting and phased decoupling architecture, it endows context-based segmentation with unprecedented fine-grained attribute control capabilities. Figure 2As shown, in existing methods, the highlighted areas in the Ref Image serve as mask cues. In this invention, the Ref Imageemantic provides semantic information cues, while the Ref Image part corresponds to part-level cues. This invention provides a solution that supports standard contextual semantic / instance segmentation as well as fine-grained, appearance-constrained segmentation. Specifically, this invention employs a decoupled two-stage pipeline framework: First, the first decoding stage (DecoderStage1) utilizes semantic cues to perform coarse instance-level localization, locking the approximate range of the target object, i.e., semantic and instance-level segmentation; then, the second decoding stage (Decoder Stage2) uses independent, composable parts and color appearance constraints to refine and optimize the preliminary results. This design explicitly separates the feature flows of "what the object is" (semantics) and "what the object looks like" (appearance) within the model, thus fundamentally solving the feature coupling problem of existing technologies. The solution provided by this invention has the following main advantages: (1) A combined prompting mechanism is proposed: For the first time, the semantics of objects, the shape of components and the color attributes are decoupled and combined into a unified segmentation framework, allowing users to accurately describe complex targets by flexibly combining simple visual prompts.

[0055] (2) Design a phased decoupled fusion architecture: A two-stage decoding pipeline of "coarse localization-fine optimization" was designed. The architecture first filters the target category through semantic prompts and performs conventional semantic segmentation. Then, it combines the user-specified component and color features for fine segmentation, thereby achieving efficient fusion while decoupling the representation and significantly improving the control accuracy of fine-grained attributes.

[0056] (3) Achieving high-precision and high-flexibility segmentation: This invention achieves industry-leading performance in semantic-component-color driven combined segmentation tasks, significantly reducing the false detection rate in fine-grained queries. At the same time, the framework is flexibly designed and can seamlessly switch between standard prompt-driven segmentation and combined prompt-driven segmentation, maintaining high competitiveness on general segmentation benchmarks.

[0057] To more clearly demonstrate the technical solution and its effects provided by the present invention, the embodiments of the present invention will be described in detail below with reference to specific examples.

[0058] I. Overall Overview of the Plan

[0059] The method (CDICS) provided by this invention aims to solve the problems of low segmentation accuracy and ambiguity caused by feature coupling in existing technologies when dealing with complex segmentation tasks requiring multi-dimensional constraints of "semantic-part-color". The overall network architecture of this invention is as follows: Figure 3 As shown, the core of the design is an innovative two-stage decoupled decoder architecture that orthogonally decomposes the complex segmentation task into two independent sub-problems: high-level coarse semantic localization and low-level appearance constraint refinement. By designing a dedicated processing stage for each sub-problem, this invention effectively avoids mutual interference between features of different granularities, significantly improving the model's understanding and segmentation capabilities of combined cues.

[0060] In short, the technical solution of this invention includes the following key parts: (1) Encoder stage: responsible for converting the target image and multimodal combined cues (semantics, parts, colors) into a high-dimensional feature representation.

[0061] (2) First decoding stage (Decoder Stage 1): Responsible for performing coarse semantic localization, finding all object instances in the target image that match the specified semantic category.

[0062] (3) Second decoding stage (Decoder Stage 2): Based on the previous stage, appearance constraints such as components and colors are applied to refine the segmentation results.

[0063] (4) Final classification and segmentation: A unique dual matching mechanism is used to generate the final, high-precision segmentation mask.

[0064] (5) Loss function: End-to-end training is carried out using a combination loss function based on the Hungarian algorithm matching mechanism and phased supervision.

[0065] II. Detailed introduction of the plan.

[0066] The following section will provide a detailed introduction to each of the five parts mentioned above.

[0067] 1. Encoder stage: Multimodal feature extraction and prototype generation.

[0068] The goal of this stage is to gather all input information (target image) Semantic reference image Component Reference Color reference The features are encoded into high-dimensional features that can be used by the subsequent decoding module. The encoder involved can be implemented using an existing pre-trained visual model.

[0069] For example, a pre-trained DINOv2 model can be used as the backbone network. DINOv2 is a self-supervised learning visual foundation model proposed by MetaResearch in 2023. This model achieves powerful and highly discriminative visual feature extraction capabilities through self-supervised learning, demonstrating excellent performance in few-shot and semantic segmentation tasks, and providing a high-quality feature foundation for subsequent fine-grained discrimination. The target image... Semantic reference image Component Reference The initial processing is the same as before; it will be directly encoded into a spatial feature map. , , The dimension of the spatial feature map is ,in This represents the size (height and width) of the feature map after the image has been processed by DINOv2. The number of channels is 256.

[0070] To extract specific reference object information from the reference image, mask pooling is used to extract semantic and component prototype vectors. , Spatial feature maps of two reference information, using corresponding masks , Average pooling is performed only on feature vectors belonging to the target semantic region and the target component region, thereby generating compact semantic prototypes and component prototypes respectively. , The dimensions of these two prototype features are... .

[0071] To address the challenge of directly fusing color information with the high-dimensional features output directly from DINOv2, this invention designs a Color Fusion module. For example... Figure 4 As shown, this module first uses a color reference image. With target image Convert to the CIELAB color space, then calculate the pixel-by-pixel CIEDE2000 color difference between the target image and the reference color to generate a color similarity map. In the target image, pixels that are more similar in color to the reference color have higher intensity values, as shown in the following formula: ; in, Represents the normalization function. The function calculates the color difference between two colors. The smaller the color difference, the more similar the two colors are. (This is reflected in a color similarity graph.) The higher the value, the stronger the indication for that area.

[0072] Subsequently, the color similarity map is upscaled to a higher dimension using a convolutional network; for example, a shallow convolutional network module, FiLM, can be chosen. FiLM, proposed by Perez et al. in their paper "FiLM: Visual Reasoning with a GeneralConditioning Layer," is often used to inject an additional information stream into the intermediate features of the network. As shown in the following equation, the FiLM module upscales the color similarity map... Generate channel-level affine transformation parameters , Modulate target image feature map The processed output is a feature map of the target image. Spatially aligned color-enhanced feature map (referred to as color feature map).

[0073] ; ; The main purpose of this is to apply appropriate enhancements to the feature map regions that match the reference color, ultimately producing an enhanced feature map of the target color region. .

[0074] 2. First decoding stage: rough semantic instance localization.

[0075] The goal of this stage is to locate all object instances belonging to a specified semantic category in the target image, completely ignoring fine-grained attributes such as parts or colors. This invention employs a Transformer-based decoder architecture to achieve this function, which has become an advanced technical solution in the field of instance segmentation. First, semantic prototypes are segmented using Transformer blocks. and target image feature map Perform interactive processing to obtain the semantic prototype after processing by the Transformer block. and target image feature map Then, using a set of learnable features To perform instance queries, The dimension is ,in It is a pre-set large value, representing the maximum number of graphs that can be learned. Each instance feature, for example, can be set to 200. These queries are performed on the target image feature map through self-attention and cross-attention mechanisms. Interact. (In semantic prototype) Guided by this process, instance queries gradually learn and focus on regions in the image that match the target semantics. This stage ultimately outputs a set of instance features. and processed semantic prototype features ,in It will be used to generate a coarse mask for the second phase. , dimension This allows for the definition of clear candidate regions for the second-stage fine-grained analysis. Furthermore, during the training phase, [the system / mechanism] will also utilize [this technology / method / mechanism]. To obtain the predicted semantic category and prediction mask.

[0076] 3. Second decoding stage: Refinement of appearance constraint attributes.

[0077] After determining the semantic identity of the target, the core task of this stage is to refine the segmentation results by interpreting the "color-part" combination cues within the candidate area provided in the first stage, so as to finally obtain the target object under appearance constraints.

[0078] To effectively utilize the information from the first stage, a Feature Map Weighting module is introduced to leverage the masking information from the first stage. For example... Figure 5 As shown, this module will output multiple instance masks from the first decoding stage. This is combined into a single comprehensive foreground score map. This score map is then compared with the target feature map. By performing dot product weighting, the model's attention is focused more on foreground objects, effectively filtering out background noise.

[0079] To accurately apply the combined appearance constraint of "containing a specific color corresponding to a specific component" to object features, this invention designs a novel Part Color Feature Fusion module. For example... Figure 6 As shown, this module first utilizes a component prototype that contains only semantic information of the target component. To guide and modulate color enhancement feature maps This allows for spatially selective modulation of the feature map, resulting in a semantically guided feature map. As shown in the following formula: ; in, This represents the calculation of patch-by-patch cosine similarity in spatial representation. This operation significantly enhances region features that simultaneously satisfy semantic and color constraints. Then, this semantically guided feature map, containing precise location and appearance information, is utilized. Reverse enhancement (implemented based on Transformer blocks) component prototype This generates an appearance-aware component feature. This allows it to represent a target part with a specific appearance in the image. Then, the part features perceived by this appearance are... semantic prototype after the first decoding stage The final composite prototype feature is obtained through fusion (implemented based on Transformer blocks). The dimension of this prototype feature is the same as that of the other prototype features, which is 1. This feature fully contains all the constraint information regarding "semantic-part-color". This feature will be combined with the semantic prototype. Perform a concat operation to serve as a multi-class prototype feature. (shape is) This will be used for subsequent matching and classification.

[0080] Similar to the first decoding stage, the second decoder in the second decoding stage uses a weighted target feature map. And through multi-category prototype features To guide the first phase of instance features The process of refining the data yields the refined instance features. and multi-category prototype features after the second decoding stage. Similarly, if it's a training process, then instance features also need to be utilized. Obtain the predicted semantic category and prediction mask .

[0081] 4. Final classification and segmentation.

[0082] The final classification employs a dual matching mechanism. The refined instance features... The features of each candidate instance need to be simultaneously compared with the multi-class prototype features processed in the second decoding stage. The model consists of two components: a semantic prototype processed in the second decoding stage and a combined prototype processed in the second decoding stage. An instance is only considered the final target if its features highly match both prototype features. This design allows the model to effectively distinguish instances with different attributes within the same semantic category, thus generating a final fine-grained segmentation mask.

[0083] 5. Loss function.

[0084] The optimization objective of this invention is the final predicted instance mask. This invention employs Hungarian loss to calculate the two-stage loss. Hungarian loss is an effective mechanism in instance segmentation tasks to solve the assignment problem between predicted and ground truth labels. It uses the Hungarian algorithm (also known as the Kuhn-Munkres algorithm) to establish the optimal one-to-one match between predicted and ground truth labels. This algorithm is a classic combinatorial optimization algorithm that can find the minimum cost perfect match in a bipartite graph in polynomial time, thus efficiently solving the assignment problem and is therefore widely used in modern instance segmentation frameworks. This invention performs loss supervision at each decoding stage of the model. This staged supervision design is a key manifestation of the decoupling idea of ​​this invention, forcing functional decoupling and specialization between the two decoding stages. The total training loss of the model is the sum of the Hungarian losses at each stage, expressed as: ; For each stage For prediction-true pairs successfully matched using the Hungarian algorithm, its loss is... It consists of a classification item and a mask segmentation item, as shown in the following formula: ; in, It is the optimal match found by the Hungarian algorithm. It is the number of instance pairs matched in the i-th decoding stage, that is, the number of instance features that match the real instance in the corresponding decoding stage. It is for predicting semantic categories The standard negative log-likelihood loss, also known as cross-entropy loss, is used to penalize classification errors. Used to evaluate the prediction mask With the real mask The differences between them. The calculation methods for the relevant information involved in this section can be found in the previous explanation, and will not be repeated here.

[0085] To achieve robust and accurate segmentation, this invention employs a composite mask loss, which is the sum of the binary cross-entropy (BCE) loss and the Dice loss, expressed as: ;

[0086] in, Binary Cross-Entropy Loss (BCE loss) treats segmentation as a pixel-by-pixel binary classification problem. It is very effective in ensuring pixel-level detail accuracy, but its drawback is that when there is a severe imbalance in the number of pixels between the target object and the background, the loss may be dominated by the dominant background pixels, leading to unstable training. Dice loss is a region-overlap-based loss function derived from the Dice coefficients. It directly optimizes the overlapping region between the predicted and ground truth masks. Its main advantage lies in its strong robustness to severe class imbalances, common in segmentation tasks (e.g., a small object might only occupy a small portion of the image pixels), effectively preventing the model from ignoring small targets. This mature technique leverages the complementary advantages of both methods. It ensures fine pixel-level accuracy, while Dice loss guarantees good overall region overlap. This combination leads to a more stable training process and superior overall segmentation performance.

[0087] In particular, in this embodiment of the invention, while training the two decoding modules based on the above total training loss, the color fusion module in the encoding part is also trained at the same time.

[0088] III. Effect Verification.

[0089] like Figure 7 As shown, examples of the actual effects of the present invention are illustrated. Each row corresponds to an actual example. The first to last columns correspond to: semantic reference image; component reference image; color reference image; target image; ground truth mask for the first decoding stage (Stage1 GT); ground truth mask for the second decoding stage (Stage2 GT); the segmentation stage of the first stage (Stage1 Pred), i.e., conventional context segmentation; and the segmentation result of the second stage (Stage2 Pred), i.e., appearance constraint context segmentation.

[0090] To comprehensively evaluate the capabilities and advantages of the above-mentioned schemes provided in the embodiments of the present invention, comprehensive experiments were conducted on four benchmark datasets. Among them, PACO and PartImageNet datasets were specifically re-annotated and combined for the combined task to obtain ColorPACO and ColorPartImageNet. At the same time, the standard COCO-Ins and COCO-20i datasets were used to evaluate the general segmentation capabilities.

[0091] The evaluation metrics used are as follows: (1) Segmentation accuracy index:

[0092] Intersection-over-Union (IoU): This measures the degree of overlap between the predicted mask and the true mask; a higher IoU is better.

[0093] Average Precision (AP): Used to evaluate instance segmentation performance, taking into account both precision and recall; a higher AP is better. The AP at an IoU threshold of 0.5 is also reported (AP50).

[0094] (2) Instruction discrimination capability index: False Positive Rate (FPR): Used to evaluate the model's ability to distinguish invalid instructions. It is defined as the proportion of pixels that the model incorrectly predicts as foreground pixels out of all true background pixels under mismatched instructions. The calculation formula is shown below: ;

[0095] Here, FP (false positive) is the number of background pixels incorrectly predicted as foreground, and TN (true negative) is the number of background pixels correctly predicted as background. A lower FPR indicates that the model is better at accurately interpreting and rejecting invalid instructions, rather than segmenting randomly.

[0096] 1. Experimental details.

[0097] In the experiments, this invention was implemented using the PyTorch open-source framework, and pre-trained weights from SINE were used for initialization. To balance combinatorial understanding and general segmentation capabilities, the model was jointly trained on the ColorPACO and COCO-Ins datasets with a 1:1 sampling ratio. The optimizer used was AdamW, with an initial learning rate of... The batch size was set to 160, and 50 epochs were trained on 8 NVIDIA A6000 GPUs.

[0098] To enhance discrimination during training, samples are divided into positive and negative samples. Positive samples are those where the object, part, and color described in the prompt all exist in the target image. Negative samples are those where the target image contains an object of the prompt category, but its part and color attributes do not match. For negative samples, the mask is all zeros (i.e., an empty mask), and the ratio of positive to negative samples is set to 4:1 during training.

[0099] 2. Experimental results.

[0100] Comparison Method: To ensure the fairness and effectiveness of the comparison, this invention is compared with two mainstream methods, and the official source code and pre-trained weights are used preferentially.

[0101] (1) In-context Segmentation: including SegGPT, Matcher, SINE, and LDIS. To adapt to such methods, this invention provides a reference image as a visual cue that perfectly matches the "semantic-part-color" description of the target image.

[0102] (2) Referring Segmentation: including OMG-LLaVA, PSALM, HyperSeg, and DETRIS. This invention follows their standard usage, formatting text instructions as "a [obj] with a [color][part]".

[0103] Table 1 shows the evaluation results on three datasets, demonstrating comprehensive superiority of the method presented in this invention across all key tasks. Table 2 shows the results for instance segmentation.

[0104] Table 1: Comparison of semantic segmentation evaluation metrics on COCO-20i, ColorPACO, and ColorPartImageNet datasets

[0105] On the ColorPACO dataset (core combinatorial task): the method of this invention achieved an IoU of 59.4% and an FPR of 1.1%. Compared with the highest-performing baseline model HyperSeg (IoU 40.7%, FPR 7.8%), the method of this invention improves the IoU by 19.2 percentage points while reducing the FPR by nearly an order of magnitude. This demonstrates that the invention can not only more accurately locate target regions defined by color, part, and object category, but also has excellent command discrimination ability, effectively avoiding erroneous segmentation when the command and image do not match.

[0106] On the ColorPartImageNet dataset (out-of-domain generalization task): the method of this invention demonstrates strong out-of-domain generalization ability, achieving an IoU of 84.7% and an FPR of 0.6%. It is noteworthy that while some context-in-context methods (such as SegGPT) have high IoU (88.6%), their FPR is also extremely high (23.3%), indicating that they tend to ignore attribute constraints in instructions and perform "oversegmentation." In contrast, the method of this invention maintains extremely high segmentation accuracy while keeping a very low false positive rate, proving that its understanding of combined instructions is not limited to the training domain.

[0107] On the COCO-20i dataset (a general segmentation task): the method of this invention achieved an IoU of 64.9%, on par with the strong SINE baseline performance. This result strongly demonstrates that the specialized designs introduced in this invention to enhance combinatorial understanding (such as two-stage training and part-color fusion modules) do not compromise the model's original basic segmentation performance, exhibiting good compatibility and robustness.

[0108] Table 2: Comparison of evaluation metrics for instance segmentation on the ColorPACO dataset

[0109] In the ColorPACO instance segmentation task, this invention also performs excellently, achieving an AP of 24.1% and an AP50 of 41.5%, significantly outperforming all comparable methods. This demonstrates that the advantages of this invention lie not only in pixel-level accurate classification but also in the effective differentiation of different instances.

[0110] In summary, a comprehensive quantitative comparison shows that the method of this invention has significant advantages in handling complex combined instruction segmentation tasks. It not only surpasses existing technologies in segmentation accuracy and instance discrimination ability, but also has generalization ability and the ability to identify invalid instructions. To a certain extent, it solves the shortcomings of existing models in understanding fine-grained attributes and has high application value.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0112] Example 2 This invention also provides a multi-image-driven fine-grained segmentation system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 8 As shown, the system mainly includes: The encoding module is used to encode the input target image, semantic reference image, component reference image and color reference image respectively to obtain the corresponding encoded features, which are called target image feature map, semantic prototype, component prototype and color feature map; The first decoding module, applied to the first decoding stage, is used to perform instance query using a set of learnable features, combine semantic prototypes with target image feature maps, locate all object instances belonging to the specified semantic category in the target image, output the first stage instance features, generate an instance mask using the first stage instance features, and output the semantic prototype processed in the first decoding stage. The second decoding module, applied in the second decoding stage, enhances the color feature map with the component prototype and fuses it with the semantic prototype processed in the first decoding stage to obtain combined prototype features. Then, it connects these combined features with the semantic prototype processed in the first decoding stage to obtain multi-category prototype features. It generates a weighted target feature map using an instance mask. It refines the instance features of the first stage by combining the multi-category prototype features and the weighted target feature map to obtain refined instance features. At the same time, it transforms the multi-category prototype features to obtain the multi-category prototype features processed in the second decoding stage. The segmentation module is used to segment the refined instance features based on the multi-category prototype features processed in the second decoding stage and based on a dual matching mechanism to obtain the final segmentation mask.

[0113] Since the foregoing embodiments have already described the main technical details of the system in detail, they will not be repeated here.

[0114] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0115] Example 3 The present invention also provides a processing device, such as Figure 9 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0116] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0117] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0118] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0119] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0120] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A fine-grained segmentation method driven by multiple images, characterized in that, include: The input target image, semantic reference image, component reference image and color reference image are encoded to obtain the corresponding encoded features, which are called target image feature map, semantic prototype, component prototype and color feature map; In the first decoding stage, a set of learnable features are used to perform instance query. Combining the semantic prototype and the target image feature map, all object instances belonging to the specified semantic category are located in the target image. The first-stage instance features are output, and the instance mask is generated using the first-stage instance features. The semantic prototype processed in the first decoding stage is also output. In the second decoding stage, the color feature map and the component prototype are mutually enhanced, and then fused with the semantic prototype processed in the first decoding stage to obtain the combined prototype features. These features are then connected with the semantic prototype processed in the first decoding stage to obtain multi-category prototype features. A weighted target feature map is generated using an instance mask; the instance features in the first stage are refined by combining the multi-class prototype features and the weighted target feature map to obtain the refined instance features; at the same time, the multi-class prototype features are transformed to obtain the multi-class prototype features processed in the second decoding stage. The multi-category prototype features processed in the second decoding stage are used to segment the refined instance features based on a dual matching mechanism to obtain the final segmentation mask.

2. The fine-grained segmentation method driven by multiple images according to claim 1, characterized in that, The encoding of the input target image, semantic reference image, component reference image, and color reference image includes: The target image, semantic reference image, and component reference image are encoded using pre-trained visual models to obtain the target image feature map. Semantic reference image features Features of component reference images Mask pooling is used to extract semantic reference image features. Features of component reference images Extract the corresponding semantic prototypes from each. With component prototype ; The color reference image and the target image are converted to the same color space, and the color difference is calculated pixel by pixel to obtain a color similarity map. The color similarity map is generated using a convolutional network. The affine transformation parameters, and the feature map of the target image. Modulation is performed to obtain a color feature map. .

3. The fine-grained segmentation method driven by multiple images according to claim 1, characterized in that, The method utilizes a set of learnable features for instance querying, combining semantic prototypes with target image feature maps to locate all object instances belonging to a specified semantic category in the target image, and outputs the first-stage instance features, including: semantic prototype through Transformer blocks and target image feature map Perform interactive processing to obtain the semantic prototype after processing by the Transformer block. and target image feature map The first decoder in the first decoding stage utilizes a set of learnable features. To perform instance queries, the target image feature map is processed by the Transformer block using self-attention and cross-attention mechanisms. To interact, and in the semantic prototype processed by the Transformer block. Guided by this process, instance queries gradually learn and focus on regions in the image that match the specified semantic category. This stage ultimately outputs the first-stage instance features. .

4. The fine-grained segmentation method driven by multiple images according to claim 1, characterized in that, The step of mutually enhancing the color feature map and the component prototype, and fusing it with the processed semantic prototype obtained in the first decoding stage to obtain the combined prototype feature includes: Using component prototypes Guide and modulate color feature maps The semantically guided feature map is obtained. , is represented as: ； in, Represents the calculation of patch-by-patch cosine similarity in space; Using semantically guided feature maps Reverse reinforcement component prototype This generates a visually perceptible component feature. ; The perceived features of the component The processed semantic prototype obtained in the first decoding stage The final combined prototype features are obtained through fusion. .

5. The fine-grained segmentation method driven by multiple images according to claim 3, characterized in that, The use of instance mask The generated weighted target feature map includes: Instance mask The foreground score map is merged into a single comprehensive foreground score map, which is then combined with the target image feature map processed by the Transformer block. Perform dot product weighting to obtain the weighted target feature map. .

6. The fine-grained segmentation method driven by multiple images according to claim 1, characterized in that, The refinement of the first-stage instance features by combining multi-category prototype features and weighted target feature maps yields the following refined instance features: In the second decoder of the second decoding stage, a weighted target feature map is used. And through multi-category prototype features To guide the first phase of instance features The process of refining the data yields the refined instance features. .

7. The fine-grained segmentation method driven by multiple images according to claim 1, characterized in that, The first decoding stage is implemented through the first decoding module, and the second decoding stage is implemented through the second decoding module. The Hungarian algorithm is used to calculate the loss of the two decoding stages and optimize the two decoding modules. Loss in two decoding stages for: ； in, The loss for the i-th decoding stage, Corresponding to the first decoding stage and the second decoding stage; Loss in the i-th decoding stage , is represented as: ； in, It is the optimal match found by the Hungarian algorithm. It is the number of instance pairs matched in the i-th decoding stage, that is, the number of instance features that match the real instance in the corresponding decoding stage; It is for predicting semantic categories The standard negative log-likelihood loss is used. In the first decoding stage, the first-stage instance features are used to predict the semantic category. In the second decoding stage, the refined instance features are used to predict the semantic category. Used to evaluate the prediction mask With the real mask The difference between the two is that in the first decoding stage, the first stage instance features are used to obtain the prediction mask, while in the second decoding stage, the refined instance features are used to obtain the prediction mask.

8. A fine-grained segmentation system driven by multiple images, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The encoding module is used to encode the input target image, semantic reference image, component reference image and color reference image respectively to obtain the corresponding encoded features, which are called target image feature map, semantic prototype, component prototype and color feature map; The first decoding module, applied to the first decoding stage, is used to perform instance query using a set of learnable features, combine semantic prototypes with target image feature maps, locate all object instances belonging to the specified semantic category in the target image, output the first stage instance features, generate an instance mask using the first stage instance features, and output the semantic prototype processed in the first decoding stage. The second decoding module, applied in the second decoding stage, enhances the color feature map with the component prototype and fuses it with the semantic prototype processed in the first decoding stage to obtain combined prototype features. Then, it connects these combined features with the semantic prototype processed in the first decoding stage to obtain multi-category prototype features. It generates a weighted target feature map using an instance mask. It refines the instance features of the first stage by combining the multi-category prototype features and the weighted target feature map to obtain refined instance features. At the same time, it transforms the multi-category prototype features to obtain the multi-category prototype features processed in the second decoding stage. The segmentation module is used to segment the refined instance features based on the multi-category prototype features processed in the second decoding stage and based on a dual matching mechanism to obtain the final segmentation mask.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images