Training-free reference instance segmentation reasoning and memory bank construction method
By employing a training-free reference instance segmentation method, combined with candidate mask generation and semantic awareness fusion, the high training cost and insufficient generalization ability of existing technologies are addressed, achieving efficient and automated instance-level segmentation suitable for accurate recognition in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN HUAHAN WEIYE TECH
- Filing Date
- 2025-11-25
- Publication Date
- 2026-04-28
AI Technical Summary
Existing instance segmentation methods rely on large-scale labeled data for training, which is time-consuming and costly. They have limited generalization ability and are difficult to adapt to new target categories. Furthermore, traditional methods are not robust enough in complex scenarios and are prone to misidentification and missed detection.
We adopt a training-free reference instance segmentation method, which integrates candidate mask generation, feature extraction and semantic awareness fusion with in-memory library construction to achieve automated segmentation without fine-tuning. We use a pre-trained model to generate candidate masks and optimize the prediction results through a soft suppression mechanism to improve segmentation accuracy and robustness.
It achieves efficient and automated instance-level segmentation, can quickly identify new target categories without retraining, reduces computational overhead and deployment costs, improves segmentation accuracy and cross-domain adaptability, and is suitable for instance identification in complex scenarios.
Smart Images
Figure CN121937709A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine vision, specifically to a training-free reference instance segmentation inference and in-memory library construction method. Background Technology
[0002] In the field of machine vision, deep learning-based image segmentation technology has been widely applied in various scenarios such as industrial inspection, medical diagnosis, and agricultural remote sensing. The core idea of current mainstream instance segmentation methods is to learn the mapping relationship from reference images to target images by training on a large-scale dataset.
[0003] However, existing methods heavily rely on large-scale, finely labeled pixel-level data for training, which is time-consuming, costly, and computationally expensive, resulting in poor real-time performance. Furthermore, when faced with new target categories, model fine-tuning is required, leading to the reintroduction of training costs. Since the model's performance is highly dependent on the distribution of training data, its performance drops sharply when it recognizes a completely new object, usually requiring data collection and fine-tuning, thus reintroducing training costs and limiting generalization ability. In addition, traditional post-processing methods typically employ a rigid "black and white" suppression mechanism, where the lower-scoring candidate is directly deleted once the spatial overlap between two candidates exceeds a threshold. This strategy is extremely sensitive to threshold settings and lacks adaptability to different scenarios.
[0004] Therefore, designing an automated instance segmentation method that is training-free, highly efficient, and has strong generalization ability has become an urgent problem to be solved. Summary of the Invention
[0005] The main technical problem this invention addresses is how to design an automated instance segmentation method that is training-free, highly efficient, and has strong generalization capabilities.
[0006] According to the first aspect, one embodiment provides a training-free reference instance segmentation reasoning method, comprising:
[0007] Obtain the target image to be segmented, and perform feature extraction on the target image to obtain the dense feature map corresponding to the target image;
[0008] The target image is input into the candidate mask generation unit to generate multiple candidate masks;
[0009] Each candidate mask and the dense feature map corresponding to the target image are input into the candidate feature extraction unit to obtain the candidate feature representation corresponding to each candidate mask;
[0010] Obtain a memory library containing multiple categories, each corresponding to a category prototype;
[0011] The candidate feature representation corresponding to each candidate mask is input into the similarity calculation and classification unit to obtain the initial prediction result corresponding to each candidate mask based on the candidate feature representation corresponding to each candidate mask and all category prototypes in the memory library;
[0012] All candidate masks and their corresponding candidate feature representations and initial prediction results are input into the semantic perception fusion unit to optimize the initial prediction results corresponding to each candidate mask through a soft suppression mechanism, so as to obtain the optimized prediction results of each candidate mask.
[0013] The segmentation result corresponding to the target image is obtained by filtering the optimized prediction results for each candidate mask.
[0014] According to the second aspect, one embodiment provides a method for constructing a training-free reference instance segmentation memory library, comprising:
[0015] Obtain a reference image set, which contains multiple reference images, wherein each reference image corresponds to one or more instance masks;
[0016] For any reference image: perform feature extraction on the reference image to obtain a dense feature map of the reference image; based on the dense feature map of the reference image and its corresponding instance mask, obtain the feature subset corresponding to each instance in the reference image;
[0017] The feature subsets corresponding to all instances are stored according to category units. All category units constitute a memory library. The resulting memory library is applied to the reference instance segmentation inference method. For example, the resulting memory library can be applied to the above-mentioned training-free reference instance segmentation inference method.
[0018] According to a third aspect, one embodiment provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the above-described training-free reference instance segmentation inference method and training-free reference instance segmentation memory library construction method.
[0019] According to the above embodiment, a training-free reference instance segmentation inference method creates a semantically aware suppression criterion by combining the spatial overlap and feature similarity between candidate masks. When the features of overlapping objects are highly similar, strong suppression is applied; otherwise, they are retained. This enables the inference process to accurately identify independent individuals in complex layouts, significantly improving segmentation accuracy in dense and complex scenes. This semantically aware soft fusion mechanism makes the semantic instance identification capability of the inference process more accurate. Based on the flexible and robust soft suppression mechanism, a "soft" score decay strategy is adopted. The confidence score of each candidate instance is dynamically reduced according to the spatial and semantic similarity between each candidate instance and a candidate instance with a higher confidence score, rather than directly deleting instances judged as redundant. This progressive processing method provides greater flexibility in handling ambiguous and difficult overlap situations, effectively reducing the false negative rate caused by erroneous deletion, and improving the robustness of the automated segmentation framework and the completeness of the final result. Furthermore, since the entire network framework is built on a pre-trained base model, but the core semantic perception fusion unit does not contain any trainable parameters, it achieves a completely training-free, fine-tuning-free, plug-and-play automated instance segmentation method that can be directly applied to new object categories and visual domains. It completely eliminates the dependence on labeled data and high training costs, ensuring its powerful cross-domain generalization ability, making it a highly automated and easy-to-deploy general instance segmentation system.
[0020] According to the above embodiment, a training-free reference instance segmentation in-memory database construction method first extracts the instance mask corresponding to each instance from each reference image in the labeled reference image set, thereby obtaining the feature subset corresponding to each instance. Using the pixel-level instance mask as a guide, it is ensured that only features belonging to the target instance are stored in the in-memory database. While avoiding the interference of background noise, a raw feature pool organized by category is constructed. Then, through two-stage hierarchical aggregation, the scattered, low-level pixel-level raw features are refined into highly generalized and stable category-level concepts, obtaining the category prototype corresponding to each category. This allows the segmentation and recognition of the new target category to be achieved while ensuring inference accuracy when facing a new target category, only the category units in the in-memory database need to be added and the corresponding category prototype is obtained through a small number of samples. For different domains, the in-memory database is updated by obtaining the category prototypes corresponding to different categories in that domain, without introducing new training costs. Thus, the reference instance segmentation inference method only requires a small number of reference samples to achieve fast and accurate instance-level segmentation of target images in multiple domains and large scales. Attached Figure Description
[0021] Figure 1 This is a flowchart of a training-free reference instance segmentation reasoning method;
[0022] Figure 2 This is a flowchart of a training-free reference instance segmentation memory library construction method;
[0023] Figure 3 This is a schematic diagram of the network structure of the automated segmentation framework;
[0024] Figure 4 This is a schematic diagram of the target image to be segmented.
[0025] Figure 5 This is a schematic diagram of the segmentation results of the target image. Detailed Implementation
[0026] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.
[0027] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.
[0028] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).
[0029] Existing methods rely on large-scale manually labeled data for training. Acquiring this data requires a significant investment of time, manpower, and cost, which severely restricts the promotion and application of algorithms in practical engineering. In recent years, suggestive segmentation models (such as the Segment Anything model) have emerged. Although they can reduce the labeling cost to some extent, they rely on manual interactive prompts (such as points, boxes, and coarse masks) or complex domain-specific prompt generation mechanisms. Directly using such models makes it difficult to meet the segmentation needs of automated, large-scale, and cross-domain applications in industrial production scenarios.
[0030] Traditional segmentation methods face several significant challenges: First, their scalability is limited, as their reliance on human prompts makes them difficult to scale for applications requiring automated batch processing. Second, their generalization ability is insufficient, as existing automated processes are designed for specific domains and contain numerous domain-specific heuristics, lacking cross-domain universality. Third, they still require time-consuming training; as alternatives, reference segmentation methods often still require model fine-tuning when faced with new target categories, reintroducing training costs.
[0031] Furthermore, traditional segmentation methods struggle to infer the complete shape of objects with severe occlusion, such as when more than half of an object is obscured. This can easily lead to misidentification as multiple instances or missed detection altogether. Additionally, the feature maps of small objects gradually lose spatial information during downsampling operations like pooling, making them difficult to detect and segment. Therefore, traditional segmentation methods lack robustness when dealing with complex scenes. Moreover, they are prone to class confusion when faced with objects that have complex textures, similar appearances, or ambiguous semantics.
[0032] In response to the characteristics of existing technologies such as "reliance on complex distance metrics", "lack of instance-level identification capabilities" and "reliance on manual threshold tuning", this invention aims to solve the difficulties faced by the industry in deploying advanced segmentation models, such as high development costs and poor cross-domain adaptability. It significantly improves the automation level and application scope of segmentation models, thereby greatly improving the overall performance of automated instance segmentation.
[0033] This invention follows the human cognitive paradigm of "learning by analogy," that is, "learning concepts from a small number of samples and then applying them to new things." By innovatively integrating the semantic understanding ability of the visual basic model with the precise localization ability of the prompting model, an automated segmentation framework that does not require fine-tuning is constructed. This automated segmentation framework only requires a small number of reference samples to perform fast and accurate instance-level segmentation of target images in multiple fields and on a large scale, thereby achieving a truly efficient, universal, and low-cost automated segmentation solution.
[0034] Compared to similar methods, this invention achieves accurate differentiation of overlapping instances in complex scenes through an innovative two-stage feature aggregation and semantic-aware merging strategy, improving segmentation accuracy and realizing efficient instance-level identification. By adopting a soft score decay strategy, it effectively reduces the false deletion rate, and this flexible soft suppression mechanism ensures the integrity of the inference results. Furthermore, it achieves a completely training-free system architecture, supports cross-domain zero-shot generalization, and significantly reduces deployment costs and computational overhead. In short, this invention achieves efficient, accurate, and fully automatic instance-level segmentation of any image, meeting the real-time processing needs of practical industrial applications.
[0035] The automated segmentation framework in this embodiment is as follows: Figure 3 As shown, it mainly consists of a memory library construction unit B, a feature aggregation unit A, and an inference segmentation unit S. When performing instance segmentation based on this automated segmentation framework, the entire segmentation process can be divided into two stages: the preparation stage and the inference stage.
[0036] In the preparation phase, the original features (feature subset) of each instance are extracted and stored from the reference image by the memory library construction unit B. Then, the original features corresponding to each instance are further refined and abstracted by the feature aggregation unit A to generate a robust category prototype. Thus, the category prototype corresponding to each category is obtained and stored in the memory library.
[0037] In the inference phase, the target image to be segmented is input into the inference segmentation unit S for end-to-end automated processing. The inference segmentation unit S includes a candidate mask generation unit G, a candidate feature extraction unit C, a similarity calculation and classification unit P, and a semantic-aware fusion unit F. Specifically, the candidate mask generation unit G generates multiple candidate masks corresponding to the target image; then, based on the dense feature map corresponding to the target image, the candidate feature extraction unit C obtains the feature subset corresponding to each candidate mask; next, the similarity calculation and classification unit P matches the feature subset corresponding to each candidate mask with all category prototypes in the memory database to obtain the initial prediction result for each candidate mask; finally, the semantic-aware fusion unit F optimizes the initial prediction results corresponding to all candidate masks by comprehensively considering the spatial overlap and feature similarity between candidate masks to suppress redundant detection and retain true overlapping instances (such as closely adjacent or partially overlapping different instances), thus obtaining the final instance segmentation result of the target image.
[0038] Please refer to Figure 1 Some embodiments provide a training-free reference instance segmentation inference method, which includes the following steps:
[0039] Step S100: Obtain the target image to be segmented, extract features from the target image, and obtain the dense feature map corresponding to the target image.
[0040] A schematic diagram of the target image to be segmented obtained in this embodiment is shown below. Figure 4 As shown, in the target image, the cracks that need to be segmented are similar in color to the black spots in the background, and because the distribution of these black spots is not uniform, dark stripes of different shapes will be formed in some directions; in some areas, the contrast between the spots and the substrate may be higher than that of the real cracks; in addition, a real crack may terminate at a black spot or extend along the boundary of the spot, etc. The existence of these factors can easily lead to misidentification of the target by traditional segmentation methods.
[0041] In this embodiment, the obtained target image is input into the inference segmentation unit S of the automated segmentation framework. A pre-trained visual feature extractor is used to extract high-dimensional features from the target image, resulting in a dense feature map corresponding to the target image. The visual feature extractor is mainly used to extract deep, multi-scale features containing rich semantic information from the target image during the inference stage and the reference image during the preparation stage, obtaining dense feature maps corresponding to the target image and the reference image. This provides a high-quality representation foundation for subsequent feature matching and recognition.
[0042] To balance strong generalization performance and inference effect, this embodiment preferably adopts the DINO series network structure as the network structure of the visual feature extractor. The DINO series network structure is a self-supervised learning model based on Vision Transformer, which has the advantages of "strong zero-shot generalization ability", "cross-domain adaptation without fine-tuning" and "rich feature semantic information", and maintains excellent high-precision recognition performance even in unknown domains.
[0043] Step S110: Input the target image into the candidate mask generation unit to generate multiple candidate masks.
[0044] The role of the candidate mask generation unit G is to automatically segment all potential regions that may be target objects from the target image using a pre-trained cue-based segmentation model without requiring any prior knowledge of the categories, thereby generating a set of high-quality candidate masks.
[0045] To achieve category-independent and high-quality segmentation, this embodiment preferably employs a multi-scale semantic segmentation network (MSN), such as the SAM network, as the model structure for the candidate mask generation unit G. MSNs offer advantages such as high-quality segmentation, category independence, and automated suggestion mechanisms, enabling efficient generation of comprehensive candidate objects. The target image is used as input to the candidate mask generation unit, which, through its internal network point auto-suggestion mechanism, outputs multiple candidate masks corresponding to the target image for subsequent classification and filtering by other units.
[0046] In this context, there may be one or more instances in the target image. Among the multiple candidate masks generated based on the target image, different candidate masks may correspond to different instances, or they may correspond to the same instance (such as multiple candidate masks actually being masks of the same instance with different semantic granularities). In this embodiment, the "instance" corresponding to each candidate mask at this time is called a "candidate instance".
[0047] Step S120: Input each candidate mask and the dense feature map corresponding to the target image into the candidate feature extraction unit to obtain the candidate feature representation corresponding to each candidate mask.
[0048] In this embodiment, the candidate feature extraction unit C is a key bridge connecting spatial localization and semantic recognition. Its core design idea is to efficiently transform the spatial domain candidate mask output by the candidate mask generation unit G into a feature representation (i.e., candidate feature representation) that can be compared in the semantic space through matrix multiplication pooling. This achieves accurate alignment and efficient fusion of features and masks, and realizes a feature extraction paradigm that is extremely efficient in computation and highly accurate in space, greatly improving the inference speed of the entire segmentation process.
[0049] The candidate feature extraction unit C receives the dense feature map corresponding to the target image extracted by the visual feature extractor and all candidate masks corresponding to the target image. At this time, each candidate mask corresponding to the target image corresponds to a candidate instance.
[0050] In the candidate feature extraction unit C, for any candidate mask: the spatial size of the candidate mask is resampled to make the size of the candidate mask the same as that of the dense feature map corresponding to the target image; the resampled candidate mask is multiplied with the dense feature map corresponding to the target image to obtain the candidate feature subset of the candidate instance corresponding to the candidate mask; the candidate feature subset corresponding to the candidate instance is normalized to obtain the candidate feature representation corresponding to the candidate mask.
[0051] For example, for the c-th candidate mask M of the target image c First, for the candidate mask M c A resampling operation is performed so that the resampled candidate mask has the same size as the dense feature map of the target image. Then, the resampled candidate mask is multiplied by the corresponding dense feature map of the target image to obtain the high-dimensional feature vectors corresponding to all pixels in the area covered by the candidate mask. At this point, all the obtained high-dimensional feature vectors constitute the candidate feature subset of the candidate instance corresponding to the candidate mask. Then, the candidate feature subset corresponding to the candidate mask is L1 normalized to obtain the candidate feature representation corresponding to the candidate mask.
[0052] Step S130: Obtain a memory library containing multiple categories, each corresponding to a category prototype.
[0053] In this embodiment, the memory library and the category prototype corresponding to each category in the memory library are obtained through the memory library construction unit B and the feature aggregation unit A in the preparation stage. The memory library is obtained through the memory library construction unit B, which uses pixel-level instance masks as guidance to ensure that only features belonging to the target instance are stored in the memory library. While avoiding the interference of background noise, an original feature pool organized by category is constructed as the input of the subsequent feature aggregation unit A to obtain the category prototype corresponding to each category in the memory library.
[0054] In memory library construction unit B, a reference image set is first obtained. This reference image set covers all categories and contains multiple reference images. Each reference image has been pre-annotated with all target instances in that reference image. Each reference image in the set corresponds to one or more instance masks. Then, for the r-th reference image I... r Let M be the mask of all instances corresponding to the reference image. r ={M r1 M r2 M rt}, where t is the total number of target instances contained in the reference image, and M rt This represents the t-th instance mask corresponding to the r-th reference image.
[0055] Then, for any reference image in the set of reference images for the labeled instance: feature extraction is performed on the reference image to obtain the dense feature map of the reference image; based on the dense feature map of the reference image and the mask of each instance corresponding to it, the feature subset (original features) corresponding to each instance in the reference image is obtained; the feature subsets corresponding to all instances are stored according to category units, and all category units constitute a memory library.
[0056] Using the r-th reference image I r For example, the reference image I is first processed using a pre-trained visual feature extractor. r High-dimensional feature extraction is performed to obtain the dense feature map F of the reference image. r At this point, each pixel in the reference image corresponds to a high-dimensional feature vector; and each instance mask corresponding to the reference image is processed by changing the image resolution (or resampling) to make the resolution of each instance mask of the reference image consistent with the size of the dense feature map corresponding to the reference image; then the dense feature map of the reference image and its corresponding instance mask, such as M rtElement-wise multiplication is performed to accurately extract all feature vectors corresponding to each instance. All feature vectors corresponding to each instance constitute the feature subset corresponding to that instance, thus obtaining the feature subset corresponding to each instance in the reference image.
[0057] All instances in all reference images are divided according to their category. Then, the feature subsets corresponding to all instances in a single category are stored in the same category cell of the memory bank. At this time, the memory bank contains multiple memory cells, and each memory cell contains multiple instances and their corresponding feature subsets.
[0058] The core design idea of feature aggregation unit A in this embodiment is to adopt a hierarchical abstraction paradigm from instance to category. Based on the commonalities between different instances belonging to the same category, through two-stage hierarchical aggregation, the scattered, low-level pixel-level original features are extracted into a category-level concept with high generalization and stability. This process not only provides a reliable benchmark for subsequent accurate semantic matching, but also enables the entire network framework to have a powerful few-sample learning ability to efficiently summarize the core characteristics of the category from a very small number of samples.
[0059] Feature aggregation unit A further refines and abstracts the original features (feature subsets corresponding to each instance) stored in in-memory library construction unit B based on a two-stage aggregation strategy, ultimately generating stable and robust category prototypes representing each category. Feature aggregation unit A receives the feature subsets corresponding to each instance in each category unit of the in-memory library and obtains the category prototype for each category based on the feature subsets corresponding to each instance in each category unit. Specifically, for any category in the in-memory library: based on the feature subsets corresponding to each instance in the category unit corresponding to that category, the instance-level prototype corresponding to that instance is obtained; based on all instance-level prototypes corresponding to that category, the category prototype corresponding to that category is obtained.
[0060] For example, feature aggregation unit A first receives a subset of features belonging to the same instance, performs average pooling on the feature subset corresponding to the instance to obtain an instance-level prototype that can represent the instance; then it receives all instance-level prototypes belonging to the same category, performs average pooling on all instance-level prototypes corresponding to the category again, and finally obtains a robust and highly generalized category prototype that can represent the category.
[0061] It should be noted that traditional methods require fine-tuning the model when facing new target categories, leading to the reintroduction of training costs. However, in this embodiment, it is only necessary to increase the category units in the memory library and obtain the corresponding category prototypes through a small number of samples to achieve segmentation and recognition of new target categories while maintaining inference accuracy. For different domains, the memory library can be updated by obtaining the category prototypes corresponding to different categories in that domain, and subsequent inference can be performed based on the inference segmentation units using this memory library.
[0062] Step S140: Input the candidate feature representation corresponding to each candidate mask into the similarity calculation and classification unit to obtain the initial prediction result corresponding to each candidate mask based on the candidate feature representation corresponding to each candidate mask and all category prototypes in the memory library.
[0063] The similarity calculation and classification unit P is used to perform the core recognition and classification tasks. The initial prediction result for each candidate mask includes the category label and the initial confidence score of the candidate mask. Specifically, for any candidate mask: the similarity between the candidate feature representation corresponding to the candidate mask and all category prototypes in the memory is calculated, and the maximum similarity is taken as the initial confidence score of the candidate mask; the category to which the category prototype corresponding to the maximum similarity belongs is taken as the category label of the candidate mask.
[0064] For example, the similarity calculation and classification unit P receives the candidate feature representation corresponding to each candidate mask output by the candidate feature extraction unit C and all category prototypes in the memory bank, and performs normalization operations on each candidate feature representation and category prototype respectively. Then, for any candidate mask: the normalized candidate feature representation of the candidate mask is multiplied by each normalized category prototype to obtain the cosine similarity between the feature representation corresponding to the candidate mask and any category prototype; then, the largest cosine similarity among all cosine similarities corresponding to the candidate mask is obtained, the category to which the category prototype corresponding to the largest cosine similarity belongs is used as the category label of the candidate mask, and the largest cosine similarity is used as the initial confidence score corresponding to the candidate mask. At this time, the initial confidence score and the category label of the candidate mask constitute the initial prediction result of the candidate mask.
[0065] Step S150: Input each candidate mask and its corresponding candidate feature representation and the initial prediction result into the semantic perception fusion unit, so as to optimize the initial prediction result corresponding to each candidate mask through the soft suppression mechanism, and obtain the optimized prediction result of each candidate mask.
[0066] The semantic perception fusion unit F is responsible for refining the post-processing of a large number of preliminary classification results (i.e., initial prediction results) output by the similarity calculation and classification unit P. It is the key to ensuring the instance-level accuracy of the final output result. Its main goal is to directly solve the redundant detection problem generated by the candidate mask generation unit G. For example, the candidate mask generation unit G generates masks of different levels from the whole to the local of the target object. That is, multiple overlapping candidate masks actually point to the same object, and accurately distinguish multiple different instances that are closely adjacent or partially overlapping in complex scenes.
[0067] In this context, when the spatial overlap between two candidate masks of a target image is high, these two candidate masks may correspond to the same instance. However, in complex scenes, there may be situations where different instances are closely adjacent or partially overlapped. In addition to sharing the commonalities of the category, different instances of the same category also have their own unique and distinguishable features (i.e., individuality, such as appearance). Therefore, for two candidate masks belonging to different instances, their corresponding candidate feature representations also differ. Thus, this embodiment makes a comprehensive judgment on each candidate mask based on the spatial overlap and feature similarity between each candidate mask and other candidate masks, thereby suppressing redundant detection and retaining real overlapping instances.
[0068] In the semantic awareness fusion unit F, the candidate feature representations corresponding to all candidate instances and the initial prediction results output by the similarity calculation and classification unit P are first received; the categories are divided according to the category labels in the initial prediction results corresponding to all candidate masks, and the candidate masks corresponding to each category are sorted according to the initial confidence score, and the initial prediction results corresponding to each candidate mask in the category are optimized according to the obtained sorting results.
[0069] For example, all candidate instances belonging to the same category are sorted in descending order of their initial confidence scores in the initial prediction results. The higher the initial confidence score, the more reliable the category label corresponding to the candidate mask. Then, according to the sorting results, each candidate mask in the category is compared with other candidate masks, thereby optimizing the initial prediction results corresponding to each candidate mask based on the spatial overlap and feature similarity between the candidate masks.
[0070] For any candidate mask m in a category: the candidate mask m and all other candidate masks in the category generate a set of candidate masks respectively. At this time, the candidate mask m corresponds to multiple sets of candidate masks.
[0071] Based on the coverage area corresponding to candidate mask m in each group of candidate masks and the coverage area corresponding to another candidate mask, the spatial overlap of the candidate mask group is obtained.
[0072] Let any candidate mask belonging to the same category as candidate mask m be candidate mask m'. Then, for a set of candidate masks formed by candidate mask m and candidate mask m': obtain the intersection area between the area covered by candidate mask m and the area covered by candidate mask m'; obtain the spatial overlap of the set of candidate masks based on the area of the intersection area and the area covered by candidate mask m. At this time, the spatial overlap of the set of candidate masks formed by candidate mask m and candidate mask m' can be expressed as:
[0073]
[0074] Where IoS(m, m') represents the spatial overlap of a set of candidate masks formed by candidate mask m and candidate mask m'; m∩m' represents the intersection area between the coverage areas of candidate mask m and candidate mask m'; Area(m∩m') represents the area of the intersection area between the coverage areas of candidate mask m and candidate mask m', i.e., the number of pixels covered by the intersection area; Area(m) represents the area covered by candidate mask m, i.e., the number of pixels contained in the area covered by candidate mask m.
[0075] Calculate the spatial overlap of the candidate mask m with each other candidate mask of the same category in turn to obtain multiple spatial overlaps of the candidate mask m.
[0076] To incorporate semantic information, this embodiment further obtains the feature similarity of the candidate mask group based on the candidate feature representation corresponding to the candidate mask m and the candidate feature representation corresponding to another candidate mask. Then, for a candidate mask group consisting of the candidate mask m and another candidate mask m' in the same category: normalization operations are performed on the candidate feature representation corresponding to the candidate mask m and the candidate feature representation corresponding to the other candidate mask m' respectively; the similarity is calculated based on the normalized candidate feature representation corresponding to the candidate mask m and the normalized candidate feature representation corresponding to the other candidate mask m' to obtain the feature similarity of the candidate mask group.
[0077] For example, if the candidate feature representations corresponding to candidate mask m and candidate mask m' are normalized using their respective L2 norms, then the feature similarity between the candidate masks m and m' can be expressed as:
[0078]
[0079]
[0080] in, Y represents the feature similarity between the candidate mask m and the candidate mask m' formed by the two candidate masks; m Y represents the candidate feature representation corresponding to the candidate mask m; m' ||Y represents the candidate feature representation corresponding to the candidate mask m'; m ||2 represents Y m L2 norm; ||Y m' ||2 represents Y m' L2 norm; D m D represents the normalized candidate feature representation corresponding to the candidate mask m; m' The normalized candidate feature representation corresponding to the candidate mask m' is indicated; "·" represents the vector dot product symbol.
[0081] In this embodiment, the suppression factor corresponding to the candidate mask is obtained based on the spatial overlap and feature similarity of the candidate mask. Then, the spatial overlap and feature similarity of each candidate mask are fused with soft weights to obtain the semantically aware suppression factor. Finally, the attenuation factor corresponding to the candidate mask m is obtained based on all the suppression factors corresponding to the candidate mask m.
[0082] For example, for a set of candidate masks formed by candidate mask m and candidate mask m', the spatial overlap and feature similarity of the candidate masks are multiplied together. The result of this multiplication is the suppression factor corresponding to the candidate mask set. Similarly, each set of candidate masks formed by candidate mask m and other candidate masks of the same category will yield a suppression factor, thus obtaining multiple suppression factors corresponding to candidate mask m. Then, the largest suppression factor among all suppression factors corresponding to candidate mask m is obtained, and the difference between the value 1.0 and the largest suppression factor is calculated. The square root of this difference is taken as the attenuation factor corresponding to candidate mask m. Therefore, the attenuation factor corresponding to candidate mask m can be expressed as:
[0083]
[0084] Where max represents the maximum value operation, f m This represents the attenuation factor corresponding to the candidate mask m. The larger the attenuation factor, the smaller the attenuation of the initial confidence score corresponding to the candidate mask. Conversely, the smaller the attenuation factor, the greater the attenuation of the initial confidence score corresponding to the candidate mask.
[0085] The initial confidence score of candidate mask m is optimized based on the attenuation factor corresponding to the candidate mask m to obtain the optimized confidence score of candidate mask m. In this embodiment, the initial confidence score of candidate mask m is multiplied by the attenuation factor, and the result of the multiplication is taken as the optimized confidence score of candidate mask m.
[0086] It should be noted that when two candidate masks correspond to the same instance, their feature similarity is high, but their spatial overlap will differ significantly. For example, one candidate mask may include the entire instance, while the other may only include a local portion of the instance. Although the intersection area between these two candidate masks is the same, the spatial overlap of the candidate mask that only includes a local portion will be greater than that of the candidate mask that includes the entire instance. In this case, the suppression factor obtained by the latter based on spatial overlap and feature similarity will be higher than that of the former. Furthermore, since the initial confidence score of the candidate mask that includes the entire instance is often also higher, the latter will ultimately obtain a lower confidence score after subsequent confidence score optimization. In other words, even within the same set of candidate masks, different candidate masks will not achieve the same degree of suppression.
[0087] When two candidate masks correspond to different instances that are closely adjacent or partially overlapping, their spatial overlap is smaller than when they correspond to the same instance. Since they correspond to different instances, the feature similarity between their corresponding candidate feature representations is also lower, and their corresponding suppression factor is also lower. Therefore, the suppression of the candidate mask is lower. The larger the corresponding decay factor, the smaller the decay of the initial confidence score, thus preserving instances that are different in appearance but spatially close.
[0088] The optimized prediction result of the candidate mask m is obtained based on the category label of the candidate mask m and the optimized confidence score. For example, the decay factor corresponding to the candidate mask m is multiplied by the initial confidence score corresponding to the candidate mask m, and the result of the multiplication is used as the optimized confidence score of the candidate mask; at this time, the optimized confidence score of the candidate mask m and its category label constitute the optimized prediction result of the candidate mask.
[0089] Step S160: Filter the optimized prediction results for each candidate mask to obtain the segmentation result corresponding to the target image.
[0090] After obtaining the optimized prediction result for each candidate mask, all candidate masks are filtered based on preset screening criteria, such as the optimized confidence score or the number to be retained, to obtain the final segmentation result of the target image. In this embodiment, the segmentation result of the target image is as follows: Figure 5 As shown.
[0091] For example, firstly, the candidate masks are classified according to the category labels in the optimized prediction results. Then, all candidate masks corresponding to each category are sorted according to the optimized confidence scores, and the candidate masks are retained according to the actual number of retained ones. The actual number of retained ones is the minimum of the number of candidate instances corresponding to each category and the preset number. The specific value of the preset number can be set according to the actual situation. In this embodiment, the preset number is 100.
[0092] This embodiment creates a semantically aware suppression criterion by combining the spatial overlap and feature similarity between candidate masks. Strong suppression is applied when overlapping objects have highly similar features, while they are retained otherwise. This allows the inference process to accurately identify independent individuals in complex layouts, significantly improving segmentation accuracy in dense and complex scenes. Compared to traditional methods that rely solely on spatial overlap for suppression, making it difficult to distinguish between "redundant detections" and "closely related different instances," the automated segmentation framework in this embodiment, based on this semantically aware soft fusion mechanism, possesses accurate semantic instance discrimination capabilities. Furthermore, unlike traditional "black and white" hard deletion methods, this embodiment uses a flexible and robust soft suppression mechanism, employing a "soft" score decay strategy. It dynamically reduces the confidence score of each candidate instance based on its spatial and semantic similarity to candidate instances with higher confidence scores, rather than directly deleting instances judged as redundant. This progressive processing provides greater flexibility in handling ambiguous and difficult overlap situations, effectively reducing the false negative rate caused by erroneous deletions and improving the robustness of the automated segmentation framework and the completeness of the final result. Furthermore, since the entire network framework is built on a pre-trained base model and the core semantic perception fusion unit does not contain any trainable parameters, this means that the embodiment of the present invention is a plug-and-play solution that does not require fine-tuning. It is a completely training-free automated architecture that can be directly applied to new object categories and visual domains, completely eliminating the dependence on labeled data and high training costs. This ensures its powerful cross-domain generalization ability, making it a highly automated and easy-to-deploy general instance segmentation system.
[0093] Please refer to Figure 2 Some embodiments provide a training-free method for constructing a reference instance segmentation memory library, which includes the following steps:
[0094] Step S200: Obtain a reference image set, which contains multiple reference images, each of which corresponds to one or more instance masks.
[0095] Step S210: For any reference image: perform feature extraction on the reference image to obtain a dense feature map of the reference image; based on the dense feature map of the reference image and its corresponding instance mask, obtain the feature subset corresponding to each instance in the reference image.
[0096] Step S220: Store the feature subsets corresponding to all instances according to category units. All category units constitute a memory library, and the resulting memory library is applied to the reference instance segmentation inference method.
[0097] In this embodiment, the category prototype corresponding to each category is obtained through a feature aggregation unit. Specifically, the feature subset corresponding to each instance in each category unit of the memory library is input into the feature aggregation unit, and the category prototype corresponding to each category is obtained based on the feature subset corresponding to each instance in each category unit. For any category in the memory library: the instance-level prototype corresponding to the instance is obtained based on the feature subset corresponding to each instance in the category unit corresponding to the category; and the category prototype corresponding to the category is obtained based on all instance-level prototypes corresponding to the category.
[0098] It should be noted that the various method steps in this embodiment can refer to step S130 in the above-described reference instance segmentation reasoning method. The specific implementation method has been described in the above embodiment and will not be repeated here.
[0099] This embodiment first extracts the instance mask corresponding to each instance from each reference image in the labeled reference image set, thereby obtaining the feature subset corresponding to each instance. Using the pixel-level instance mask as a guide, it ensures that only features belonging to the target instance are stored in the memory database. While avoiding the interference of background noise, it constructs an original feature pool organized by category. Then, through a two-stage hierarchical aggregation, the scattered, low-level pixel-level original features are refined into highly generalized and stable category-level concepts, obtaining the category prototype corresponding to each category. This allows the network model to be segmented and identified when facing a new target category without fine-tuning. It is only necessary to add category units in the memory database and obtain the corresponding category prototypes through a small number of samples, which can achieve segmentation and recognition of new target categories while ensuring inference accuracy. For different domains, the memory database can be updated by obtaining the category prototypes corresponding to different categories in that domain. Subsequent inference can then be performed based on the memory database through the inference segmentation unit.
[0100] Some embodiments of the present invention also disclose a computer-readable storage medium comprising a program executable by a processor to implement the reference instance segmentation inference method and the reference instance segmentation memory library construction method as described in any of the embodiments herein.
[0101] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.
[0102] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.
Claims
1. A training-free reference instance segmentation reasoning method, characterized in that, include: Obtain the target image to be segmented, and perform feature extraction on the target image to obtain the dense feature map corresponding to the target image; The target image is input into the candidate mask generation unit to generate multiple candidate masks; Each candidate mask and the dense feature map corresponding to the target image are input into the candidate feature extraction unit to obtain the candidate feature representation corresponding to each candidate mask; Obtain a memory library containing multiple categories, each corresponding to a category prototype; The candidate feature representation corresponding to each candidate mask is input into the similarity calculation and classification unit to obtain the initial prediction result corresponding to each candidate mask based on the candidate feature representation corresponding to each candidate mask and all category prototypes in the memory library; All candidate masks and their corresponding candidate feature representations and initial prediction results are input into the semantic perception fusion unit to optimize the initial prediction results corresponding to each candidate mask through a soft suppression mechanism, so as to obtain the optimized prediction results of each candidate mask. The segmentation result corresponding to the target image is obtained by filtering the optimized prediction results for each candidate mask.
2. The reference instance segmentation reasoning method as described in claim 1, characterized in that, The memory library is obtained through a memory library construction unit, wherein in the memory library construction unit: Obtain a reference image set, which contains multiple reference images, wherein each reference image corresponds to one or more instance masks; For any reference image: perform feature extraction on the reference image to obtain a dense feature map of the reference image; based on the dense feature map of the reference image and its corresponding instance mask, obtain the feature subset corresponding to each instance in the reference image; The feature subsets corresponding to all instances are stored according to category units, and all category units constitute an in-memory library.
3. The reference instance segmentation reasoning method as described in claim 2, characterized in that, The category prototype corresponding to each category is obtained through a feature aggregation unit; wherein, the feature aggregation unit receives a feature subset corresponding to each instance in each category unit in the memory library, and obtains the category prototype corresponding to each category based on the feature subset corresponding to each instance in each category unit.
4. The reference instance segmentation reasoning method as described in claim 3, characterized in that, In the feature aggregation unit, for any category in the memory library: based on the feature subset corresponding to each instance in the category unit corresponding to the category, the instance-level prototype corresponding to the instance is obtained; based on all instance-level prototypes corresponding to the category, the category prototype corresponding to the category is obtained.
5. The reference instance segmentation reasoning method as described in claim 4, characterized in that, The step of inputting each candidate mask and the dense feature map corresponding to the target image into the candidate feature extraction unit to obtain the candidate feature representation corresponding to each candidate mask includes: Each candidate mask corresponds to a candidate instance. For any candidate mask: the spatial size of the candidate mask is resampled to make the size of the candidate mask the same as that of the dense feature map corresponding to the target image; the resampled candidate mask is multiplied by the dense feature map corresponding to the target image to obtain a subset of candidate features corresponding to the candidate instance; the subset of candidate features corresponding to the candidate instance is normalized to obtain the candidate feature representation corresponding to the candidate mask.
6. The reference instance segmentation reasoning method as described in claim 5, characterized in that, The step of obtaining the initial prediction result for each candidate mask based on the candidate feature representation corresponding to each candidate mask and all category prototypes in the memory database includes: The initial prediction result for each candidate mask includes the category label corresponding to the candidate mask and the initial confidence score corresponding to the candidate mask; For any candidate mask: calculate the similarity between the candidate feature representation corresponding to the candidate mask and all category prototypes in the memory library, and take the maximum similarity as the initial confidence score corresponding to the candidate mask; take the category to which the category prototype corresponding to the maximum similarity belongs as the category label corresponding to the candidate mask.
7. The reference instance segmentation reasoning method as described in claim 6, characterized in that, The step involves inputting all candidate masks and their corresponding candidate feature representations, along with the initial prediction results, into a semantic-aware fusion unit. This unit optimizes the initial prediction results for each candidate mask using a soft suppression mechanism, resulting in an optimized prediction result for each candidate mask. This includes: The candidate masks are classified according to the category labels in the initial prediction results corresponding to all candidate masks, and the candidate masks corresponding to each category are sorted according to the initial confidence score. The initial prediction results corresponding to each candidate mask in the category are optimized based on the sorting results. Specifically, for any candidate mask m in the category: a set of candidate masks is generated from the candidate mask m and other candidate masks in the category; the spatial overlap of the candidate mask set is obtained based on the coverage area of the candidate mask m and the coverage area of another candidate mask in each set; the feature similarity of the candidate mask set is obtained based on the candidate feature representation of the candidate mask m and the candidate feature representation of another candidate mask in the set; the suppression factor of the candidate mask set is obtained based on the spatial overlap and feature similarity; the attenuation factor of the candidate mask m is obtained based on all the suppression factors; the initial confidence score of the candidate mask m is optimized based on the attenuation factor, resulting in the optimized confidence score of the candidate mask m; and the prediction result of the candidate mask m after optimization is obtained based on the category label of the candidate mask m and the optimized confidence score.
8. The reference instance segmentation reasoning method as described in claim 7, characterized in that, The step of obtaining the spatial overlap of a group of candidate masks based on the coverage area corresponding to candidate mask m and the coverage area corresponding to another candidate mask in each group of candidate masks includes: For this set of candidate masks: obtain the intersection area between the coverage area corresponding to candidate mask m and the coverage area corresponding to another candidate mask; obtain the spatial overlap of this set of candidate masks based on the area of the intersection area and the area of the coverage area corresponding to candidate mask m.
9. The reference instance segmentation reasoning method as described in claim 7, characterized in that, The step of obtaining the feature similarity of the candidate mask group based on the candidate feature representation corresponding to candidate mask m and the candidate feature representation corresponding to another candidate mask includes: For this set of candidate masks: normalize the candidate feature representations corresponding to the candidate mask m and the candidate feature representations corresponding to the other candidate mask respectively; calculate the similarity between the normalized candidate feature representations corresponding to the candidate mask m and the normalized candidate feature representations corresponding to the other candidate mask to obtain the feature similarity of this set of candidate masks.
10. The reference instance segmentation reasoning method as described in claim 7, characterized in that, The step of obtaining the attenuation factor corresponding to candidate mask m based on all suppression factors corresponding to candidate mask m includes: Obtain the largest suppression factor among all suppression factors corresponding to the candidate mask m, calculate the difference between the value 1.0 and the obtained largest suppression factor, and take the square root of the obtained difference as the attenuation factor corresponding to the candidate mask m.
11. The reference instance segmentation reasoning method as described in claim 7, characterized in that, The step of filtering the optimized prediction results for each candidate mask to obtain the segmentation result corresponding to the target image includes: The candidate masks are categorized according to their category labels in the optimized prediction results. All candidate masks for each category are sorted according to their optimized confidence scores, and the candidate masks are retained based on the actual number of masks retained. The actual number of retained instances is the minimum of the number of candidate instances for each category and the preset number.
12. A method for constructing a training-free reference instance segmentation memory library, characterized in that, include: Obtain a reference image set, which contains multiple reference images, wherein each reference image corresponds to one or more instance masks; For any reference image: perform feature extraction on the reference image to obtain a dense feature map of the reference image; based on the dense feature map of the reference image and its corresponding instance mask, obtain the feature subset corresponding to each instance in the reference image; The feature subsets corresponding to all instances are stored according to category units, and all category units constitute a memory library, which is used in the reference instance segmentation inference method.
13. The reference instance segmentation memory library construction method as described in claim 12, characterized in that, Also includes: The feature aggregation unit obtains the category prototype corresponding to each category; wherein, the feature subset corresponding to each instance in each category unit of the memory library is input into the feature aggregation unit, and the category prototype corresponding to each category is obtained according to the feature subset corresponding to each instance in each category unit.
14. The reference instance segmentation memory library construction method as described in claim 13, characterized in that, The step of obtaining the category prototype corresponding to each category based on the feature subset corresponding to each instance in each category unit includes: For any category in the memory library: based on the feature subset corresponding to each instance in the category unit corresponding to the category, obtain the instance-level prototype corresponding to the instance; based on all instance-level prototypes corresponding to the category, obtain the category prototype corresponding to the category.
15. The reference instance segmentation memory library construction method as described in any one of claims 12-14, characterized in that, The memory library is applied to the reference instance segmentation reasoning method as described in any one of claims 1-11.
16. A computer-readable storage medium, characterized in that, The medium stores a computer program that can be executed by a processor to implement the method as described in any one of claims 1-15.