Cascade detection segmentation combined with zero-shot instance segmentation method based on dual feature matching

Abstract

This invention relates to the field of computer vision and discloses a zero-shot instance segmentation method combining cascaded detection and segmentation with two-dimensional feature matching. The method includes: offline construction of a two-dimensional feature template library for the target object; online execution of three-level cascaded inference on the scene image: first, open-vocabulary detection is performed to obtain candidate boxes; then, a general segmentation model is used to obtain a clean foreground mask; finally, the two-dimensional features of the candidate regions are extracted and mixed with the template library for hybrid matching and multi-view consistency verification to identify and segment the target instance. This invention, under strict zero-shot conditions without the need for CAD models and model fine-tuning, solves the contradiction between efficiency and accuracy through a cascaded architecture, possessing the advantages of high accuracy, high robustness, and plug-and-play functionality.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and more specifically, to an instance segmentation method, particularly to a method that can achieve accurate instance segmentation based solely on a few two-dimensional reference images under zero-sample conditions, without requiring a three-dimensional CAD model of the target object or model fine-tuning. Background Technology

[0002] Instance segmentation, a core task of advanced visual perception, aims to simultaneously locate and extract the contours of objects in an image. It is a key technology for applications such as robot grasping, augmented reality, autonomous driving, and industrial quality inspection. Traditional deep learning methods (such as Mask R-CNN) rely on large-scale labeled data of specific categories for training, belonging to a "closed set" recognition paradigm. This makes it difficult to generalize to unseen objects and cannot adapt to the needs of dynamic and open environments.

[0003] For the segmentation of new objects, existing technologies mainly have three limitations: First, methods based on matching 3D CAD models with templates require the prior acquisition of accurate 3D models of the objects, resulting in poor universality, low computational efficiency, and domain differences between virtual rendering and real scenes; Second, methods based on fine-tuning with a small number of samples (such as fine-tuning SAM) have some adaptability, but still require labeled data and suffer from catastrophic forgetting and cumbersome model management; Third, open-vocabulary methods based on language description (such as using CLIP) struggle to accurately describe the unique details of objects through text, and the segmentation results are often coarse.

[0004] While the development of fundamental visual models (such as Grounding DINO, SAM, and DINOv2) has provided powerful detection, segmentation, and feature extraction capabilities, systematically integrating these models to build an efficient zero-shot instance segmentation system remains a challenge. Simple concatenation schemes inherently present a contradiction: to ensure recall, the candidate boxes output by general detectors contain a large amount of background noise, and directly performing feature matching in this region severely degrades accuracy; while using full-image dense matching incurs enormous computational overhead and is inefficient. Therefore, "efficiency" and "accuracy" are difficult to balance in zero-shot instance segmentation. This invention aims to fundamentally resolve this contradiction. Summary of the Invention

[0005] This invention aims to overcome the aforementioned shortcomings by proposing a zero-shot instance segmentation method that combines cascaded detection and segmentation with two-dimensional feature matching. Its primary objective is to achieve accurate segmentation of new object instances under strict zero-shot constraints, without requiring a CAD model of the target object or any model fine-tuning. A deeper objective is to collaboratively resolve the inherent contradiction between "search efficiency" and "matching accuracy" through an innovative cascaded architecture and matching mechanism, enabling the system to simultaneously possess the ability to process efficiently and output with high precision. Furthermore, this invention also strives to maintain high robustness and reliability when facing real-world challenges such as changes in viewpoint, partial occlusion, and complex backgrounds.

[0006] To achieve this objective, the technical solution of the present invention is: a zero-shot instance segmentation method combining cascaded detection segmentation and two-dimensional feature matching, comprising the following steps:

[0007] S1. Offline construction of feature template library: Obtain multiple reference images of the target object; extract the global semantic feature vector and local appearance feature map of each reference image; store all extracted feature pairs to form an offline two-dimensional feature template library;

[0008] S2. Perform online three-level cascaded reasoning on the input scene image, including:

[0009] S21. First-level detection: Input the scene image into the open vocabulary object detection model to obtain a set of bounding boxes containing multiple potential object regions;

[0010] S22. Second-level segmentation: Input each bounding box obtained in the first level into the general instance segmentation model to obtain the foreground mask corresponding to each candidate object; extract the corresponding foreground region image from the scene image based on each foreground mask;

[0011] S23. Third-level matching and decision-making: For each candidate object, perform the following operations:

[0012] S231. Feature extraction: Extracting online global semantic feature vectors and online local appearance feature maps from the foreground region image;

[0013] S232. Hybrid matching: The online global semantic feature vector and online local appearance feature map extracted in step S231 are matched and calculated with the corresponding global semantic feature vector and local appearance feature map in the offline dual-dimensional feature template library to obtain the template-level comprehensive matching score of the candidate object relative to each reference template.

[0014] S233, Object-level determination: Based on the comprehensive matching scores of multiple templates corresponding to each candidate object, perform multi-view consistency verification; determine the candidate objects that pass the verification as valid candidates, and calculate their final matching scores;

[0015] S234. Target decision: Select the candidate object with the highest final matching score from all valid candidates as the target instance; if its final matching score exceeds the preset threshold, output the foreground mask corresponding to the target instance as the instance segmentation result.

[0016] Furthermore, in step S1, the extraction of the global semantic feature vector specifically involves: processing the reference image using a self-supervised visual feature model based on VisionTransformer, and extracting its output CLS marker features as the global semantic feature vector.

[0017] Furthermore, in step S1, the extraction of local appearance feature maps specifically involves: processing the reference image using a self-supervised visual feature model based on Vision Transformer, extracting all patch markers output by the last Transformer Block, and reshaping them into feature maps as local appearance feature maps.

[0018] Furthermore, in step S232, the mixed matching specifically includes:

[0019] Calculate the global semantic similarity between the online global semantic feature vector and the global semantic feature vector of the reference template;

[0020] Calculate the local appearance similarity between the online local appearance feature map and the local appearance feature map of the reference template. The local appearance similarity is obtained by calculating the similarity of the feature vectors at corresponding positions in space and then performing a global maximum pooling operation.

[0021] The global semantic similarity and the local appearance similarity are adaptively weighted and fused to obtain the template-level comprehensive matching score.

[0022] Furthermore, in the adaptive weighted fusion, the weights of global semantic similarity and local appearance similarity are dynamically calculated based on the cosine distance between the global semantic feature vector of the corresponding reference template and a general semantic vector, as well as the variance of its local appearance feature map.

[0023] Furthermore, in step S233, the multi-view consistency check specifically involves:

[0024] Set a first threshold;

[0025] The number of candidate objects whose template-level comprehensive matching scores exceed the first threshold is counted.

[0026] If the number exceeds the proportional threshold set based on the total number of reference templates, the candidate object is determined to meet the multi-view consistency requirement; otherwise, it is determined to be an invalid candidate and is removed.

[0027] Furthermore, in step S233, the final matching score of the candidate object that has passed the verification is set to the maximum value among all its template-level comprehensive matching scores.

[0028] Furthermore, in step S2, the open vocabulary target detection model is GroundingDINO, the general instance segmentation model is Segment Anything Model, and the model used for feature extraction in steps S1 and S231 is DINOv2.

[0029] Furthermore, in step S22, extracting the foreground region image based on the foreground mask specifically involves multiplying the foreground mask by the scene image element-wise to obtain a foreground region image with suppressed background.

[0030] Furthermore, in step S234, the preset threshold is 0.6.

[0031] Compared with the closest existing technology, the beneficial effects of this invention are significant and multifaceted:

[0032] 1. Achieving a synergistic breakthrough in efficiency and accuracy under zero-sample conditions: This invention systematically resolves the inherent contradiction between "efficiency" and "accuracy" in this field through an innovative three-level cascaded architecture. The first-level detection achieves an order-of-magnitude reduction in computational complexity; the second-level segmentation achieves a substantial improvement in feature quality (background noise suppression); and the third-level fine-grained matching on clean features ensures highly reliable recognition results. Experiments show that while maintaining high accuracy, the processing speed can be improved by more than 50% compared to traditional dense matching methods.

[0033] 2. Adaptive Weighted Fusion Enhances Scene Adaptability: In the adaptive weighted fusion, the weights of global semantic similarity and local appearance similarity are dynamically calculated based on the cosine distance between the global semantic feature vector of the corresponding reference template and a general semantic vector, as well as the variance of its local appearance feature map. This design enables the system to automatically adapt to object characteristics (such as texture richness or shape uniqueness) without requiring manual parameter tuning by the user. For example, the IoU of segmentation for texture-rich objects can be improved by 10%-12%, and the recognition accuracy for objects with unique shapes can be improved by 8%-10%.

[0034] 3. A dual-dimensional hybrid matching mechanism enhances discriminative ability and robustness: The hybrid matching integrates global semantic features and local appearance features. Global features grasp the object category and overall structure; local features, through a global max-pooling strategy, focus on the most discriminative local details and are insensitive to occlusion and deformation. The two complement each other, enabling the matching process to grasp both the big picture and the finer details, significantly improving discriminative ability in complex scenes and with interference from similar objects.

[0035] 4. Multi-view consistency verification effectively reduces the risk of false matching: The multi-view consistency verification requires candidate objects to achieve a high degree of similarity in more than a certain proportion (e.g., 1 / 3) of the reference viewpoints in order to be recognized as valid candidates. This strategy effectively eliminates false matches caused by accidental similarity from a single viewpoint, reducing the false matching rate by more than 40% in scenarios with similar interference objects, while not affecting the system's adaptability to changes in the target object's viewpoint, thus improving the reliability of high-precision application scenarios.

[0036] 5. Possesses true "plug-and-play" deployment flexibility and strong generalization: All core modules in the entire system process (detector, segmenter, feature extractor) are pre-trained general-purpose base models, requiring no annotation, training, or fine-tuning for specific targets. Users only need to provide a few reference photos of the target object, and the system can complete the adaptation and be put into real-time use within seconds, which has significant advantages in scenarios where the target object needs to be frequently changed. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0038] Figure 1 This is the system architecture and flowchart of the overall method of the present invention.

[0039] Figure 2 A detailed diagram illustrating the process of building an offline two-dimensional feature template library.

[0040] Figure 3 This is a schematic diagram of the input scene and the output of the detection box during the online inference stage.

[0041] Figure 4 This is a schematic diagram of the SAM segmentation results during the online inference phase.

[0042] Figure 5 This is a detailed data flow diagram of the third-level two-dimensional hybrid matching calculation process. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] This invention provides a zero-shot instance segmentation method combining cascaded detection and segmentation with two-dimensional feature matching, the overall process of which is as follows: Figure 1 As shown, it includes the following steps:

[0045] S1. Offline construction of feature template library: Obtain N multi-angle reference images of the target object, denoted as set. The value of N is typically 4 to 12 to cover its main appearance perspective. These images can come from real-world photography or 3D model rendering (such as using the Pyrender engine). For reference image; extract the first... Reference image global semantic feature vector and local appearance feature map ,in, For feature dimension, and The spatial height and width of the feature map. This represents the number of feature channels; all extracted feature pairs Store the data to form an offline two-dimensional feature template library. ;

[0046] S2, For the input scene image Perform online three-level cascaded reasoning, including:

[0047] S21, First-level detection: The scene image... The input is fed into an open vocabulary object detection model (Grounding DINO is used in this embodiment of the invention) to obtain a set of bounding boxes containing multiple potential object regions. ;

[0048] S22, Second-level segmentation: Segment each bounding box obtained in the first level... The data is input into a general instance segmentation model to obtain the foreground mask corresponding to each candidate object; the corresponding foreground region image is extracted from the scene image based on each foreground mask.

[0049] S23. Third-level matching and decision-making: For each candidate object, perform the following operations:

[0050] S231. Feature Extraction: For each candidate object, based on its clean mask... From the original image Crop out the area The region is forward-propagated using the exact same DINOv2 model (weight sharing) as in the offline phase to extract its online global semantic feature vector. and online local appearance feature map ;

[0051] S232. Hybrid Matching: The online global semantic feature vector and online local appearance feature map extracted in step S231 are matched with the corresponding global semantic feature vector and local appearance feature map in the offline dual-dimensional feature template library to obtain the template-level comprehensive matching score of the candidate object relative to each reference template, specifically including:

[0052] a) Template-level two-dimensional similarity calculation: For the first... The candidate object and the first Zhang reference image:

[0053] Global semantic similarity Calculate the cosine similarity between two semantic feature vectors. :

[0054] ;

[0055] in Represents the dot product of vectors; Represents the L2 norm; Indicates from the first Global semantic feature vector extracted from the reference image; The closer the value is to 1, the higher the semantic similarity.

[0056] Local appearance similarity This calculation is more precise. First, the online feature map is calculated. Compared with local appearance feature map The cosine similarity between each pair of feature vectors at corresponding spatial locations yields a similarity matrix. Then, a global max pooling operation is performed on the matrix:

[0057] ;

[0058] in, This represents the global maximum pooling operation. Candidate objects The local feature vector at position (h,w) in the local appearance feature map.

[0059] This operation focuses on the most similar local feature pair between two objects, making it extremely robust to non-rigid deformation, local occlusion, and misalignment. .

[0060] b) Adaptive similarity fusion: The similarity of the two complementary dimensions mentioned above is linearly weighted and fused to obtain candidate objects. Compared to the reference template Template-level composite matching score :

[0061] ;

[0062] in, The adaptive weights for global semantic similarity. An adaptive weight for maximizing local appearance similarity. This is used to balance the contribution ratio of global semantic information and local detailed information in the final decision. The weights are dynamically calculated using an offline benchmark, requiring no manual parameter tuning.

[0063] ;

[0064] in, For the first The semantic uniqueness of each reference template For the first The texture richness of each template is automatically adapted to the characteristics and attributes of the object.

[0065] S233, Object-level Judgment: Based on the comprehensive matching scores of multiple templates corresponding to each candidate object, a multi-view consistency check is performed; candidate objects that pass the check are determined as valid candidates, and their final matching scores are calculated, specifically including:

[0066] A candidate object will with Comparison with reference templates, generating Template-level scores The aggregation strategy employs "maximum value + multi-view consistency check," and the specific steps are as follows:

[0067] Step 1: Set the similarity threshold The score of the candidate object is greater than or equal to the score of all reference templates. Number of templates ,Right now ( (This is an indicator function; it returns 1 if the condition is met, and 0 otherwise).

[0068] Step 2: Consistency Check: If... ( If the function is rounded up (ensuring coverage of at least 1 / 3 of the reference viewpoint), then the candidate object is determined to satisfy multi-view consistency, and the maximum value among the N scores is taken as its final matching score. :

[0069] ;

[0070] in, For the first The candidate object is relative to the first Template-level comprehensive matching score for the reference view.

[0071] Step 3: Invalid removal: If If the candidate object is determined to be only a random similarity from a single perspective (false match), it will be directly eliminated and will not participate in subsequent decisions.

[0072] This strategy retains robustness to changes in viewpoint while fundamentally eliminating mismatches from a single viewpoint through multi-viewpoint consistency verification, thus solving the problem of recognition reliability under interference from similar objects of the same type.

[0073] S234. Target Decision: From all valid candidates, select the candidate object with the highest final matching score as the target instance:

[0074] ;

[0075] If the highest score ( As a matching threshold, in the embodiments of the present invention If the default value is 0.6, then the high-quality mask generated by SAM corresponding to the target object will be output. As the result of instance segmentation; otherwise, output "No target object detected".

[0076] In step S1, the extraction of the global semantic feature vector specifically involves processing the reference image using a self-supervised visual feature model based on Vision Transformer, and extracting the output CLS marker features as the global semantic feature vector.

[0077] The extraction of local appearance feature maps is specifically carried out by processing the reference image using a self-supervised visual feature model based on Vision Transformer, extracting all patch markers from the output of the last Transformer Block, and reshaping them into feature maps as local appearance feature maps.

[0078] In step S232, the mixed matching specifically includes:

[0079] Calculate the global semantic similarity between the online global semantic feature vector and the global semantic feature vector of the reference template;

[0080] Calculate the local appearance similarity between the online local appearance feature map and the local appearance feature map of the reference template. The local appearance similarity is obtained by calculating the similarity of the feature vectors at corresponding positions in space and then performing a global maximum pooling operation.

[0081] The global semantic similarity and the local appearance similarity are adaptively weighted and fused to obtain the template-level comprehensive matching score.

[0082] In the adaptive weighted fusion, the weights of global semantic similarity and local appearance similarity are dynamically calculated based on the cosine distance between the global semantic feature vector of the corresponding reference template and a general semantic vector, as well as the variance of its local appearance feature map.

[0083] In step S233, the multi-view consistency check specifically involves:

[0084] Set a first threshold;

[0085] The number of candidate objects whose template-level comprehensive matching scores exceed the first threshold is counted.

[0086] If the number exceeds the proportional threshold set based on the total number of reference templates, the candidate object is determined to meet the multi-view consistency requirement; otherwise, it is determined to be an invalid candidate and is removed.

[0087] In step S233, the final matching score of the candidate object that has passed the verification is set to the maximum value among all its template-level comprehensive matching scores.

[0088] In step S2, the open vocabulary target detection model is GroundingDINO, the general instance segmentation model is Segment Anything Model, and the feature extraction model used in steps S1 and S231 is DINOv2.

[0089] In step S22, extracting the foreground region image based on the foreground mask specifically involves multiplying the foreground mask by the scene image element-wise to obtain a foreground region image with suppressed background.

[0090] In step S234, the preset threshold is 0.6.

[0091] The following detailed description of the zero-sample instance segmentation method combining cascaded detection and segmentation with two-dimensional feature matching provided by the present invention will be given with specific embodiments.

[0092] Example 1: Zero-sample grasping system for specific parts based on industrial cameras

[0093] Scenario: On a flexible automated production line, a robot needs to pick up a new type of connector part from a bin. There is no existing CAD model for this part, and the production line may frequently change the type of part to be picked up. The vision system is required to quickly learn new parts and accurately segment the target instance in real-time from the cluttered bin to guide the robotic arm to pick it up.

[0094] Hardware: Hikvision 20-megapixel RGB camera fixed above the material bin, industrial computer equipped with NVIDIA RTX 4090, and six-axis robotic arm.

[0095] Software implementation:

[0096] Step A. Offline learning:

[0097] 1. Workers placed a single new connector on a white background turntable and used an industrial computer to control an industrial camera to automatically acquire eight images at a uniform surrounding angle. This constructed a reference set covering the main viewpoints at a relatively low acquisition cost, providing a foundation for multi-view consistency verification.

[0098] 2. Call the DINOv2 model loaded on the GPU (specifically, dinov2_vitl14 version). For each image... The model is input with a reference image of a certain size. Leveraging the powerful generalization ability of a self-supervised pre-trained model, zero-shot learning is achieved, avoiding training for specific target objects.

[0099] 3. Feature extraction (see details) Figure 2 (as shown)

[0100] 3.1. Obtain the CLS markers from the output of the last layer of the model to obtain the global semantic feature vector. , dimension It obtains a highly abstract semantic feature that represents "what" the object as a whole is.

[0101] 3.2. Obtain all patch tokens from the output of the last layer of the model and reshape them into feature maps. , dimension (Because ViT-L / 14 divides the image into 14×14 patches). This preserves the spatial structure and local details of the image, representing "what an object looks like".

[0102] 4. Calculate the baseline weights for each template:

[0103] 4.1. Calculate the variance of the partial appearance feature map for each reference test view. ,For example ;

[0104] 4.2. Load the pre-stored generic "part" semantic vectors Calculate the cosine distance between the global semantic feature vector of each reference image and the reference image itself. ,For example This provides an objective basis for dynamically calculating weights in the subsequent online matching stage, enabling the system to adapt to objects with different characteristics and improve segmentation accuracy.

[0105] 5. Take these 8 groups After serialization, the data is saved to the hard drive and used as a feature template library for this connector model. This completes the creation of a one-time "visual ID card," which can be quickly accessed for subsequent online inference.

[0106] Step B. Online reasoning and crawling (each crawling loop):

[0107] 1. Image Acquisition: An industrial camera captures a scene image of a material bin containing multiple messy parts. (Resolution 1280×1024).

[0108] 2. Level 1: Open Vocabulary Test

[0109] After downsampling the image to 800×640, input it into the Grounding DINO model along with the text prompt "object";

[0110] The model outputs 15 bounding boxes, filtering out boxes with a confidence level below 0.25, ultimately retaining approximately 10 candidate boxes covering different parts such as connectors, screws, and gaskets. A schematic diagram of the output bounding box effect is shown below. Figure 3 As shown, the problem of searching the entire image at the pixel level is transformed into the problem of identifying only about 10 candidate regions, achieving a significant reduction in computational complexity for the first time, which is the first cornerstone of the system's efficient operation.

[0111] 3. Second stage: Instance segmentation and purification:

[0112] Use the coordinates of each candidate box (which need to be scaled back to the original image size) as input to call the SAM model;

[0113] SAM generates a high-precision mask for each object within the bounding box. The foreground mask (with the background stripped away) after SAM processing can be seen in [reference needed]. Figure 4 As shown, with Figure 3 By comparing the original detection frame with the original frame, it is clear that background noise has been effectively suppressed. Key observation: For the target connector, the mask is accurate down to the toothed edges and the center hole, and completely removes the adjacent screw images and the material box mesh background;

[0114] Record all candidate masks .

[0115] This step acts like a sophisticated "visual filter," "stripping" the foreground object from the detection box, which contains about 30%-50% background, resulting in a clean foreground region image with greatly suppressed background noise. This provides a clean visual signal carrier for subsequent high-precision matching.

[0116] 4. Level 3: Feature Matching and Decision Making

[0117] 4.1. Feature Extraction: For each mask In the original image A mask is applied to the foreground area, which is then cropped by expanding the bounding box by 5 pixels. After scaling to 224×224, the foreground area is fed into DINOv2 to obtain online features. .

[0118] 4.2. Matching Calculation (see...) Figure 5 (as shown)

[0119] a) Load the 8 reference templates in the template.

[0120] b) For the j-th candidate (assuming it is the target connector):

[0121] calculate With 8 The cosine similarity is obtained. ;

[0122] calculate With 8 Appearance similarity: for each Calculate the 14×14 similarity matrix and take the maximum value, then... The global maximum pooling strategy focuses only on the most similar local feature pair between two objects, thus exhibiting strong robustness to non-rigid deformation, local occlusion, and misalignment.

[0123] Calculate adaptive weights: Taking the first reference template as an example, Eight sets of weights are calculated sequentially; the fusion weights are dynamically adjusted based on the characteristics of the reference template itself (semantic uniqueness and texture richness), which realizes automatic adaptation to objects with different attributes and improves the overall segmentation accuracy and robustness of the system.

[0124] Calculate the 8 template-level scores: , to obtain, for example ;

[0125] Multi-view consistency check execution: setting threshold Count the number of scores ≥ 0.65 among the 8 scores. ;determination To ensure consistency across multiple perspectives, the maximum value is taken as the object-level score. .

[0126] c) For candidates with similar part models:

[0127] Repeat the above calculations to count the number of scores ≥0.65 in the reference template. (Only one perspective is coincidentally similar);

[0128] Multi-perspective consistency check execution: If it is determined to be a false match, it will be directly removed.

[0129] d) For other candidates such as screws and washers: if the matching score is less than 0.5, they are directly eliminated.

[0130] This strategy fundamentally eliminates interference objects that are only similar to the target object from a few angles (such as similar parts of different models), significantly reducing the false match rate.

[0131] 4.3. Decision-making:

[0132] The system found the highest score Corresponding candidate index Mask the candidate Convert to a binary image and calculate its minimum bounding rectangle and centroid pixel coordinates.

[0133] Output and Execution: The image coordinates of the target connector are converted into a 3D position in the robot's base coordinate system using a hand-eye calibration matrix, along with its contour direction, and sent to the robot controller. The robot then plans its grasping path and completes precise grasping.

[0134] Performance and Effect

[0135] Accuracy: In multiple test scenarios involving different placement postures and occlusion conditions, the system demonstrated high segmentation accuracy and target recognition reliability, and was able to accurately capture the contour details of the target object.

[0136] Efficiency: The end-to-end processing flow responds quickly and can meet the cycle time requirements of various production lines. The additional overhead brought by adaptive weight fusion and multi-view consistency verification is extremely low and will not have a significant impact on the overall processing efficiency.

[0137] Flexibility: When the production line needs to replace new parts, there is no need to modify the code or retrain the model. It only needs to complete an offline learning step in a short time to quickly adapt to the new target and put it into production.

[0138] Robustness: The adaptive weight fusion design enables the system to adapt well to parts with rich textures and unique shapes, and can still maintain stable segmentation performance in scenes with a certain degree of occlusion; the multi-view consistency verification effectively reduces the risk of mismatch caused by similar interference objects, and fully verifies the effectiveness of the technical solution of the present invention.

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A zero-shot instance segmentation method of cascade detection segmentation combined with double feature matching, characterized in that, Includes the following steps: S1. Offline construction of feature template library: Obtain multiple reference images of the target object; extract global semantic feature vectors and local appearance feature maps of each reference image; All extracted feature pairs are stored to form an offline two-dimensional feature template library; S2. Perform online three-level cascaded reasoning on the input scene image, including: S21. First-level detection: Input the scene image into the open vocabulary object detection model to obtain a set of bounding boxes containing multiple potential object regions; S22. Second-level segmentation: Input each bounding box obtained in the first level into the general instance segmentation model to obtain the foreground mask corresponding to each candidate object; extract the corresponding foreground region image from the scene image based on each foreground mask; S23. Third-level matching and decision-making: For each candidate object, perform the following operations: S231. Feature extraction: Extracting online global semantic feature vectors and online local appearance feature maps from the foreground region image; S232. Hybrid Matching: The online global semantic feature vector and online local appearance feature map extracted in step S231 are matched with the corresponding global semantic feature vector and local appearance feature map in the offline dual-dimensional feature template library to obtain the template-level comprehensive matching score of the candidate object relative to each reference template, specifically including: Calculate the global semantic similarity between the online global semantic feature vector and the global semantic feature vector of the reference template; Calculate the local appearance similarity between the online local appearance feature map and the local appearance feature map of the reference template. The local appearance similarity is obtained by calculating the similarity of the feature vectors at corresponding positions in space and then performing a global maximum pooling operation. The global semantic similarity and the local appearance similarity are adaptively weighted and fused to obtain the template-level comprehensive matching score. In the adaptive weighted fusion, the weights of the global semantic similarity and the local appearance similarity are dynamically calculated based on the cosine distance between the global semantic feature vector of the corresponding reference template and a general semantic vector, as well as the variance of its local appearance feature map. S233, Object-level Judgment: Based on the comprehensive matching scores of multiple templates corresponding to each candidate object, a multi-view consistency check is performed; candidate objects that pass the check are determined as valid candidates, and their final matching scores are calculated. The multi-view consistency check is specifically as follows: Set a first threshold; The number of candidate objects whose template-level comprehensive matching scores exceed the first threshold is counted. If the number exceeds the proportional threshold set based on the total number of reference templates, the candidate object is determined to satisfy multi-view consistency; otherwise, it is determined to be an invalid candidate and is removed. S234. Target decision: Select the candidate object with the highest final matching score from all valid candidates as the target instance; if its final matching score exceeds the preset threshold, output the foreground mask corresponding to the target instance as the instance segmentation result.

2. The zero-shot instance segmentation method of claim 1, wherein, In step S1, the extraction of the global semantic feature vector specifically involves: processing the reference image using a self-supervised visual feature model based on Vision Transformer, and extracting the output CLS marker features as the global semantic feature vector.

3. The zero-shot instance segmentation method of claim 1 or 2, wherein, In step S1, the extraction of local appearance feature map is specifically as follows: the reference image is processed using a self-supervised visual feature model based on Vision Transformer, all patch markers output by the last Transformer Block are extracted, and reshaped into a feature map as a local appearance feature map.

4. The zero-shot instance segmentation method of claim 1, wherein, In step S233, the final matching score of the candidate object that has passed the verification is set to the maximum value among all its template-level comprehensive matching scores.

5. The zero-shot instance segmentation method of claim 1, wherein, In step S2, the open vocabulary target detection model is GroundingDINO, the general instance segmentation model is Segment Anything Model, and the feature extraction model used in steps S1 and S231 is DINOv2.

6. The zero-shot instance segmentation method combining cascaded detection and segmentation with two-dimensional feature matching according to claim 1, characterized in that, In step S22, extracting the foreground region image based on the foreground mask specifically involves multiplying the foreground mask by the scene image element-wise to obtain a foreground region image with suppressed background.

7. The zero-shot instance segmentation method combining cascaded detection and segmentation with two-dimensional feature matching according to claim 1, characterized in that, In step S234, the preset threshold is 0.6.

Images