A method and system for zero-shot image segmentation based on object three-dimensional models

Abstract

The application relates to the field of image segmentation technology in computer vision, in particular to a zero-shot image segmentation method and system based on a three-dimensional model of an object. The method comprises the following steps: for any untrained object, a three-dimensional model of the object is rendered into a two-dimensional RGB reference image under a plurality of preset discrete viewing angles by using a graphics rendering engine, the reference image is input into a visual basic model DINOv3, a global feature vector representing semantic information of the object is extracted, and a reference feature library is constructed; a target image to be segmented is input into a visual basic model SAM2, and a plurality of candidate object masks are generated; for each candidate mask, an object image corresponding to the candidate mask is cropped from the target image, and the object image is input into the DINOv3 to extract a semantic feature vector of the candidate object; by calculating the cosine similarity between the candidate object feature vector and each template feature vector in the reference feature library, the top similar degrees with the highest values are selected, and an arithmetic mean value of the similar degrees is calculated, and the mean value is taken as the classification confidence of the candidate object mask; the class label of each candidate object mask is determined according to the confidence, and the class label of the target object and a corresponding pixel-level segmentation mask are output.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology in computer vision, and more specifically, to a zero-shot image segmentation method and system based on a three-dimensional object model and a large visual fundamental model (SAM2+DINOv3). Background Technology

[0002] In applications such as industrial automation, robotic grasping, and augmented reality, accurate object detection and segmentation are crucial prerequisites for subsequent downstream tasks and operations. Most image segmentation methods, such as Mask R-CNN and Mask2Former, rely primarily on supervised deep learning techniques, which typically require training on large-scale labeled datasets for specific target objects.

[0003] However, existing supervised learning segmentation methods face significant challenges in practical applications. First, when introducing new, untrained objects, these methods typically require re-collecting data, manually labeling it, and retraining the model—a time-consuming and labor-intensive process that struggles to meet the demands of industrial scenarios with frequent changes in product types. Second, the generalization ability of supervised methods is limited by the distribution of training data, making it difficult to directly transfer to unseen object categories.

[0004] To address these issues, researchers have proposed zero-shot segmentation methods for unknown objects. Early methods primarily relied on depth information acquired by RGB-D cameras for clustering and segmentation, limiting their application in scenarios with only a monocular RGB camera. Other zero-shot methods based on object CAD models, such as MegaPose, while utilizing the object's 3D information, depend on known 2D bounding boxes as input, making them unable to independently and automatically discover and segment objects from the entire image. Furthermore, their feature matching relies solely on single-view similarity, resulting in poor robustness.

[0005] In recent years, the development of large-scale vision foundation models, such as the Segment Everything Model (SAM) and the Self-Supervised Visual Feature Model (DINO), has provided new possibilities for zero-shot image understanding. Some existing methods attempt to combine these large-scale models to achieve segmentation by matching features from rendered CAD models with real images. However, these methods still have room for improvement in terms of discriminative power of feature extraction, fineness of segmentation boundaries, and temporal stability in processing video stream data. Furthermore, the domain gap between synthetic rendered images and real images remains one of the main factors limiting matching accuracy.

[0006] Therefore, how to fully utilize the latest large-scale visual model technologies (such as SAM2 and DINOv3), combined with uniform multi-view sampling and robust feature matching strategies, to construct a high-precision, robust, automated segmentation method in monocular RGB images that requires no manual sample annotation and only uses 3D CAD models of objects is a pressing technical problem in the field of computer vision. This invention is proposed to address at least one of the aforementioned problems in the existing technology. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a zero-shot image segmentation method and system based on a three-dimensional object model, which eliminates the dependence on manually labeled data, improves the segmentation robustness in complex scenes, and achieves high-precision and high-efficiency zero-shot segmentation of unknown objects in monocular RGB images, at least partially solving the problems existing in the prior art.

[0008] In a first aspect, embodiments of the present invention provide a method for zero-shot image segmentation based on a three-dimensional model of an object, comprising the following steps: Step 1: Construct an object reference feature library: Obtain the 3D CAD model of the object to be detected, and use a graphics rendering engine to render it from multiple preset discrete viewpoints. The three-dimensional CAD model is then rendered into a two-dimensional RGB reference template image. The reference template image is then input into the pre-trained visual basic model DINOv3 to extract global feature vectors representing the semantic information of the object, and a reference feature library containing multi-view information is constructed.

[0009] Step 2: Generate candidate object masks for the entire image: Input the RGB image of the target to be segmented into the pre-trained visual base model SAM2. Utilize its automatic mask generation function to generate several class-independent candidate object masks across the entire image without manual prompting.

[0010] Step 3: Extract semantic feature vectors of candidate objects: For each candidate object mask, use the mask to remove the background from the target RGB image, and crop the object region image according to the mask bounding box; perform scaling and filling operations on the object region image while maintaining the aspect ratio to unify it to a fixed resolution; input the processed image into the DINOv3 model to extract the visual descriptors of the candidate objects.

[0011] Step 4: Calculate classification confidence and output segmentation results: Calculate the cosine similarity between the visual descriptor of the candidate object and the global feature vector of each template in the reference feature library; for each object class, select the top [segment name missing] with the highest score. The similarity is calculated and its arithmetic mean is used as the classification confidence score; the category label of each candidate object mask is determined according to the classification confidence score, and the category label of the target object and its corresponding pixel-level segmentation mask are output.

[0012] Secondly, embodiments of the present invention provide a zero-shot image segmentation system based on a three-dimensional object model, comprising: The reference feature construction module includes a rendering unit and a feature encoding unit. The rendering unit is configured to determine discrete viewpoints based on an icosahedral sampling strategy and generate a multi-view RGB reference image of a 3D CAD model through a rendering engine. The feature encoding unit is configured to load the DINOv3 model and extract the class token cls token of the reference image to construct a reference feature library.

[0013] The candidate mask generation module includes a SAM2 inference unit; the SAM2 inference unit is configured to receive the target image, run the image encoder and mask decoder of the SAM2 model, and automatically generate a set of candidate object masks covering the entire image through a grid point hint strategy.

[0014] The candidate feature extraction module includes an image preprocessing unit and a query feature extraction unit. The image preprocessing unit is configured to perform background removal, region cropping, and standardized scaling and filling operations on candidate objects to construct a batch of images of uniform size. The query feature extraction unit is configured to use the DINOv3 model to extract features from the batch of images and generate visual descriptors for candidate objects.

[0015] The feature matching and classification module includes a similarity calculation unit and a confidence aggregation unit; the similarity calculation unit is configured to calculate the cosine similarity matrix between candidate objects and reference templates; the confidence aggregation unit is configured to execute... The mean aggregation algorithm calculates the classification confidence and outputs the final object category and segmentation mask based on the maximum confidence criterion.

[0016] Compared with the prior art, the beneficial effects achieved by the present invention are: This invention utilizes 3D CAD models of objects to directly generate reference features. Combined with the powerful zero-shot generalization capabilities of the visual foundation models SAM2 and DINOv3, it completely eliminates the reliance of traditional supervised segmentation methods on large-scale manually labeled data and retraining models for new objects, significantly reducing the cost and time required to introduce new objects in industrial scenarios. Simultaneously, by converting candidate objects of different sizes into image batches with uniform resolution, this invention achieves efficient parallel computation of feature extraction, greatly improving the system's inference speed. Regarding the matching mechanism, this invention abandons the traditional strategy of matching a single maximum value and innovatively uses the top-k highest similarity mean as the classification confidence, effectively smoothing prediction noise caused by perspective differences or local occlusion, and enhancing recognition robustness in complex and cluttered scenes. Furthermore, thanks to the more refined edge-capturing capabilities of the SAM2 model and the more discriminative semantic representation space of the DINOv3 model, this invention not only generates higher-quality segmentation masks but also more effectively bridges the domain gap between synthetically rendered images and real-world images, thereby achieving high-precision, high-efficiency, and automated segmentation of unknown objects under zero-shot conditions. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a system framework diagram of a zero-shot image segmentation method based on a three-dimensional object model according to the present invention; Figure 2 This is a schematic diagram of constructing a reference feature module; Figure 3 This is a visualization of the reference template for the mustard_bottle object in the YCB-V dataset. Figure 4 This is a schematic diagram of the candidate mask generation module; Figure 5 This is a schematic diagram of the candidate feature extraction module; Figure 6 This is a schematic diagram of the feature matching and classification module; Figure 7 This is a visual diagram illustrating the test results of a zero-shot image segmentation method and system based on a 3D model of an object. Detailed Implementation

[0018] The technical solution of this invention is applicable to scenarios requiring high-precision pixel-level segmentation of unknown objects, such as industrial automation, robot grasping, augmented reality, and intelligent detection. It is particularly suitable for zero-sample segmentation requirements in industrial settings where product types frequently change. The technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] Please see Figures 1-7 The system components of this invention include: Reference feature building modules: such as Figure 2 As shown, this module is primarily responsible for establishing the visual feature index of the object's 3D model during the offline phase. Specifically, this module first utilizes the Pyrender graphics rendering engine to render the 3D CAD model of the object to be detected from multiple preset discrete viewpoints based on the icosahedral geometry, generating a model such as... Figure 3 The multi-view 2D RGB reference template image shown (taking the mustard_bottle object in the YCB-V dataset as an example).

[0020] Regarding the selection of the number of viewpoints, this invention is based on the icosahedral geometry. By subdividing it once, 42 approximately uniformly distributed vertices can be obtained, which are then used as viewpoints. Experimental verification on the YCB-V dataset shows that when... The accuracy reaches over 90%; when When the time is reduced to 20, the accuracy drops to about 85%; while when Increasing the number to 162 only improved the accuracy to around 92%, but the construction and matching time of the reference feature library increased by nearly three times. Therefore, selecting... As the preferred value, it ensures high accuracy while also taking into account computational efficiency.

[0021] Subsequently, the module calls the pre-trained visual foundation model DINOv3 as a feature extractor to perform forward inference on each rendered reference template image, extracting the class label cls token output by the model as the global semantic feature vector under that viewpoint. The feature vectors of all objects under all viewpoints are collected and stored to build a reference feature library for subsequent matching.

[0022] Candidate mask generation module: such as Figure 4As shown, this module aims to automatically discover all potential object regions in an image without any human interaction or category prior. The module receives the target RGB image to be segmented as input and loads a pre-trained SAM2 model. Utilizing SAM2's automatic mask generation mode, the module drives the SAM2 image encoder and mask decoder to perform inference by generating high-density grid point cues across the entire image, outputting several binary candidate object masks.

[0023] Select the feature extraction module: such as Figure 5 As shown, this module is used to transform geometric masks into semantic feature representations. The module first performs image preprocessing on each candidate object mask generated in the previous step: it uses the mask to mask the original image to remove background noise, crops the object region based on the mask's bounding box, and performs aspect ratio-preserving scaling and padding operations. All object regions of different sizes are then uniformly adjusted to a fixed resolution, forming an image batch. It should be noted that the selection of the above fixed resolution (224×224, 256×256, or 512×512) is merely illustrative and does not constitute a limitation of the invention. Those skilled in the art can adjust it according to actual inference efficiency and accuracy requirements; for example, a smaller resolution can be used when pursuing higher inference speed, and a larger resolution can be used when pursuing higher segmentation accuracy.

[0024] Subsequently, the module inputs the processed images in batches into the DINOv3 model, extracts the global feature vector of each candidate object, and thus maps the objects in the image into the same semantic feature space as the reference feature library.

[0025] Feature matching and classification module: such as Figure 6 As shown, this module is responsible for completing the final zero-shot segmentation task. This module calculates the cosine similarity matrix between the candidate object feature vector and the feature vectors of all templates in the reference feature library.

[0026] To improve the robustness of recognition, this module does not simply rely on maximum similarity, but instead employs... Mean aggregation strategy: For each candidate object and each object category, select the top-ranked objects with the highest mean values. The arithmetic mean of the similarity scores is used as the final confidence score for the candidate object to belong to the category.

[0027] Through this The mean aggregation strategy effectively smooths out prediction noise caused by differences in viewpoint or local occlusion, enhancing recognition robustness. For The choice of value, if (That is, only the highest similarity is taken), which can easily lead to mismatches when there is partial occlusion of objects; if Value too large (e.g.) This would reduce the confidence of correctly classifying a match due to the introduction of a large number of low-quality matches. Tests have shown that... In this case, It can achieve the best balance between suppressing noise and maintaining feature specificity, and the system has the best robustness in object recognition in cluttered scenes.

[0028] Finally, the system classifies and determines the candidate masks based on the confidence scores, and outputs the following: Figure 7 The final test results are shown in the visualization.

[0029] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A zero-shot image segmentation method based on a 3D model of an object, characterized in that: The method includes: Step 1: Construct an object reference feature library: Obtain the computer-aided design (CAD) model of the object to be detected, and use a graphics rendering engine to generate a 2D RGB reference template image of the object under a preset discrete viewpoint; input the reference template image into the pre-trained visual basic model DINOv3, extract the class label cls token output by the model as a global feature vector, and construct a reference feature library containing multi-view semantic information of the object. Step 2: Generate candidate object masks for the entire image: Input the RGB image of the target to be segmented into the pre-trained visual base model SAM2, and use its automatic mask generation mode to generate several class-independent candidate object masks across the entire image. Step 3: Extract the semantic feature vector of the candidate objects: For each candidate object mask generated in Step 2... The target image undergoes background removal, cropping, and normalization. The processed image is then input into DINOv3 to extract visual descriptors for candidate objects, forming a set of candidate object visual descriptors. Step 4: Calculate classification confidence and output segmentation results: Compare the candidate object visual descriptors obtained in Step 3 with the reference feature library constructed in Step 1. The cosine similarity between the feature vectors of each template is used; for each class of objects to be detected, the top-ranked templates with the highest cosine similarity are selected. The cosine similarity values ​​are calculated and their arithmetic mean is used as the classification confidence of the candidate object mask belonging to the category. The category with the highest classification confidence is assigned to the candidate object mask, and the final target object category label and its corresponding pixel-level segmentation mask are output.

2. The zero-shot image segmentation method based on a 3D object model as described in claim 1, characterized in that, Step 1 includes: S101—Discrete view rendering: using the geometric center of the object's 3D model as the center of a sphere, based on the preset icosahedral geometry. A discrete viewpoint is used to ensure that the viewpoint can uniformly cover the object; the 3D CAD model is rendered into an RGB reference template image at each viewpoint using a rendering engine (BlenderProc); the total number of the reference template images is denoted as . , The value range is 30-60. S102—Reference Feature Extraction: The RGB reference template image is cropped and standardized; the processed template image is input into the DINOv3 feature extractor; the class token (cls token) output by the model is extracted as the feature vector of this viewpoint, and the set of feature vectors of all viewpoints is denoted as the reference feature library V of the object, with a dimension of . ,in, For the number of objects, The feature dimensions output by the DINOv3 model.

3. The zero-shot image segmentation method based on a 3D object model as described in claim 2, characterized in that, The The preferred number of discrete viewpoints is 42.

4. The zero-shot image segmentation method based on a 3D object model as described in claim 1, characterized in that, In step 2, automatic mask generation involves inputting the RGB image to be tested into the SAM2 model. Using its image encoder and mask decoder, a set of masks is automatically generated across the entire image. Candidate object segmentation mask ; The number of candidate object masks It is not fixed and adapts to the content of the input image.

5. The zero-shot image segmentation method based on a 3D object model as described in claim 1, characterized in that, Step 3 includes: S301—Background removal and region cropping: For each candidate object mask The original input RGB image is masked using this mask to remove the background outside the masked area (i.e., object pixels are retained, and background pixels are set to zero). Then, the bounding box is calculated based on the mask, and the object region image is cropped from the background-removed image. S302—Image Standardization: To address the problem of inconsistent candidate object image sizes hindering parallel computation, the cropped object region image undergoes unified image processing. First, the original aspect ratio of the image is maintained during scaling. The scaled image is then padded to a fixed resolution, selectable as 224×224, 256×256, or 512×512. All standardized candidate object images are packaged into an image batch for efficient parallel processing. S303—Candidate Feature Extraction: The image batch is input into the DINOv3 model to extract the class label clstoken corresponding to each image. The extracted feature vectors are used as a set of candidate object visual descriptors. Its dimensions are .

6. The zero-shot image segmentation method based on a 3D object model as described in claim 5, characterized in that, The fixed resolution is preferably 224×224.

7. The zero-shot image segmentation method based on a 3D object model as described in claim 1, characterized in that, Step 4 includes: S401—Similarity matrix calculation: Calculate the set of visual descriptors for candidate objects. Compared with the reference feature library The cosine similarity between the two objects generates a similarity matrix (dimension 1) between the candidate object and the reference feature library. The similarity matrix represents the degree of similarity between each candidate object mask and each viewpoint template for each object category. S402 – Calculation of the mean confidence score: For each object category and its candidate masks, from… Select the viewpoints with the highest similarity scores. The fractions, the Value range 3-10, preferred. ; calculate this The arithmetic mean of the similarity scores is used as the similarity score of the candidate mask belonging to the object category, thus obtaining a dimension of The confidence matrix. S403 - Category determination and mask output: Apply the maximum value function argmax to the object category dimension of the confidence matrix, assign the object category ID with the highest score to the corresponding candidate object mask, and use the highest score as the final confidence score; output the segmentation result containing the mask, predicted category ID and confidence score.

8. The zero-shot image segmentation method based on a 3D object model as described in claim 7, characterized in that, The The preferred value is 5.

9. A zero-shot image segmentation system based on a three-dimensional object model that implements the method of any one of claims 1-8, characterized in that: The system includes a reference feature construction module, a candidate mask generation module, a candidate feature extraction module, and a feature matching and classification module, which are sequentially connected through data interaction. The functions of each module are as follows: Reference Feature Construction Module: This module processes the 3D CAD model of the object to be detected in the offline stage. It integrates a graphics rendering engine and a pre-trained visual base model DINOv3. The graphics rendering engine is configured to preset 30-60 discrete viewpoints based on the icosahedral geometry, with the object's geometric center as the sphere's center, and renders the 3D CAD model at these preset discrete viewpoints to generate multiple 2D RGB reference template images. DINOv3 is configured to extract the class marker clstoken from each reference template image as a global feature vector and store the feature vectors from all viewpoints as a reference feature library. Candidate Mask Generation Module: This module receives the target RGB image to be segmented and loads the pre-trained visual base model SAM2. It is configured to run the automatic mask generation mode of SAM2, generating several binary candidate object masks (proposals) across the entire image without relying on specific category cues, and then transmitting these candidate object masks to the candidate feature extraction module. Candidate feature extraction module: used to process the candidate object mask and extract its semantic feature vector; this module includes an image preprocessing unit and a feature extraction unit. The image preprocessing unit is configured to perform background removal operation, crop the object region according to the mask bounding box, scale and zero-padding operation while maintaining the aspect ratio for each candidate object mask, and unify all candidate object images to a fixed resolution to construct an image batch. The feature extraction unit is configured to reuse the DINOv3 model to extract the semantic feature vector of each candidate object in the image batch, and then transmit the semantic feature vector to the feature matching and classification module. The feature matching and classification module is used to perform feature alignment and category determination; this module is configured to calculate the semantic feature vector of the candidate object and the reference feature library. The module calculates the cosine similarity matrix between the objects. It integrates an aggregation calculation unit and a category determination unit. The aggregation calculation unit is configured to sort the similarities of each candidate object across all viewpoints within a specific category, selecting the highest-valued similarity. The arithmetic mean of the similarity calculation is used as the classification confidence score; The category determination unit is configured to determine the category label of the candidate object mask according to the maximum confidence criterion, and output the final target object category label and the corresponding pixel-level segmentation mask.

Images