A zero-shot object instance segmentation system and method under a robot environment

Abstract

The present invention discloses a zero-sample object instance segmentation system and method in a robot environment, which belongs to the field of robot visual perception technology. First, the depth image is preprocessed with Viridis color mapping, and then the SAM (Segment Anything Model) model is used to generate initial object mask candidates; at the same time, the pre-trained ViT (Vision Transformer) is used as the feature description model of the scene to process the image, extract the attention map of the last layer, and construct a feature weighting mechanism based on the information entropy of the attention map; then the similarity matrix of the image block and the background block is calculated to remove the non-object mask, and the K-Medoids clustering algorithm is used to obtain representative sampling points for each remaining object mask candidate. Finally, these points are input into the SAM model as prompt information to obtain accurate object instance segmentation results. The present invention makes full use of the zero-sample generalization ability of the visual basic model and can achieve accurate segmentation of unseen objects without additional training, with good versatility and practical value.

Description

Technical Field

[0001] The present invention relates to the field of robot visual perception technology, and in particular to a zero-sample object instance segmentation system and method in a robot environment. Background Art

[0002] With the widespread use of service robots in unstructured environments such as homes, hospitals, and supermarkets, there is an urgent need for robots to be able to identify and segment unseen objects. In real-world applications, robots may encounter a vast number of unseen objects, and building models for every type of object is impractical. Therefore, enabling robots to segment unseen objects is crucial for improving their manipulation, grasping, and processing capabilities in diverse real-world environments. This is of great value in promoting the practical application of service robots in both domestic and industrial settings.

[0003] Currently, object segmentation techniques primarily include approaches based on supervised learning, unseen object instance segmentation (UOIS), interactively perceived UOIS, and foundational models. Supervised learning-based segmentation methods rely heavily on large-scale, manually annotated datasets for training. However, in robotic manipulation scenarios, large-scale, real-world datasets with sufficient object diversity are lacking, and constructing such datasets is costly and challenging. Unseen object instance segmentation (UOIS) methods based on synthetic data aim to enable models to acquire a sense of "objectness" and generalize to new objects. Some of these methods combine the effective generalization capabilities of depth images with the ability to generate clear masks from RGB images, achieving good segmentation performance in real-world environments. However, synthetic data often fails to accurately represent real-world object characteristics such as texture, lighting conditions, and depth noise. This sim2real (sim-to-real) gap significantly impacts model performance in real-world environments. Interactively perceived UOIS methods actively acquire labels for unseen objects through robot actions such as grasping and pushing, and then fine-tune the pre-trained model using newly acquired real-world data. However, these methods either rely on synthetic data for training, which suffers from a sim2real gap, or require post-processing steps, which lack flexibility during actual robot operation. Recently proposed foundational models such as the Segmented Arbitrary Object Model (SAM) have shown significant progress in various computer vision applications. Trained on a dataset of 11 million images and over 1 billion masks, SAM demonstrates excellent zero-shot generalization capabilities and can efficiently generate accurate object masks based on cues such as bounding boxes and specific points. However, the main limitation of SAM is that it cannot always capture complete instances, often leading to over-segmentation.

[0004] In summary, existing technologies face the following challenges: traditional supervised learning methods require large amounts of labeled data, which is difficult to obtain; methods based on synthetic data suffer from a significant gap between simulation and reality; interactive methods lack real-time performance and flexibility; and existing basic models still fall short in segmenting complete object instances. Therefore, a new technical solution is urgently needed to address these issues. Summary of the Invention

[0005] To address the challenges of traditional supervised learning methods requiring large amounts of hard-to-obtain labeled data, the significant simulation-to-reality gap in synthetic data-based methods, the lack of real-time and flexibility in interactive methods, and the inadequacy of existing basic models in complete object instance segmentation, the present invention provides a zero-shot object instance segmentation system and method in a robotic environment. These systems leverage the zero-shot generalization and feature representation capabilities of basic visual models to address the UOIS challenge without requiring additional training.

[0006] The technical solution adopted by the zero-sample object instance segmentation system and method in a robot environment of the present invention is:

[0007] A zero-sample object instance segmentation system in a robot environment, characterized by:

[0008] Image acquisition module: the robot uses an RGB-D camera to acquire the RGB image and depth image of the current scene for subsequent processing;

[0009] The processing module maps the depth image collected by the image acquisition module into the RGB color space;

[0010] The initial segmentation module inputs the processed colorized depth image into the image segmentation visual basic model to generate a variety of object mask candidates;

[0011] The precise segmentation module uses the pre-trained ViT to extract the attention map of the image acquired by the image acquisition module, then calculates the information entropy of the attention map of each attention head, constructs a feature weighting mechanism, and then calculates the background similarity matrix to screen the object mask area and sample it. Finally, the representative points obtained by sampling are input into the SAM model as prompt information to achieve precise segmentation of object instances.

[0012] A further improvement of the above technical solution of the present invention is that: the mask includes a foreground object, a background area and a partial noise mask.

[0013] A zero-shot object instance segmentation method in a robotic environment, using the above segmentation system, includes the following steps:

[0014] S1. Obtain the color and depth information image of the current scene;

[0015] S2, using Viridis color mapping method to map the depth image to RGB color space;

[0016] S3. Input the preprocessed colorized depth image into the SAM model, set the non-maximum suppression threshold to a low value, and generate diverse object mask candidates;

[0017] S4. Use the pre-trained ViT as the feature description model for scene representation to process the image and extract the attention map between the category label (CLS) of each attention head and the labels of all image patches;

[0018] S5. Attention map of different attention heads based on information entropy a i Perform weighting;

[0019] S6, calculating the similarity matrix between the background block and all image blocks to determine whether a block belongs to the background;

[0020] S7, applying K-Medoids clustering algorithm to sample in the filtered object mask area;

[0021] S8. The representative points obtained by sampling are input into the SAM model as prompt information, and the accurate segmentation capability of the model is used to generate improved object instance segmentation results.

[0022] The further improvement of the above technical solution of the present invention is that: the step S3 is specifically, in the segmentation stage, given a colored depth image The SAM model with default settings is used for processing, and the non-maximum suppression threshold is set to 0.5 to generate N m Mask candidates that are independent of object category where for i=1,2,...,N m , there are m i ∈{0,1} H×W .

[0023] The further improvement of the above technical solution of the present invention is that: the step S4 is specifically to process the input RGB image by using ViT trained by DINOv2 as the feature description model of scene representation ViT divides the image into K×K non-overlapping image blocks, with a total number of N p =HW / K 2 , in the last layer of ViT, there are N h Attention heads, each attention head i will generate an attention map Represents the attention weight between the category label and all image patch labels, and the key feature representation of the last layer is recorded as Where D represents the dimension of the feature vector.

[0024] A further improvement of the above technical solution of the present invention is that: the step S5 is specifically as follows: for each attention map a i , its information entropy is expressed by the formula Calculation, where P(·) represents the normalization operation; the weight is calculated by the formula Calculation is performed so that attention heads with low entropy values ​​get higher weights.

[0025] A further improvement of the above technical solution of the present invention is that the index of the background block in step S6 is obtained by the formula Determine; weighted feature F ω By element-wise multiplication F ω =W⊙F is obtained and reshaped into The cosine similarity matrix between the final background block features and all other image block features It is used to represent the score of each image block as the background; when the average score of the image block corresponding to the mask exceeds the preset threshold τ, the mask is identified as the background and removed, and the object mask set M is obtained after removing the non-object mask o .

[0026] A further improvement of the above technical solution of the present invention is that: in step S7, in each candidate object mask area M o The cluster center of the K-Medoids clustering algorithm is used as the positive sample point.

[0027] A further improvement of the above technical solution of the present invention is that in step S8, the positive sample point prompts obtained in S7 are used to guide SAM to perform segmentation, thereby obtaining an improved segmentation result.

[0028] Due to the adoption of the above technical solution, the technical effects achieved by the present invention are as follows:

[0029] This invention fully utilizes the advantages of the visual foundation model in zero-sample generalization and feature representation, combined with the depth information and color information obtained by the RGB-D camera, to achieve high-precision object instance segmentation. The present invention combines the image segmentation and feature extraction visual foundation model to achieve the recognition and segmentation of new objects without additional training; at the same time, the present invention improves the recognition ability of key object areas in the image through a multi-layer attention mechanism and a feature weighting method based on information entropy; and the present invention designs a mask optimization method based on the background similarity matrix, which effectively improves the accuracy of the object instance segmentation results. These technological innovations improve the generalization ability of the unseen object instance segmentation model and provide reliable technical support for the autonomous operation of robots in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram of a zero-shot object instance segmentation system in a robotic environment of the present invention;

[0031] Figure 2 Flowchart of the zero-shot object instance segmentation method in a robot environment of the present invention;

[0032] Figure 3 This is the overall network structure diagram of the zero-shot object instance segmentation method in the robot environment of the present invention;

[0033] Figure 4 This is a visualization example of the segmentation results of the zero-shot object instance segmentation method in the robot environment of the present invention on the OCID dataset;

[0034] Figure 5 This is a diagram of the zero-sample object instance segmentation method in a robot environment of the present invention being applied to a robot physical grasping test process. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. In the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0036] The present invention provides a zero-sample object instance segmentation system and method in a robot environment.

[0037] Example 1

[0038] The zero-sample object instance segmentation system in the robot environment of the present invention is specifically arranged with reference to Figure 1 It can be seen that the zero-shot object instance segmentation system in the robot environment includes an image acquisition module, a processing module, an initial segmentation module and a precise segmentation module connected in sequence.

[0039] Furthermore, in the above-mentioned image acquisition module, the robot uses an RGB-D camera to acquire the RGB image and depth image of the current scene for subsequent processing.

[0040] The processing module maps the depth image collected by the image acquisition module to the RGB color space.

[0041] The initial segmentation module feeds the processed colorized depth image into the image segmentation visual foundation model to generate a variety of object mask candidates. These masks include foreground objects, background regions, and partial noise masks, providing candidate regions for subsequent precise segmentation.

[0042] The above-mentioned precise segmentation module uses the pre-trained ViT extraction image acquisition module to obtain the attention map of the image to effectively capture the structural and semantic information of the objects in the image. It then calculates the information entropy of the attention map of each attention head and constructs a feature weighting mechanism to highlight the key object areas. It then calculates the background similarity matrix, screens the object mask area and samples it. Finally, the representative points obtained by sampling are input into the SAM model as prompt information to achieve precise segmentation of object instances.

[0043] Example 2

[0044] The second embodiment is to use the segmentation method of the segmentation system in the first embodiment, such as Figure 2 As shown in the figure, the method includes the following steps: First, the depth image is colorized and then fed into the segmentation model to obtain object-agnostic mask candidates. Simultaneously, the image is fed into the description model for feature extraction. These extracted features effectively capture the semantic information of the objects in the image, helping to filter out non-object masks from the segmentation model's segmentation results, thereby obtaining an initial segmentation result. Subsequently, the K-Medoids clustering algorithm is applied within the filtered object mask region to extract positive sample points, which are then used as cues to further refine the initial segmentation result. The final segmentation result has high object boundary segmentation accuracy and can segment unseen objects without additional training.

[0045] refer to Figure 3 It can be seen that the segmentation method in this embodiment specifically includes the following steps:

[0046] S1. Obtain the color and depth information image of the current scene.

[0047] S2. Depth image preprocessing: Viridis color mapping method is used to map the depth image to RGB color space.

[0048] S3. Generate an initial mask, input the preprocessed colorized depth image into the SAM model, set the non-maximum suppression threshold to a low value (such as 0.5), and generate a variety of object mask candidates. These masks include foreground objects, background areas, and partial noise masks, providing candidate areas for subsequent precise segmentation.

[0049] S4. Feature Extraction: We process images using the pre-trained ViT as a feature description model for scene representation and extract attention maps between the category labels (CLS) of each attention head and the labels of all image patches. These attention maps reflect the semantically meaningful visual patterns formed by the ViT model during self-supervised learning and can effectively capture the structural and semantic information of objects in the image.

[0050] S5. Feature weighting: Based on the extracted attention map, the information entropy of each image region is calculated, and a feature weighting mechanism is constructed. This mechanism assigns weights according to the information richness of the region, so that regions with significant visual features obtain higher weight values, which helps to highlight key object areas.

[0051] S6. Calculate the background similarity matrix. Construct a similarity matrix between the image block and the predefined background block. Use cosine similarity to measure the degree of association between different regions. Determine whether a block belongs to the background by calculating the similarity matrix between the background block and all image blocks.

[0052] S7: Region sampling: Within the filtered object mask region, the K-Medoids clustering algorithm is applied for sampling. The K-Medoids clustering algorithm uses iterative optimization to find the most representative set of sample points that can effectively represent the spatial distribution of the object.

[0053] S8: Final Segmentation: The representative points obtained by sampling are input into the SAM model as hints, leveraging the model's precise segmentation capabilities to generate improved object instance segmentation results. This step pays special attention to the accuracy of object edge regions, improving the fineness of segmentation through a multi-point hinting strategy.

[0054] The segmentation method of the present invention was validated on the OCID dataset, a widely used instance segmentation method for unseen objects in indoor environments. In this embodiment, the segmentation method employs two main visual foundation models: the SAM model for segmentation and the ViT feature description model for scene representation. The SAM model uses the ViT-H / 16 backbone network and sets several key parameters to ensure optimal segmentation performance: the bounding box non-maximum suppression threshold is set to 0.5 to control the overlap tolerance between predicted bounding boxes; a higher value allows more overlapping prediction results to be retained; the image crop overlap ratio is set to 0, indicating that there is no overlapping area between adjacent image crops; the minimum mask area is set to 0, allowing detection of segmented regions of any size; the number of cue points processed per batch is set to 64; and the stability score threshold is set to 0.95 to ensure that only highly reliable segmentation results are retained.

[0055] In terms of the feature description model, this embodiment adopts the ViT-B / 14 architecture and uses DINOv2 pre-trained weights. During the mask filtering stage, this embodiment sets the background similarity threshold τ to 0.47. During the point cue generation stage, this embodiment applies the K-Medoids clustering algorithm to each target candidate region, generating three point cues.

[0056] The computational steps for testing a set of RGB-D images are as follows:

[0057] Input: preprocessed RGB image and depth image;

[0058] Step 1: This method maps the depth image to RGB color space, specifically using the viridis color mapping scheme. The converted depth map is used as the input for the segmentation task. This process helps to extract complete object mask candidates that do not rely on color and texture information.

[0059] Step 2: In the segmentation stage, given a colorized depth image The SAM model was processed using the default settings, but the NMS threshold was set to a lower value of 0.5 to generate N m There are mask candidates M={m1,m2,...,m Nm}, where for i=1,2,...,N m , there are m i ∈{0,1} H×W .

[0060] Step 3: Process the input RGB image through ViT trained with DINOv2 ViT divides the image into K×K non-overlapping image blocks, with a total number of N p =HW / K 2 In the last layer of ViT, there are N h Attention heads, each attention head i will generate an attention map Represents the attention weight between the category label (CLS) and all image patch labels. The key feature representation of the last layer is denoted as Where D represents the dimension of the feature vector.

[0061] Step 4: Attention map a of different attention heads based on information entropy i For each attention map, its information entropy is calculated by the formula Calculate, where P(·) represents the normalization operation. The weight is calculated by the formula Calculation ensures that attention heads with lower entropy values ​​receive higher weights.

[0062] Step 5: Calculate the similarity matrix between the background block and all image blocks to determine whether a block belongs to the background. The index of the background block is calculated by the formula OK. Weighted feature F ω By element-wise multiplication F ω =W⊙F is obtained and reshaped into The final cosine similarity matrix S between the background block and all other image blocks cosUsed to calculate the score of each image block as background. When the average score of the image block corresponding to the mask exceeds the preset threshold τ, the mask is identified as background and removed. After removing the non-object mask, the object mask set M is obtained. o .

[0063] Step 6: In each candidate object mask region M o The cluster centers of the K-Medoids clustering algorithm are used as positive sample points. The obtained point hints are used to guide SAM to perform segmentation, thereby obtaining improved segmentation results.

[0064] Output: Instance segmentation results of objects in the image.

[0065] The segmentation method in this invention has good performance in the field of indoor scene object instance segmentation. First, a verification test was conducted on the widely used OCID dataset, such as Figure 4 The segmentation results shown in the figure demonstrate that the segmentation method of the present invention can fully utilize the zero-sample recognition capability and powerful feature representation capability of the visual base model to effectively segment the target objects in the RGB-D image.

[0066] In order to further verify the feasibility of the method in actual application scenarios, a field test was conducted using the Fetch mobile robot. During the test, the scene image was first captured by the RGB-D camera on the robot's head, and then the objects in the image were segmented using the segmentation method proposed in this invention. After the object segmentation is completed, the system calls Contact-GraspNet to perform grasping posture planning, and MoveIt performs the specific motion planning. Figure 5 As shown in the figure, the experiment fully records the entire process of the robot from target recognition, object segmentation to final successful grasping, which fully demonstrates the effectiveness of this method in practical applications.

[0067] In the above-mentioned embodiments, the present invention provides a system and method for zero-shot object instance segmentation in a robotic environment. This system leverages the powerful capabilities of the visually grounded model in zero-shot generalization and feature representation to achieve zero-shot instance segmentation of objects in indoor robotic environments without additional training. By using the explicit visual representation of the pre-trained ViT, the present invention effectively removes non-object masks, significantly improving segmentation accuracy.

[0068] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A zero-shot object instance segmentation method in a robotic environment, characterized by: The following steps are included: S1. Obtain the color and depth information image of the current scene; S2, using Viridis color mapping method to map the depth image to RGB color space; S3. Input the preprocessed colorized depth image into the SAM model, set the non-maximum suppression threshold to a low value, and generate diverse object mask candidates; The step S3 is specifically as follows: in the segmentation stage, given a colored depth image The SAM model with default settings is used for processing, and the non-maximum suppression threshold is set to 0.5 to generate N m Mask candidates that are independent of object category where for i=1,2,...,N m , there are m i ∈{0,1} H×W ; S4. Use the pre-trained ViT as the feature description model for scene representation to process the image and extract the attention map between the category label of each attention head and the labels of all image patches; Specifically, step S4 is to process the input RGB image using ViT trained by DINOv2 as the feature description model for scene representation. ViT divides the image into K×K non-overlapping image blocks, with a total number of N p =HW / K 2 , in the last layer of ViT, there are N h Attention heads, each attention head i will generate an attention map Represents the attention weight between the category label and all image patch labels, and the key feature representation of the last layer is recorded as Where D represents the dimension of the feature vector; S5. Attention map of different attention heads based on information entropy a i Perform weighting; The step S5 is specifically as follows: for each attention map a i , its information entropy is expressed by the formula Calculation, where P(·) represents the normalization operation; the weight is calculated by the formula Calculation, so that the attention head with low entropy value gets higher weight; S6, calculating the similarity matrix between the background block and all image blocks to determine whether a block belongs to the background; S7, applying K-Medoids clustering algorithm to sample in the filtered object mask area; The index of the background block in step S6 is calculated by the formula Determine; weighted feature F ω By element-wise multiplication F ω =W⊙F is obtained and reshaped into The cosine similarity matrix between the final background block features and all other image block features Used to indicate the score of each image patch as background; When the average score of the image block corresponding to the mask exceeds the preset threshold τ, the mask is identified as the background and removed. After removing the non-object mask, the object mask set M is obtained. o ; S8. The representative points obtained by sampling are input into the SAM model as prompt information, and the accurate segmentation capability of the model is used to generate improved object instance segmentation results.

2. The method for zero-shot object instance segmentation in a robotic environment according to claim 1, wherein: The step S7 is to perform a masking operation on each candidate object mask area M. o The cluster center of the K-Medoids clustering algorithm is used as the positive sample point.

3. The method for zero-shot object instance segmentation in a robotic environment according to claim 2, wherein: The step S8 uses the positive sample point prompts obtained in S7 to guide SAM to perform segmentation, thereby obtaining an improved segmentation result.

4. A zero-shot object instance segmentation system in a robotic environment, characterized by: Implementing the segmentation method according to any one of claims 1 to 3, comprising: Image acquisition module: the robot uses an RGB-D camera to acquire the RGB image and depth image of the current scene for subsequent processing; The processing module maps the depth image collected by the image acquisition module into the RGB color space; The initial segmentation module inputs the processed colorized depth image into the image segmentation visual basic model to generate a variety of object mask candidates; The precise segmentation module uses the pre-trained ViT extraction image acquisition module to obtain the attention map of the image, then calculates the information entropy of the attention map of each attention head, constructs a feature weighting mechanism, and then calculates the background similarity matrix to screen the object mask area and sample it. Finally, the representative points obtained by sampling are input into the SAM model as prompt information to achieve precise segmentation of object instances.

5. The zero-shot object instance segmentation system in a robotic environment according to claim 4, characterized in that: The mask includes a foreground object, a background area, and a partial noise mask.

Images