Zero sample object instance segmentation system and method in robot environment

Abstract

The invention discloses a zero sample object instance segmentation system and method in a robot environment, and belongs to the technical field of robot visual perception, and the method comprises the steps: firstly carrying out the Vibriis color mapping preprocessing of a depth image, and then generating an initial object mask candidate through an SAM (Segment Anything Model) model; meanwhile, a pre-trained Vision Transform is used as a feature description model of a scene to process the image, an attention map of the last layer is extracted, and a feature weighting mechanism is constructed based on the information entropy of the attention map; a similarity matrix of the image blocks and the background blocks is calculated to remove non-object masks, a K-Medoids clustering algorithm is adopted for each remaining object mask candidate to obtain representative sampling points, and finally the points serve as prompt information to be input into an SAM model to obtain an accurate object instance segmentation result. According to the method, the zero sample generalization ability of the visual basic model is fully utilized, accurate segmentation of the unseen object can be realized without additional training, and the method has good universality and practical value.

Description

Technical Field

[0001] The present invention relates to the field of robot visual perception technology, and in particular to a zero-sample object instance segmentation system and method in a robot environment. Background Art

[0002] With the widespread application of service robots in unstructured environments such as homes, hospitals, and supermarkets, there is an urgent need for robots to be able to identify and segment unseen objects. In actual application scenarios, robots may encounter a large number of unseen objects, and it is impractical to build models for each type of object. Therefore, enabling robots to segment unseen objects is of great significance for improving their operation, grasping, and processing capabilities in diverse real-world environments, which is of great value in promoting the practical application of service robots in home and industrial environments.

[0003] At present, object segmentation technology mainly includes solutions based on supervised learning, unseen object instance segmentation (UOIS), interactive perception UOIS, and basic models. The segmentation method based on supervised learning relies heavily on large-scale manually annotated datasets for training. However, in the robot operation scenario, there is a lack of large-scale real datasets with sufficient object diversity, and it is costly and difficult to build such datasets. The unseen object instance segmentation (UOIS) method based on synthetic data aims to enable the model to obtain the concept of "objectness" and generalize to new objects. Some of these methods combine the effective generalization ability of depth images and the ability of RGB images to generate clear masks, and achieve good segmentation performance in real environments. However, synthetic data often cannot accurately represent the characteristics of objects in the real world, such as texture, lighting conditions, and depth noise. This gap between simulation and reality (sim2real) seriously affects the performance of the model in real environments. The UOIS method based on interactive perception actively obtains the labels of unseen objects through the robot's grasping, pushing and other actions, and uses the newly acquired real data to fine-tune the pre-trained model. However, these methods either rely on synthetic data for training with a sim2real gap or require post-processing steps, which lack flexibility during actual robot operation. Recently proposed basic models such as the Segmented Arbitrary Object Model (SAM) have shown significant progress in various computer vision applications. Trained on a dataset of 11 million images and over 1 billion masks, SAM demonstrates excellent zero-shot generalization capabilities and can efficiently generate accurate object masks based on cues such as bounding boxes and specific points. However, the main limitation of SAM is that it cannot always capture complete instances, often resulting in over-segmentation issues.

[0004] In summary, the existing technologies have the following problems: traditional supervised learning methods require a large amount of annotated data that is difficult to obtain; methods based on synthetic data have a significant gap between simulation and reality; methods based on interaction lack real-time and flexibility; and existing basic models are still insufficient in complete object instance segmentation. Therefore, it is urgent to propose a new technical solution to solve the above problems. Summary of the invention

[0005] In order to solve the problem that the current traditional supervised learning methods require a large amount of annotated data that is difficult to obtain; in order to solve the problem that the methods based on synthetic data have a significant gap between simulation and reality; in order to solve the problem that the methods based on interaction are not real-time and flexible enough; and in order to solve the problem that the existing basic models are still insufficient in complete object instance segmentation. The present invention provides a zero-sample object instance segmentation system and method in a robot environment, which can utilize the zero-sample generalization and feature representation capabilities of the visual basic model to cope with the UOIS challenge without additional training.

[0006] The technical solution adopted by the zero-sample object instance segmentation system and method in a robot environment of the present invention is:

[0007] A zero-sample object instance segmentation system in a robot environment, characterized by:

[0008] Image acquisition module: the robot uses an RGB-D camera to acquire the RGB image and depth image of the current scene for subsequent processing;

[0009] A processing module maps the depth image collected by the image acquisition module to an RGB color space;

[0010] The initial segmentation module inputs the processed colorized depth image into the image segmentation visual basic model to generate a variety of object mask candidates;

[0011] The precise segmentation module uses the pre-trained ViT to extract the attention map of the image acquired by the image acquisition module, then calculates the information entropy of the attention map of each attention head, constructs a feature weighting mechanism, and then calculates the background similarity matrix to screen the object mask area and sample it. Finally, the representative points obtained by sampling are input into the SAM model as prompt information to achieve precise segmentation of object instances.

[0012] A further improvement of the above technical solution of the present invention is that: the mask includes a foreground object, a background area and a partial noise mask.

[0013] A zero-sample object instance segmentation method in a robot environment, using the above segmentation system, comprises the following steps:

[0014] S1, obtain the color and depth information image of the current scene;

[0015] S2, using Viridis color mapping method to map the depth image to RGB color space;

[0016] S3, input the preprocessed colorized depth image into the SAM model, set the non-maximum suppression threshold to a low value, and generate diverse object mask candidates;

[0017] S4, using the pre-trained ViT as the feature description model for scene representation to process the image and extract the attention map between the category label (CLS) of each attention head and the labels of all image patches;

[0018] S5. Attention map of different attention heads based on information entropy a i weighting;

[0019] S6, calculating the similarity matrix between the background block and all image blocks to determine whether a block belongs to the background;

[0020] S7, applying K-Medoids clustering algorithm to sample in the filtered object mask area;

[0021] S8. The representative points obtained by sampling are input into the SAM model as prompt information, and the accurate segmentation ability of the model is used to generate improved object instance segmentation results.

[0022] A further improvement of the above technical solution of the present invention is that: the step S3 is specifically, in the segmentation stage, given a colorized depth image The SAM model with default settings was used for processing, and the non-maximum suppression threshold was set to 0.5 to generate N m Mask candidates that are independent of object category where for i=1,2,...,N m , there are m i ∈{0,1} H×W .

[0023] The further improvement of the above technical solution of the present invention is that: the step S4 is specifically to process the input RGB image by using ViT trained by DINOv2 as the feature description model of scene representation ViT divides the image into K×K non-overlapping image blocks, with a total number of N p =HW / K 2 , in the last layer of ViT, there are N h Attention heads, each attention head i will generate an attention map represents the attention weight between the category label and all image patch labels, and the key feature representation of the last layer is recorded as Where D represents the dimension of the feature vector.

[0024] A further improvement of the above technical solution of the present invention is that: the step S5 is specifically, for each attention map a i , whose information entropy is expressed by the formula Calculation, where P(·) represents the normalization operation; the weight is calculated by the formula Calculate so that the attention heads with low entropy values ​​get higher weights.

[0025] A further improvement of the above technical solution of the present invention is that the index of the background block in step S6 is obtained by the formula Determine; weighted feature F ω By element-wise multiplication F ω =W⊙F is obtained and reshaped into The final cosine similarity matrix between the background block features and all other image block features It is used to represent the score of each image block as the background; when the average score of the image block corresponding to the mask exceeds the preset threshold τ, the mask is identified as the background and removed, and the object mask set M is obtained after removing the non-object mask o .

[0026] A further improvement of the above technical solution of the present invention is that: in step S7, in each candidate object mask area M o The cluster center of the K-Medoids clustering algorithm is used as the positive sample point.

[0027] A further improvement of the above technical solution of the present invention is that the step S8 uses the positive sample point prompts obtained in S7 to guide SAM to perform segmentation, thereby obtaining an improved segmentation result.

[0028] Due to the adoption of the above technical solution, the technical effects achieved by the present invention are as follows:

[0029] The present invention makes full use of the advantages of the visual basic model in zero-sample generalization and feature representation, combined with the depth information and color information obtained by the RGB-D camera, to achieve high-precision object instance segmentation. The present invention combines the image segmentation and feature extraction visual basic model to achieve the recognition and segmentation of new objects without additional training; at the same time, the present invention improves the recognition ability of key object areas in the image through a multi-layer attention mechanism and a feature weighting method based on information entropy; and the present invention designs a mask optimization method based on the background similarity matrix, which effectively improves the accuracy of the object instance segmentation results. These technical innovations improve the generalization ability of the unseen object instance segmentation model and provide reliable technical support for the autonomous operation of robots in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A diagram of a zero-sample object instance segmentation system in a robot environment of the present invention;

[0031] Figure 2 A flowchart of a zero-sample object instance segmentation method in a robot environment of the present invention;

[0032] Figure 3 It is an overall network structure diagram of the zero-sample object instance segmentation method in the robot environment of the present invention;

[0033] Figure 4 This is a visualization example diagram of the segmentation results of the zero-sample object instance segmentation method in the robot environment of the present invention on the OCID dataset;

[0034] Figure 5 This is a diagram of the zero-sample object instance segmentation method in a robot environment of the present invention being applied to a robot physical object grasping test process. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in combination with specific implementation methods and with reference to the accompanying drawings. In the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0036] The present invention provides a zero-sample object instance segmentation system and method in a robot environment.

[0037] Embodiment 1

[0038] The zero-sample object instance segmentation system in the robot environment of the present invention is referred to in the specific arrangement Figure 1 It can be seen that the zero-sample object instance segmentation system in the robot environment includes an image acquisition module, a processing module, an initial segmentation module and a precise segmentation module which are connected in sequence.

[0039] Furthermore, in the above-mentioned image acquisition module, the robot uses an RGB-D camera to acquire the RGB image and depth image of the current scene for subsequent processing.

[0040] The above processing module maps the depth image collected by the image acquisition module to the RGB color space.

[0041] The above initial segmentation module inputs the processed colorized depth image into the image segmentation visual basic model to generate a variety of object mask candidates. These masks include foreground objects, background areas, and partial noise masks, providing candidate areas for subsequent precise segmentation.

[0042] The above-mentioned precise segmentation module uses the pre-trained ViT extraction image acquisition module to obtain the attention map of the image to effectively capture the structure and semantic information of the objects in the image, and then calculates the information entropy of the attention map of each attention head, constructs a feature weighting mechanism to highlight the key object area, and then calculates the background similarity matrix, screens the object mask area and samples it, and finally inputs the representative points obtained by sampling into the SAM model as prompt information to achieve precise segmentation of object instances.

[0043] Embodiment 2

[0044] Embodiment 2 is a segmentation method using the segmentation system in embodiment 1, such as Figure 2 As shown in the figure, the following steps are included: first, the depth image is colorized and then input into the segmentation model to obtain the mask candidates that are unknown to the object; at the same time, the image is input into the description model for feature extraction. These extracted features can effectively capture the semantic information of the object in the image, which helps to filter out the non-object mask in the segmentation result of the segmentation model, thereby obtaining the initial segmentation result. Subsequently, the K-Medoids clustering algorithm is applied to extract positive sample points in the filtered object mask area, and these points are used to further optimize the initial segmentation result. The final output segmentation result has a high object boundary segmentation accuracy, and this method can achieve the segmentation of unseen objects without additional training.

[0045] refer to Figure 3 It can be seen that the segmentation method in this embodiment specifically includes the following steps:

[0046] S1. Obtain the color and depth information image of the current scene.

[0047] S2. Depth image preprocessing: the depth image is mapped to RGB color space using the Viridis color mapping method.

[0048] S3. Generate an initial mask, input the preprocessed colorized depth image into the SAM model, set the non-maximum suppression threshold to a lower value (such as 0.5), and generate a variety of object mask candidates. These masks include foreground objects, background areas, and partial noise masks, providing candidate areas for subsequent precise segmentation.

[0049] S4. Feature extraction: Use the pre-trained ViT as a feature description model for scene representation to process the image and extract the attention map between the category label (CLS) of each attention head and all image patch labels. These attention maps reflect the semantically meaningful visual patterns formed by the ViT model during the self-supervised learning process, and can effectively capture the structure and semantic information of objects in the image.

[0050] S5. Feature weighting: Based on the extracted attention map, the information entropy of each image region is calculated, and a feature weighting mechanism is constructed. This mechanism assigns weights according to the information richness of the region, so that regions with significant visual features obtain higher weight values, which helps to highlight key object regions.

[0051] S6. Calculate the background similarity matrix, construct the similarity matrix between the image block and the predefined background block, and use cosine similarity to measure the degree of association between different regions. By calculating the similarity matrix between the background block and all image blocks, it is determined whether a block belongs to the background.

[0052] S7, region sampling, in the object mask region after screening, the K-Medoids clustering algorithm is applied for sampling. The above K-Medoids clustering algorithm finds the most representative set of sample points through iterative optimization, and these points can effectively represent the spatial distribution of the object.

[0053] S8, final segmentation, the representative points obtained by sampling are input into the SAM model as prompt information, and the model's accurate segmentation ability is used to generate improved object instance segmentation results. This step pays special attention to the accuracy of the edge area of ​​the object, and improves the fineness of the segmentation through a multi-point prompt strategy.

[0054] The segmentation method of the present invention is verified on the OCID dataset which is widely used in the instance segmentation method of unseen objects in indoor environments. In this embodiment, the segmentation method adopts two main visual basic models: the SAM model for segmentation and the feature description model ViT for scene representation. Among them, the SAM model adopts the ViT-H / 16 backbone network, and sets a number of key parameters to ensure the best segmentation performance: the bounding box non-maximum suppression threshold is set to 0.5 to control the overlap tolerance between the predicted bounding boxes. When the value is higher, more overlapping prediction results are allowed to be retained; the image crop overlap ratio is set to 0, indicating that there is no overlapping area between adjacent image crops; the minimum mask area is set to 0, allowing the detection of segmented areas of any size; the number of prompt points processed in each batch is set to 64; the stability score threshold is set to 0.95 to ensure that only highly reliable segmentation results are retained.

[0055] In terms of feature description model, this embodiment adopts ViT-B / 14 architecture and uses DINOv2 pre-trained weights. In the mask filtering stage, the background similarity threshold τ of this embodiment is set to 0.47. In the point prompt generation stage, this embodiment applies K-Medoids clustering algorithm to each target candidate area to generate 3 point prompts.

[0056] The calculation steps for a set of RGB-D image tests are as follows:

[0057] Input: preprocessed RGB image and depth image;

[0058] Step 1: This method maps the depth image to RGB color space, specifically using the viridis color mapping scheme. The converted depth map is used as the input for the segmentation task. This process helps to extract complete object mask candidates without relying on color and texture information.

[0059] Step 2: In the segmentation stage, given a colorized depth image The SAM model was processed using the default settings, but the NMS threshold was set to a lower value of 0.5 to generate N m There are mask candidates M={m1,m2,...,m Nm}, where for i = 1, 2, ..., N m , there are m i ∈{0,1} H×W .

[0060] Step 3: Process the input RGB image through ViT trained with DINOv2 ViT divides the image into K×K non-overlapping image blocks, with a total number of N p =HW / K 2 In the last layer of ViT, there are N h Attention heads, each attention head i will generate an attention map Represents the attention weight between the category label (CLS) and all image patch labels. The key feature representation of the last layer is denoted as Where D represents the dimension of the feature vector.

[0061] Step 4: Attention map a of different attention heads based on information entropy i For each attention map, its information entropy is calculated by the formula Calculated, where P(·) represents the normalization operation. The weight is calculated by the formula Calculation ensures that attention heads with lower entropy values ​​receive higher weights.

[0062] Step 5: Calculate the similarity matrix between the background block and all image blocks to determine whether a block belongs to the background. The index of the background block is calculated by the formula OK. Weighted feature F ω By element-wise multiplication F ω =W⊙F is obtained and reshaped into The final cosine similarity matrix S between the background block and all other image blocks cosIt is used to calculate the score of each image block as the background. When the average score of the image block corresponding to the mask exceeds the preset threshold τ, the mask is identified as the background and removed. After removing the non-object mask, the object mask set M is obtained. o .

[0063] Step 6: In each candidate object mask region M o The cluster centers of the K-Medoids clustering algorithm are used as positive sample points. The obtained point hints are used to guide SAM to perform segmentation, thereby obtaining improved segmentation results.

[0064] Output: instance segmentation results of objects in the image.

[0065] The segmentation method in the present invention has good performance in the field of indoor scene object instance segmentation. First, a verification test was carried out on the widely used OCID dataset, such as Figure 4 The segmentation results shown indicate that the segmentation method of the present invention can fully utilize the zero-sample recognition capability and powerful feature representation capability of the visual basic model to effectively segment the target objects in the RGB-D image.

[0066] In order to further verify the feasibility of the method in actual application scenarios, a field test was conducted using the Fetch mobile robot. During the test, the scene image was first captured by the RGB-D camera on the robot's head, and then the segmentation method proposed in this invention was used to segment the objects in the image. After the object segmentation was completed, the system called Contact-GraspNet to perform grasping posture planning, and MoveIt performed specific motion planning. Figure 5 As shown in the figure, the experiment completely records the whole process of the robot from target recognition, object segmentation to final successful grasping, which fully proves the effectiveness of this method in practical applications.

[0067] In the above embodiments, the present invention provides a zero-sample object instance segmentation system and method in a robot environment, in which the powerful capabilities of the visual base model in zero-sample generalization and feature representation can be fully utilized to achieve zero-sample instance segmentation of objects in an indoor robot environment without additional training. By using the explicit visual representation of the pre-trained ViT, the present invention effectively removes the mask of non-objects, thereby significantly improving the accuracy of segmentation.

[0068] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A zero-shot object instance segmentation system in a robotic environment, characterized by: include Image acquisition module: the robot uses an RGB-D camera to acquire the RGB image and depth image of the current scene for subsequent processing; A processing module maps the depth image collected by the image acquisition module to an RGB color space; The initial segmentation module inputs the processed colorized depth image into the image segmentation visual basic model to generate a variety of object mask candidates; The precise segmentation module uses the pre-trained ViT extraction image acquisition module to obtain the attention map of the image, then calculates the information entropy of the attention map of each attention head, constructs a feature weighting mechanism, and then calculates the background similarity matrix to screen the object mask area and sample it. Finally, the representative points obtained by sampling are input into the SAM model as prompt information to achieve precise segmentation of object instances.

2. The zero-sample object instance segmentation system in a robot environment according to claim 1, characterized in that: The mask includes a foreground object, a background area, and a partial noise mask.

3. A zero-shot object instance segmentation method in a robot environment, characterized by: Using the segmentation system according to any one of claims 1 to 2 comprises the following steps: S1, obtain the color and depth information image of the current scene; S2, using Viridis color mapping method to map the depth image to RGB color space; S3, input the preprocessed colorized depth image into the SAM model, set the non-maximum suppression threshold to a low value, and generate diverse object mask candidates; S4, using the pre-trained ViT as the feature description model for scene representation to process the image and extract the attention map between the category label (CLS) of each attention head and the labels of all image patches; S5. Attention map of different attention heads based on information entropy a i weighting; S6, calculating the similarity matrix between the background block and all image blocks to determine whether a block belongs to the background; S7, applying K-Medoids clustering algorithm to sample in the filtered object mask area; S8. The representative points obtained by sampling are input into the SAM model as prompt information, and the accurate segmentation ability of the model is used to generate improved object instance segmentation results.

4. The method for zero-sample object instance segmentation in a robot environment according to claim 3, characterized in that: The step S3 is specifically as follows: in the segmentation stage, given a colorized depth image The SAM model with default settings was used for processing, and the non-maximum suppression threshold was set to 0.5 to generate N m Mask candidates that are independent of object category where for i=1,2,...,N m , there are m i ∈{0,1} H×W .

5. The method for zero-sample object instance segmentation in a robot environment according to claim 4, characterized in that: Specifically, step S4 is to process the input RGB image using ViT trained by DINOv2 as a feature description model for scene representation. ViT divides the image into K×K non-overlapping image blocks, with a total number of N p =HW / K 2 , in the last layer of ViT, there are N h Attention heads, each attention head i will generate an attention map represents the attention weight between the category label and all image patch labels, and the key feature representation of the last layer is recorded as Where D represents the dimension of the feature vector.

6. The method for zero-sample object instance segmentation in a robot environment according to claim 5, characterized in that: The step S5 is specifically as follows: for each attention map a i , whose information entropy is expressed by the formula Calculation, where P(·) represents the normalization operation; the weight is calculated by the formula Calculate so that the attention heads with low entropy values ​​get higher weights.

7. The method for zero-sample object instance segmentation in a robot environment according to claim 6, characterized in that: The index of the background block in step S6 is expressed by the formula Determine; weighted feature F ω By element-wise multiplication F ω =W⊙F is obtained and reshaped into The final cosine similarity matrix between the background block features and all other image block features Used to indicate the score of each image patch as background; When the average score of the image block corresponding to the mask exceeds the preset threshold τ, the mask is identified as the background and removed. After removing the non-object mask, the object mask set M is obtained. o .

8. The method for zero-sample object instance segmentation in a robot environment according to claim 7, characterized in that: In step S7, in each candidate object mask area M o The cluster center of the K-Medoids clustering algorithm is used as the positive sample point.

9. The method for zero-sample object instance segmentation in a robot environment according to claim 8, characterized in that: The step S8 uses the positive sample point prompts obtained in S7 to guide SAM to perform segmentation, thereby obtaining an improved segmentation result.

Images