Method for providing gripping points for mechanical interactions on object

Abstract

The invention relates to a method (100) for providing a capture point for a mechanical interaction on an object (1), comprising the following steps:-providing (101) at least one image of the object (1), the image being a result acquired by at least one sensor (2),-providing (102) an object cue on the basis of the provided image by using an object detector and / or an unsupervised or semi-supervised segmentation method, -performing (103) a segmentation on the provided image using the provided object cues to generate an object mask representing the object (1),-performing (104) a morphological skeletonization in the provided image based on the generated object mask to obtain a polyline,-reducing (105) the polyline to a defined number of points, and-generating (105) the object (1). -selecting (106) at least one point to provide a point of gripping for the mechanical interaction on the object (1). The invention relates to a computer program, a device and a storage medium for this purpose.

Description

TECHNICAL FIELD

[0001] The invention relates to a method of providing a grasp point for a mechanical interaction on an object. Furthermore, the invention relates to a computer program, a device and a storage medium for this purpose. BACKGROUND

[0002] Finding a grasp point on an object surface with an arbitrary shape is a challenging task for manipulating objects, even if they are stationary. The variety of object appearances and the coordination of visual recognition and machine controller capable of grasping objects is challenging, for example in the case of an automatic unscrambler.

[0003] Arbitrary object recognition - especially from flexible shapes, surfaces and colors of the content surface of a carton box during unscrambling - often leads to too many unpredictable situations in designing and manufacturing a highly reliable unscrambler. Furthermore, grasping or manipulating arbitrary objects from a box or a conveyor requires locating critical points on the objects to interact with them using robot or machine controller functionality, where safety and robustness are a priority.

[0004] KR20130032990 A discloses a method for generating a grasp point by range recognition and provides a computer-readable recording medium including a program for the method to generate a grasp point by recognizing objects within the same range to recognize and grasp various shapes of objects within the same range. SUMMARY

[0005] According to an aspect of the invention, a method having the features of claim 1, a computer program having the features of claim 8, a data processing device having the features of claim 9 and a computer-readable storage medium having the features of claim 10 are provided. Further features and details of the invention are disclosed in the respective dependent claims, the description and the drawings. Features and details described in the context of the inventive method also correspond to the inventive computer program, the inventive data processing device and the inventive computer-readable storage medium, and vice versa in each case.

[0006] According to an aspect of the invention, a method for providing a grasp point for a mechanical interaction on an object is provided, the method comprising the following steps, wherein the steps can be executed repeatedly and / or in a specific order. The grasp point can be a point at which a robot arm can grasp the object to move it. The object can be an object with an irregular or arbitrary shape like for example a crumpled paper wrapped in a carton box.

[0007] In a first possible step, at least one image of the object is provided, the image being a result of an acquisition by at least one sensor. The sensor can be a camera sensor.

[0008] In a next possible step, object hints are provided based on the provided image by utilizing an object detector and / or unsupervised or semi-supervised segmentation methods. The unsupervised or semi-supervised segmentation methods used can be parameterized to leave as large object boundary edges as possible, as the object hints can be used as a region of interest window for a Segmentation Anywhere Model (SAM), which can pre-determine its pixel search space for the segmentation mask(s).

[0009] Examples for object detectors can be Convolutional Neural Networks, YOLO (You Only Look Once) or Single Shot MultiBox Detector (SSD). Unsupervised segmentation methods can include k-means clustering, which is a method that groups similar data points into clusters based on the distance between them, and Mean Shift, which is a method that identifies clusters by shifting a window over the data and finding the mode of the data points within the window. Semi-supervised segmentation methods can include Graph-based Segmentation, which is a method that uses a graph to represent the image and assigns labels to nodes based on a small set of labeled pixels, and Active Contour Models, which is a method that uses curves or surfaces to delineate objects of interest and is guided by a combination of image-based and user-defined constraints.

[0010] In a next possible step, segmentation is performed on the provided image using the provided object hints to generate object masks representing the objects. The segmentation can be zero-shot segmentation, in particular zero-shot learning semantic segmentation. The segmentation can be performed by a Segmentation Anything Model.

[0011] In a next possible step, morphological skeletonization is performed in the provided image based on the generated object masks to obtain a skeleton. Morphological skeletonization in an image is in particular a process that reduces an image to its essential structure, which can represent the geometric and topological properties of the original image. It can involve removing pixels from the image while preserving the connectivity of the remaining pixels, resulting in a simplified image that retains the important features of the original image.

[0012] In a next possible step, the polyline is reduced to a limited number of points. In this step, unnecessary points can be removed from the polyline to obtain a simplified representation of the object surface topology. Furthermore, the complexity of the polyline can be reduced by removing redundant or noisy points while preserving its essential features. Then, relevant features such as end points, length or curvature can be identified and extracted from the polyline, so that optimal grasping points can then be determined. It is possible to perform the reduction of the polyline to a limited number of points by using the Ramer-Douglas-Peucker algorithm. This can enable a more efficient point reduction process, as the Ramer-Douglas-Peucker algorithm can effectively remove unnecessary points from the polyline while preserving its essential features. This can result in a more robust and reliable grasping point selection process, ultimately facilitating precise mechanical interaction with the object.

[0013] In a next possible step, at least one point is selected to provide a grasping point for the mechanical interaction on the object. The selected point can be the point closest to the centroid of the object in the image.

[0014] In another possible step, the robotic arm can be initiated to grasp the object at the provided grasping point to move the object.

[0015] The method according to the present invention, i.e. the computer vision pipeline, can effectively determine grasping points on surfaces, in particular on surfaces with irregular structures, for mechanical interaction by exploiting a zero-shot learning semantic segmentation mask that can be used to propose grasping points without the need for large amounts of training data. The method according to the present invention can exploit the ability of object cues and transformer-based encoding to allow for efficient image understanding and object localization, ultimately facilitating the generation of polylines and the subsequent grasping point selection. The method according to the present invention can advantageously allow for the generation of object cues and image as well as cue encoding without the need for labeled training data, thus enabling a more efficient and effective method for determining grasping points on surfaces.

[0016] Providing the object cue can comprise the following steps:

[0017] - providing a bounding box of the object as an object proposal,

[0018] - inferring bounding box predictions by an object detector based on the bounding box,

[0019] - providing the object cue based on the inferred bounding box predictions.

[0020] Thus, providing object hints can comprise the additional step where bounding boxes of objects are generated as object proposals. In other words, these bounding boxes can be used to infer bounding box predictions by an object detector, and subsequently, object hints can be provided based on these inferred bounding box predictions. This can enable the method according to the present application to generate representative object hints.

[0021] Providing object hints can further comprise the steps of:

[0022] - providing segmentation patches as object proposals,

[0023] - providing object hints using an unsupervised segmentation method based on segmentation patches.

[0024] Segmentation patches can refer to small local regions within an image that have been identified and isolated. These patches can be used to improve accuracy by focusing on specific areas of interest within an image. This substitution can enable the method according to the present application to generate meaningful object hints without requiring any prior knowledge about the class or category of the objects.

[0025] Performing segmentation can comprise the steps of:

[0026] - encoding the image by a visual transformer neural network architecture,

[0027] - encoding the object hints into a latent space,

[0028] - querying the encoded image with the encoded object hints,

[0029] - decoding a mask for each object on the image.

[0030] Thus, the segmentation step can involve encoding the image using a visual transformer neural network architecture to generate a representation of the image. The visual transformer neural network architecture can be a deep learning model designed to process visual data such as images and videos. It is particularly based on a transformer architecture that was originally developed for natural language processing tasks. The visual transformer can comprise a series of self-attention layers that can allow the visual transformer to selectively focus on different parts of the input data. This can allow the visual transformer to better understand the relationships between different visual features and make more accurate predictions. This encoded image can then be used in conjunction with encoded object hints that are transformed into a latent space. The encoded image and hints can be queried together to retrieve a mask for each object on the image, enabling the generation of accurate semantic segmentation masks for proposing grasping points for mechanical interactions. For each object, multiple valid masks can be generated and ranked by a confidence score, and the highest confidence can be selected as the output for that object.

[0031] Performing morphological skeletonization comprises the steps of:

[0032] - constructing break lines based on the generated object mask,

[0033] - removing intersections and / or break line loops and / or circles from the break lines,

[0034] - identifying end points and lengths of the object to obtain positions near the center of mass of the object as grasping points for mechanical interaction.

[0035] In other words, the morphological skeletonization step in the pipeline according to the present application can involve constructing break lines based on the generated object mask, followed by removing intersections and / or break line loops and / or circles from the break lines to create a simplified representation of the object shape. This simplification can aim at identifying end points and lengths of the object to obtain positions near the center of mass of the object as grasping points for mechanical interaction. Intersections and / or break line loops and / or circles of the break lines can be identified, for example, with a Bentley-Ottoman algorithm.

[0036] In another aspect of the present application, a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the present application can be provided. Thus, the computer program according to the present application can have the same advantages as described in detail with reference to the method according to the present application.

[0037] In another aspect of the present application, an apparatus for data processing configured to carry out the method according to the present application can be provided. As the apparatus, for example, a computer executing the computer program according to the present application can be provided. The computer can comprise at least one processor which can be used to execute the computer program. Furthermore, a non-volatile data storage can be provided, in which the computer program can be stored and from which the processor can read the computer program for execution.

[0038] According to another aspect of the present application, a computer-readable storage medium comprising the computer program according to the present application and / or instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to the present application can be provided. The storage medium can be formed as a data storage device, for example, a hard disk and / or a non-volatile memory and / or a memory card and / or a solid state drive. For example, the storage medium can be integrated into a computer.

[0039] Furthermore, the method according to the present application can be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or automated. BRIEF DESCRIPTION OF DRAWINGS

[0040] Further advantages, features and details of the present application will become apparent from the following description, in which embodiments of the present application are described in more detail with reference to the drawings. In this context, features mentioned in the claims and the description can be essential to the application, either individually or in any combination.

[0041] Figure 1 A method, an object, a sensor, a computer program, a storage medium and a device according to embodiments of the present application are shown;

[0042] Figure 2 A method according to embodiments of the present application is shown. DETAILED DESCRIPTION

[0043] Figure 1 A method 100, an object 1, a sensor 2, a computer program 20, a storage medium 15 and a device 10 according to embodiments of the present application are shown.

[0044] Figure 1 Embodiments of a method 100 for providing a grasp point for a mechanical interaction on an object 1 are specifically shown. In a first step 101, at least one image of the object 1 is provided, which is a result of a capturing by at least one sensor 2. In a second step 102, an object hint is provided based on the provided image by utilizing an object detector and / or an unsupervised or semi-supervised segmentation method. In a third step 103, a segmentation is performed on the provided image using the provided object hint to generate an object mask representing the object 1. In a fourth step 104, a morphological skeletonization is performed in the provided image based on the generated object mask to obtain a skeleton line. In a fifth step 105, the skeleton line is reduced to a limited number of points. In a sixth step 106, at least one point is selected to provide a grasp point for a mechanical interaction on the object 1.

[0045] The present application according to embodiments provides a computer vision pipeline that uses zero-shot learning semantic segmentation masks to detect points on a surface that can be used as proposals for grasp points for a mechanical interaction at inference time of a machine learning model.

[0046] The computer vision pipeline according to embodiments of the present application can comprise generating an object hint that can be derived from either a bounding box based or a segmentation mask based object proposal. Thereafter, a transformer based image and hint encoding, a segmentation mask decoder, a morphological skeletonization and a post-processing operation on the skeletonized line points on the surface can be applied to obtain grasp points for a mechanical interaction.

[0047] Knowing a priori that n types of objects and k types of content may appear in an image based on object content recognition, objects in the image can be segmented using unsupervised segmentation methods. According to embodiments of the present invention, this capability is extended to include unsupervised segmentation for object cue encoding, in addition to bounding box cue encoding.

[0048] Below, for reference Figure 2 An embodiment of method 200 is described, namely the computer vision pipeline steps according to the present invention.

[0049] In the first step 201, an image is provided. Then, object cues can be provided according to step 202, specifically object cues are generated. In this first alternative, bounding boxes can be used as object proposals. Bounding box predictions can then be inferred on the image from a pre-trained object detector (e.g., YOLOv5), and all object predictions can be collected. As shown in step 203, these predictions can give corresponding bounding boxes. The corresponding bounding boxes can also be described with a category name and confidence level. This prediction can be used as an object cue later. In this second alternative ( Figure 2 (Not shown in the image), unsupervised segmentation patches can be used as object proposals. If k classes are mutually distinguishable on an image, and it is known that no other classes can appear on the image (especially if it is not an open-world use case), then unsupervised or semi-supervised segmentation methods can be used to generate object cues. The unsupervised or semi-supervised segmentation method used can be parameterized to leave the largest possible object boundary edges, because the object cues can be used as a region of interest window for an arbitrary segmentation model (SAM), whose pixel search space can be pre-determined for one or more segmentation masks.

[0050] In step 205, zero-shot semantic segmentation can be performed as a task of any object segmentation model (SAM). Here, the original entire image (or a magnified image, if needed) can be inferred and encoded by a visual converter neural network architecture. Then, according to step 204, the saved object cues (specifically, all object cues in the image) can be encoded into the latent space. The encoded image can then be queried using the object cues. Then, according to step 206, a mask can be decoded for each object in the image. For each object, multiple valid masks can be generated and sorted according to confidence scores, and the highest confidence score can be selected as the output for that object.

[0051] In a next step 207, morphological skeletonization of the objects on the image can be performed. In a first part of this step, thinning lines can be constructed based on the object masks. For each segmentation mask representing an object, morphological skeletonization can be performed, wherein erosion and dilation with a 3x3 pixel structuring element (cross) can be utilized. In a second part of this step, intersections and line loops / circles can be removed from the line skeleton topology. These can be identified, for example, with the Bentley-Ottoman algorithm.

[0052] In a next step 208, morphological post-processing can be performed on the line to obtain grasp points. Here, the line endpoints and length can be used to obtain a position near the object centroid, which can then be used as a grasp point for mechanical interaction. However, the latter approach can not always be suitable for every shape and surface, and therefore, for example, the Ramer-Douglas-Peucker algorithm can be used to reduce the line to n defined points, which define an approximated line segment of the original skeletonized line (a possible grasp point plus the number of endpoints, thus n = 3 or more). The resulting line can comprise a reduced number of line segments. In this way, the line can resemble the original line and its topology can remain unchanged.

[0053] In a next step 209, the reduced line with proposed grasp points can be provided.

[0054] The above explanations of embodiments describe the application in the context of examples. Of course, the individual features of the embodiments can be freely combined with each other, as far as this is technically reasonable, without departing from the scope of the application.

Claims

1. A method (100) for providing a grasp point for a mechanical interaction on an object (1), comprising the steps of: - providing (101) at least one image of the object (1), the image being a result of a capturing by at least one sensor (2), - providing (102) an object proposal based on the provided image by utilizing an object detector and / or an unsupervised or semi-supervised segmentation method, - performing (103) a segmentation on the provided image using the provided object proposal to generate an object mask representing the object (1), - performing (104) a morphological skeletonization in the provided image based on the generated object mask to obtain a skeleton, wherein performing (104) the morphological skeletonization comprises the steps of: o constructing the skeleton based on the generated object mask, o removing intersections and / or skeleton loops and / or circles from the skeleton, o identifying end points and lengths of the object (1) to obtain a position near the center of mass of the object (1) as a grasp point for the mechanical interaction, - reducing (105) the skeleton to a limited number of points, - selecting (106) at least one point to provide a grasp point for a mechanical interaction on the object (1).

2. The method (100) according to claim 1, characterized in that Providing (102) the object proposal comprises the steps of: - providing a bounding box of the object (1) as the object proposal, - inferring a bounding box prediction based on the bounding box by an object detector, - providing the object proposal based on the inferred bounding box prediction.

3. The method (100) according to claim 1, characterized in that Providing (102) the object proposal comprises the steps of: - providing a segmentation patch as the object proposal, - providing the object proposal using an unsupervised segmentation method based on the segmentation patch.

4. The method (100) according to any one of the preceding claims, characterized in that, Performing (103) the segmentation comprises the steps of: - encoding the image by a visual transformer neural network architecture, - encoding the object proposal into a latent space, - querying the encoded image with the encoded object proposal, - decoding a mask of each object (1) on the image.

5. The method (100) according to any one of the preceding claims, characterized by, Reducing (105) the skeleton to a limited number of points is performed by using a Ramer-Douglas-Peucker algorithm.

6. The method (100) according to any one of the preceding claims, characterized by, The segmentation is a zero-shot segmentation during performing (103).

7. A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) of any one of the preceding claims.

8. A data processing device (10) comprising means for carrying out the method (100) of any one of claims 1 to 6.

9. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) of any one of claims 1 to 6.

Images