Method for determining a gripping pose for a robot

Abstract

The invention relates to a method for determining a gripping pose for a robot, wherein reference points (404.1, 404.2, 404.3) at which an object to be gripped can be gripped are specified on a reference object (402), wherein gripping preference information (704.1, 704.2, 704.3) is ascertained by means of an image captured by a sensor (124) and each piece of gripping preference information comprises a probability distribution which indicates how precisely a (gripping) position matches a corresponding reference point (404.1, 404.2, 404.3). In particular, the method comprises: providing (304) gripping specification information (302); providing (310) end effector geometry information (312); providing (324) surroundings information (322); determining (330) gripping preference information (322) which comprises a probability distribution; determining (340) a gripping pose (342) to be used for the end effector.

Description

[0001] Method for determining a gripping pose for a robot

[0002] Description

[0003] The present invention relates to a method for determining a gripping pose for a robot, for use in determining a movement sequence for the robot to grip and / or move one or more objects in a work environment by means of an end effector of the robot, wherein the end effector comprises at least two gripping means, a computing unit and a computer program for its execution, and a robot.

[0004] Background of the invention

[0005] Robots, often also referred to as kinematics, can be used in production facilities, logistics, and other systems to, for example, transport or assemble parts. Typical types of such robots include Cartesian robots, SCARA robots, and articulated robots.

[0006] Disclosure of the invention

[0007] According to the invention, a method for determining a gripping pose for a robot, a computing unit and a computer program for its execution, as well as a robot with the features of the independent claims, are proposed. Advantageous embodiments are the subject of the dependent claims and the following description.

[0008] The invention relates generally to robots that can be used, for example, in production plants, logistics, or other facilities. Types of such robots, often also referred to as kinematics, include, for example, Cartesian robots, SCARA robots, and articulated robots. Such robots can be used to grasp and / or move objects (i.e., items or parts). A typical application is to remove an object from a box containing, for example, a large number of (identical or different) objects and, for example, place it somewhere or possibly assemble it. For this purpose, such a robot has, for example, a gripper or, more generally, an end effector. Such robots can also be referred to as manipulators or manipulation devices; the grasping, possibly involving movement, can also be described as manipulation.

[0009] In order for the robot to grasp and / or move an object in a work environment, e.g. in a box, or in or on other carriers (e.g. a container or a pallet) using the end effector, the robot, in particular its end effector, must perform a movement sequence to move the end effector to the position of the object and then, e.g. after the object has been grasped, to move the end effector away from the position of the object again, e.g. to a desired storage position.

[0010] For a successful grip, the exact pose for the gripper – hereinafter also referred to as the gripping pose – must first be determined. This requires capturing and providing environmental information, such as an image and / or a point cloud of the work environment (scene), particularly of any existing support structure, such as a box. Sensors like cameras and / or lidar sensors (laser scanners) or other depth sensors can be used for this purpose. The environmental information then includes, in particular, one or possibly several objects located within the work environment.

[0011] Based on environmental information, such as the image, a grasping pose can be determined. Based on this, the robot's movement sequence can be determined, for example, to generate control information for moving the robot, which can then be implemented via a control unit or other processing unit.

[0012] Another aspect considered within the scope of the present invention is the use of robots in which the end effector has at least two (i.e., two or more, e.g., three, four, or five) gripping means. Suitable gripping means include, for example, suction cups, which are preferably all applied simultaneously to an object (in the gripping pose of the end effector) and can grasp and thus move the object by generating a vacuum. Finding the most optimal gripping pose can be problematic in this context. This is particularly true if the individual gripping means differ from one another, e.g., with regard to their respective geometric dimensions of the gripping area (so-called "footprint") and / or their respective material (from which they are made) and / or their respective gripping force. As has been shown, with such an end effector having at least two gripping means, it is also possible to use not just one, but several (i.e.,to grasp two or even three or more objects at the same time, at least if they are suitably shaped and suitably positioned objects.

[0013] Against this background, it is proposed that grasping specification information be provided. This grasping specification information includes specifications for one or more reference points on one of the multiple reference objects, indicating at which reference points the one or more reference objects are to be grasped. Such specifications can, for example, be created in advance by a user. In the case of a single reference point, this is located on one reference object; in the case of multiple reference points, these can all be located on one reference object, or they can be located on the reference points of the multiple reference objects. The latter can mean that there is a reference point on each reference object, but it is also conceivable that at least one of the multiple reference objects has two or more reference points.

[0014] A reference object can be a comparable object to the object to be grasped, allowing the user to specify points on the object where – at least in the user's opinion – the object can be easily grasped using the end effector. Each reference point could correspond to one of at least two grasping means, but it's also possible to specify more reference points than grasping means, or even fewer reference points than grasping means. If there are multiple reference objects, they can be of the same type, but it's also possible for them to be of different types.

[0015] A concrete example of this is when the user marks the reference points on an image of the reference object (this is referred to as annotation). A point can correspond, for example, to a pixel in the image or to a group of (e.g., contiguous) pixels (i.e., individual image points). The grasping specification information can therefore correspond, for example, to an image of the reference object in which certain pixels or image points are assigned the information that the object or reference object should or can be grasped there. It is also conceivable that certain pixels or image points (or reference points in general) are assigned the information that the object or reference object should not be grasped there; this might be the case, for example, if it is known that the object in question could be easily damaged there. For more details on the specific implementation, see, for example, DE 10 2022 206 274 A1.

[0016] In practical implementation, descriptors can be calculated for these pixels or image points, which can then serve as input specifications. A machine learning model can be used for this purpose, for example. For more details on the specific implementation, see, for example, DE 10 2021 212 860 A1.

[0017] Furthermore, end-effector geometry information is provided. This information includes details about the geometric dimensions of the gripping area of ​​at least two gripping devices. This can be the so-called footprint of the end effector or the at least two gripping devices, for example, a 2D projection. In the example of three suction cups as gripping devices, the footprint could, for example, correspond to three circular areas.

[0018] Based on the environmental information (e.g., a current image) and the grasping specification information, grasping preference information is then determined. This grasping preference information comprises a probability distribution for one or more areas on the object(s) to be grasped. Such a probability distribution indicates how closely a position within the respective area corresponds to a corresponding reference point. For example, the probability distribution can specify a value for each or at least some positions (again, these could be pixels in the image example), with the value decreasing the further the position is from the reference point.The reference points can be identified in the environment information by matching one or more objects in the current image with the one or more reference objects, i.e., by bringing them into alignment. In practice, descriptors can also be calculated for these pixels or image points. It should be noted that multiple (accessible) objects can each be matched with a single reference object; this is particularly possible if the multiple objects are of the same type (as the reference object).

[0019] Such a probability distribution can also be referred to as a heat map. For further details on the concrete implementation of a heat map, see, for example, DE 10 2022 206 274 A1. It is conceivable that for each reference point identified in the environmental information, such a probability distribution with a range is determined; once all probability distributions have been determined, they can be combined to obtain the grasping preference information.

[0020] Based on the gripping preference information and the end effector geometry information, the gripping pose to be used for the end effector is then determined. This gripping pose includes, in particular, the specific poses for the individual gripping elements to grip the one or more objects as optimally as possible. It should be noted, however, that while it is advantageous for all available gripping elements to engage the one or more objects, it is also conceivable that not all, e.g., only two out of three gripping elements, engage the one or more objects.

[0021] In this process, the grasping preference information (e.g., the aforementioned heatmaps) is combined with the end effector geometry information (e.g., the aforementioned footprint), i.e., brought into agreement as closely as possible.

[0022] This can be achieved, for example, by optimizing the probability distribution values ​​corresponding to the grasping ranges. It's also conceivable that at least one sum of these probability distribution values ​​is greater than a predefined threshold. In this way, the most optimal grasping pose is found, taking into account specifications, such as user preferences. These user preferences, i.e., the reference points, don't necessarily have to represent the most optimal grasping poses themselves. A somewhat imprecise selection is permissible, as matching with the footprint will still lead to the most optimal grasping pose. The specifications or user preferences can, for example, be initially defined for a specific type of object within the context of a recurring grasping process.

[0023] In one embodiment, the gripping instruction information includes the specifications for one or at least one of the several locations on the one or more reference objects, specifying which of the at least two gripping means is to be used to grip the respective reference object at that location. This is particularly advantageous when the gripping means are different from one another, as it increases the probability of achieving the most optimal gripping pose, which then allows the one or more objects to be gripped securely.

[0024] In one embodiment, gripping quality information is also determined based on environmental information. This gripping quality information comprises a probability distribution for one or more areas on one or at least one of the multiple objects. Each probability distribution indicates how well the respective object can be gripped at a specific position within that area using the end effector. These areas can be, but do not have to be, the same as those used for gripping preference information. For example, the probability distribution for each or at least some positions (again, these can be pixels in the image example) can specify a value that increases with the probability of a successful grip at that location. This can be based, for example, on existing information (e.g.,Empirical data is used to determine how well certain positions on specific objects function as gripping poses. The gripping preference information is then further determined and improved based on the gripping quality information. In one embodiment, based on the gripping preference information and the end effector geometry information, the gripping pose to be used for the end effector is determined such that a predefined geometric relationship is fulfilled for the gripping areas of the at least two gripping means, in particular that the gripping areas lie in one plane. For this purpose, it is advantageous if the environment information and / or the gripping specification information also includes or is based on depth information or 3D information. For example, if...Since the gripping means are designed to engage the object in a single plane, the probability of a successful grasp is increased if the gripping pose is determined such that the gripping means actually engage the object in a single plane. This applies accordingly to other geometric relationships that are ultimately determined by the specific end effector and its gripping means. This also applies when multiple objects are to be grasped; for example, two objects should be positioned next to each other so that two sides, one from each object, lie in the same plane. However, shapes other than planes can also be considered here.

[0025] A computing unit according to the invention (i.e., generally a system for data processing), e.g., a control unit or a control unit of a robot, or a central server or other computing system, is, in particular in terms of programming, equipped to carry out a method according to the invention.

[0026] The invention also relates to a robot configured to receive control information as described above. In addition, or alternatively, the robot comprises a computing unit according to the invention. Furthermore, the robot comprises, in particular, a control unit and a drive unit for moving the robot. The robot may also have at least one sensor for acquiring environmental information, e.g., a camera and / or a lidar sensor.

[0027] Implementing a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, as this incurs particularly low costs, especially if an executing control unit is already available for other tasks. Finally, a machine-readable storage medium is provided with a computer program stored on it as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical storage media, such as hard drives, flash memory, EEPROMs, DVDs, etc. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0028] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.

[0029] The invention is schematically illustrated in the drawing using an exemplary embodiment and is described below with reference to the drawing.

[0030] Brief description of the drawings

[0031] Figure 1a schematically shows a robot to illustrate the invention.

[0032] Figure 1b schematically shows an end effector to illustrate the invention.

[0033] Figure 2 schematically shows a container to illustrate the invention.

[0034] Figure 3 schematically shows a process flow in one embodiment.

[0035] Figures 4, 5, 6, 7 and 8 schematically show details to explain the invention.

[0036] Figure 9 schematically shows another embodiment.

[0037] Embodiment(s) of the Invention Figure 1a schematically and by way of example shows a robot 100 to illustrate the invention. By way of example, the robot 100 has stand and arm components 102, 104, 106, so-called axes, which are each movably and movably connected by means of joints 112, 114.

[0038] Furthermore, the robot 100 has an end effector 108, e.g., a gripper. The end effector 108 is movably connected to the arm component 106 by means of a joint 116. The end effector 108 is shown only schematically in Figure 1a; in Figure 1b, the end effector 108 is shown in more detail. In particular, the end effector 108 has, by way of example, three gripping means 108.1, 108.2, 108.3, which can, for example, be designed as suction cups.

[0039] Furthermore, the robot 100 has a drive system 120, shown only schematically here, as well as a computing unit 122 designed, for example, as a control or regulation unit. This allows the drive system 120 to be controlled, for example, using control information, in order to move the robot according to a desired sequence of movements. This can include, for example, moving the axes relative to each other by means of the joints, but also rotating the axes themselves, provided that appropriate drives are available.

[0040] It should be noted that the robot 100 is only used here as an example for illustrative purposes. A robot for grasping and / or moving objects in containers can also be designed differently, for example, using an end effector that can only move linearly along several different rails.

[0041] Furthermore, a box 132 is shown in a work environment 130, containing, for example, an object 140. The robot 100 can now be controlled, for example, in such a way that it grasps and / or moves the object 140 using the end effector 108, in particular also taking it out of the box and, for example, placing it elsewhere.

[0042] Furthermore, an example sensor 124 is shown, which could be, for example, a camera or a lidar sensor. Both types of sensors can also be used. Likewise, several identical sensors can be used. As already mentioned, the use of a lidar sensor is particularly useful for obtaining a point cloud as environmental information. However, it is also conceivable to capture 3D and / or 2D images using a camera. It is also conceivable to capture only 2D images using a camera. The sensor 124 can, for example, be arranged in a suitable way in the work environment, e.g., on a ceiling and thus separately from the robot 100. The sensor 124 could, however, also be part of the robot and, for example, be arranged on the arm component 106 or the end effector 108.

[0043] The sensor 124 can now detect the working environment 130, and in particular the box 132 and its interior, including the object 140. Based on the environmental information and sensor data obtained in this way, a movement sequence can be created for the robot to grasp and / or move the object 140 using the end effector 108.

[0044] Figure 2 shows a box 232, comparable to box 132 in Figure 1, containing various objects 240, 241, 242, 243, and 244 as examples. A typical task for a robot is to remove as many objects as possible from the box. It is advantageous to first remove the easiest or safest object to grasp and so on, while adhering to certain guidelines, such as not covering forbidden areas with the end effector. It can also be seen that the objects can differ from one another, for example, being cuboid or cylindrical, and that the objects can also be randomly arranged.

[0045] An example movement sequence is shown for grasping object 240 and removing it from the box. This includes an approach path, or first part of the movement sequence 251, by which the end effector moves towards object 240 in order to grasp it. Additionally, a retraction path, or second part of the movement sequence 252, is shown, along which the end effector can be moved with object 240 out of the box.

[0046] Together, the first part of the movement sequence, 251, and the second part, 252, form a complete movement sequence. It can be seen that the movement sequence is such that the end effector must be in a specific pose—a grasping pose—in order to properly grasp object 240. It is also evident that not all grasping poses, which exist for the various objects or even just object 240, would be optimal. For example, object 240 should be grasped on its larger, flatter side, as this allows all three grasping elements to grip most effectively. This example also demonstrates that objects 240 and 243 could be grasped simultaneously if both object 240 and object 243 were lying flat on the bottom of the box and close to object 243.

[0047] The process of finding the most optimal gripping pose will be explained in more detail below.

[0048] Figure 3 schematically illustrates the sequence of a process in one embodiment. Reference is also made to Figures 4 to 8, which show a container or box and objects, and which serve to explain various steps in more detail.

[0049] In step 300, grasp specification information 302 is generated, which includes specifications for one or more reference locations on one or more reference objects, indicating at which reference locations the reference object(s) are to be grasped. This can, for example, include a user annotating an image of the reference object.

[0050] Figure 4 shows an example of such an image 400, in which a reference object 402 (shown here from above) is visible. Three reference points 404.1, 404.2, and 404.3 are shown as examples on a surface of the reference object (in the image). These reference points may, for example, have been marked by a user. They may, for example, correspond approximately to the three gripping elements of the end effector according to Figure 1b, although, as mentioned, precise positioning is not essential. Rather, it is advantageous if the reference points indicate at least approximately good—or otherwise preferred—locations for the gripping elements.

[0051] In step 304, the grasp specification information 302 obtained in this way can then be provided; this can mean, for example, that the grasp specification information 302 is stored in an executing computing unit, where it can then be provided or used as needed.

[0052] In step 310, end effector geometry information 312 is provided.

[0053] The end-effector geometry information includes information about the geometric dimensions of a gripping area of ​​at least two gripping devices. This can, for example, be a so-called footprint of the gripping devices.

[0054] Figure 5 illustrates such a Footprint 500, or rather, such end-effector geometry information. This is a projection of the gripping elements according to Figure 1b onto a plane. Corresponding to the three round gripping elements shown there, the Footprint 500 consists of three circular areas in the corresponding relation to each other.

[0055] In step 320, environmental information 322 is then acquired, e.g., using the sensor shown in Figure 1a. The environmental information 322 can, for example, be a 2D image, or the environmental information can correspond to a 2D image.

[0056] In step 324, the environment information 322 obtained in this way can then be provided; this can mean, for example, that the environment information 322 is made available in the executing computing unit for subsequent use.

[0057] Figure 6 shows such an image 600, or rather, such environmental information is depicted graphically. An object 602 can be seen there (here from above), which needs to be grasped.

[0058] In step 330, grasp preference information 332 is determined based on the environmental information 322 and the grasp specification information 302. The grasp preference information comprises a probability distribution for one or more areas of the object 604, where the probability distribution indicates how closely a position in the respective area corresponds to a corresponding reference point. As mentioned, this can be described as a heatmap. Figure 7 illustrates such a heatmap 700, or rather, such grasp preference information. Areas with a probability distribution are shown there (704.1, 704.2, 704.3); these are each represented as concentric rings. The closer a position is to the center of a set of such concentric rings (an area), the closer the position is to a position corresponding to the reference point.

[0059] In practical terms, this can involve determining, for example, the distance (or more generally a distance measure, e.g., the Euclidean norm) of the aforementioned descriptor at the position in image 600 to a corresponding descriptor at the reference position. For this purpose, it may be useful to match image 400 with image 600 to determine which locations or positions in image 600 correspond to the reference positions.

[0060] In this process, an individual heatmap can first be determined for each reference point, and all individual heatmaps are then combined to form Heatmap 700.

[0061] In step 340, based on the gripping preference information and the end effector geometry information, a suitable gripping pose 342 for the end effector is determined. Here, for example, an attempt can be made to align the footprint 500 with the heatmap 700 as closely as possible, maximizing the values ​​of the heatmap corresponding to the individual positions in the footprint. Ideally, the centers of the heatmap areas would correspond exactly to the footprint; however, this will not be the case in practice. Nevertheless, such optimization or alignment allows for the identification of the most optimal gripping pose, even if the reference points were not precisely defined.

[0062] The entire footprint can be matched in a single process, but it is also conceivable that the partial footprints of the individual gripping devices are matched individually (multi-channel instead of single-channel). This is particularly advantageous if the reference points are also assigned to individual gripping devices. Additionally, it can be provided that (step 326) gripping quality information (328) is determined based on the environmental information, which is then taken into account when determining the gripping preference information or, if necessary, only when determining the pose to be used.

[0063] At this point, let's briefly address the case of handling multiple objects simultaneously. In image 600 according to Figure 6, for example, two objects of the same type might be visible; these could be positioned so that their straight edges are (almost) touching. Accordingly, heatmap 700 according to Figure 7 would include three additional areas. Footprint 500 would then also be aligned with the heatmap, but specifically, three areas of the footprint would be matched with six areas of the heatmap. Here, for example, it could be specified that the areas of the heatmaps matched with the footprint must be distributed across the two objects (in this case, two areas and one area). For this to work, there should also be a mapping indicating which points or areas lie on which physical objects. This could be achieved, for example, by analyzing the edges of the individual objects to determine which area lies on which object.

[0064] In step 350, the gripping pose to be used is provided so that the movement sequence for the robot can be determined (step 360), as already explained in relation to Figure 2, in order to be able to move the robot accordingly according to control information.

[0065] Figure 9 schematically shows another embodiment of the invention. Here, a part of a robot 900 is shown, in particular an end effector 908 of the robot. The robot 900 can generally correspond to the robot 100 according to Figure 1a; however, the end effector 908 has four gripping means, one of which is designated 908.1. The four gripping means are, for example, designed as suction cups and arranged in a cross shape (i.e., at the four ends of a right-angled and symmetrical cross). A box 932 is also shown, in which several objects are provided that are to be gripped by means of the end effector. An object 940 is shown being gripped and lifted by means of the end effector or its gripping means.

[0066] The objects in question are, for example, so-called SMD reels, i.e., reels onto which long, thin strips are wound. Object 940 has two parallel rings, one of which is labeled 941. For each of these rings, the object has four ribs 942 arranged at right angles to each other, by means of which the respective ring is connected to a central axis or a central area (around which the strip 943 is wound, not shown here).

[0067] As can be seen from Figure 9, the object 940 can only be gripped by the gripping means 908.1 on one of the ribs at a time. In particular, suction cups (or suction grippers) only find sufficient hold on such ribs.

[0068] Therefore, the reference points already explained above (see Figure 4) can be arranged on the webs 942 for a reference object corresponding to object 940. This allows the gripping pose to be determined such that the gripping means of the end effector are each placed on one of the webs.

[0069] This example shows that it is often not enough to recognize the object as such and, for example, grasp it in the middle with a suction gripper (or another end effector or gripping device), but that it may be expedient or even necessary to grip the object at specific points.

Claims

Claims 1. Method for determining a gripping pose for a robot, for use in determining a movement sequence for the robot (100) to grasp and / or move one or more objects in a work environment by means of an end effector (108, 908) of the robot, wherein the end effector has at least two gripping means (108.1 , 108.2, 108.3, 908.4), comprising: Providing (304) grasp specification information (302) that includes specifications for one or more reference locations on one or more reference objects, specifying at which reference locations the one or more reference objects are to be grasped; Providing (310) end effector geometry information (312) which includes information about geometric dimensions of a gripping area of ​​at least two gripping means; Providing (324) environmental information (322) that has been acquired from the working environment (130) by means of at least one sensor (124); Determining (330) grasp preference information (322) based on the environment information (322) and the grasp preset information (302), wherein the grasp preference information for one or more areas on the one or more objects each comprises a probability distribution, the probability distribution indicating how closely a position in the respective area corresponds to a corresponding reference point; Determine (340), based on the grasping preference information and the end effector geometry information, a grasping pose (342) to be used for the end effector; and Providing (350) the gripping pose to be used for determining the movement sequence for the robot.

2. The method according to claim 1, wherein, in the gripping specification information, the specifications for one or at least one of the several locations on the one or at least one of the several reference objects are respectively include specifying which of the at least two gripping means is to be used to grip the respective reference object at the respective location.

3. Method according to claim 1 or 2, wherein the at least two gripping means are different from each other, in particular with regard to their respective geometric dimensions of the gripping area and / or their respective material and / or their respective gripping force.

4. Method according to one of the preceding claims, wherein the at least two gripping means are each designed as a suction cup.

5. Method according to any of the foregoing claims, further comprising: Determining (326) grasp quality information (328) based on the environment information, wherein the grasp quality information for one or more areas on the one or at least one of the several objects comprises a probability distribution, the probability distribution indicating how well the respective object can be grasped at a specific position in the area by means of the end effector; wherein the grasp preference information and / or the grasp pose to be used are further determined based on the grasp quality information.

6. Method according to one of the preceding claims, wherein, based on the gripping preference information and the end effector geometry information, the gripping pose to be used for the end effector is determined such that a predetermined geometric relationship is satisfied for the gripping areas of the at least two gripping means, in particular that the gripping areas lie in a plane.

7. Method according to any of the foregoing claims, further comprising: Determine (360) the movement sequence, taking into account the gripping pose to be used.

8. The method of claim 7, further comprising: Determine, based on the movement sequence, control information for moving the robot, and Providing control information and / or moving the robot based on the control information.

9. Computing unit (122) comprising means for carrying out the method according to any of the preceding claims.

10. Robot (100) which is configured to receive control information determined by a method according to claim 8, and / or with a computing unit according to claim 9, and with a drive system and a control or regulation unit for controlling the drive system, with an end effector having at least two gripping means for gripping and / or moving one or more objects, and preferably with at least one sensor, in particular a camera, for capturing environmental information of a working environment.

11. Computer program comprising instructions which, when the program is executed by a computer, cause it to execute the method according to claims 1 to 8.

12. Computer-readable storage medium on which the computer program according to claim 11 is stored.

Images