Operating robot with gripper

Abstract

A method for operating a robot (10) guiding a gripper (2) comprises the steps of: determining (S10) a pose of the gripper based on a pose of the robot; in the environment of the gripper, determining (S20; s40) in the image of at least one part of the grabber, at least one virtual boundary contour (S; g); based on the determined at least one boundary contour, classifying (S30; s50); determining (S60), based on the classified data, a pose of a load (3) held by the gripper; and controlling (S70) the robot and / or the gripper based on the determined pose of the load.

Description

Field of the Invention

[0001] The present invention relates to a method and a system for operating a robot guiding a gripper, and a computer program or a computer program product for executing the method described herein. Summary of the Invention

[0002] The object of the present invention is to improve the operation of a robot guiding a gripper.

[0003] The object of the present invention is achieved by a method having the features described in claim 1. Claims 7 and 8 protect a system or a computer program or a computer program product for executing the method described herein. The dependent claims relate to preferred expansions.

[0004] According to an embodiment of the present invention, the robot has a robot arm, particularly a robot arm. Additionally or alternatively, the robot, particularly the robot arm, has at least three, particularly at least six joints or (movement) axes in one embodiment, at least seven joints or (movement) axes in one embodiment, at least three, particularly at least six rotational joints or rotational axes in an expansion, and at least seven rotational joints or rotational axes in one embodiment.

[0005] According to an embodiment of the present invention, the robot guides or carries a gripper. In an expansion, the gripper is a finger gripper with one or more adjustable fingers, a suction gripper, a magnetic gripper, preferably an electromagnetic gripper, etc. In one embodiment, the gripper is preferably arranged on the robot, particularly on the robot arm, in a non-destructive detachable or replaceable manner, preferably on the end flange of the robot, particularly the robot arm.

[0006] According to an embodiment of the present invention, a method for operating a robot guiding a gripper includes the step of determining the pose of the gripper based on the position (Stellung) of the robot.

[0007] In one embodiment, the pose includes a one-, two- or three-dimensional position and / or a one-, two- or three-dimensional orientation; in one embodiment, the pose of the gripper determined based on the pose of the robot includes the one-, two- or three-dimensional position (Position) and / or the one-, two- or three-dimensional orientation (Orientierung) of the gripper relative to a reference system fixed to the robot, in one embodiment relative to the robot, in particular the base of the robot arm and / or relative to the environment of the robot; in one embodiment, the pose of the load held by the gripper determined herein includes the one-, two- or three-dimensional position and / or the one-, two- or three-dimensional orientation of the load relative to the gripper, relative to a reference system fixed to the robot, in one embodiment relative to the robot, in particular the base of the robot arm or relative to the environment of the robot. In one embodiment, the pose of the robot includes the pose of the joints or axes of the robot, preferably the detected or actual pose and / or the commanded or target pose of the joints or axes of the robot. Thus, in one embodiment, the pose of the gripper can be determined based on the (joint) pose of the robot by means of forward kinematics or transformation.

[0008] According to one embodiment of the present invention, the method further comprises the following steps: in the environment of the gripper or by using the environment of the gripper, (respectively) based on the determined pose of the gripper, one or more virtual boundary contours are determined in an image of at least a part of the gripper, wherein the one or more boundary contours preferably (respectively) have the pose of the gripper or a preset translational and / or rotational offset relative to the pose of the gripper. In one embodiment, the method further comprises the step of determining the image, and in another embodiment, the determined image can also be provided by an external entity.

[0009] In one embodiment, the image includes a three-dimensional point cloud and / or a camera image, and / or is determined by means of a 3D imaging device, in particular at least one 3D camera, at least two spatially separated stereo cameras, at least one 3D scanner, etc. In one embodiment, the data of the image gives the three-dimensional positions of the points of the gripper or its environment, preferably the three-dimensional positions of the points on the surface of the gripper or its environment.

[0010] According to one embodiment of the present invention, the method further comprises the following steps:

[0011] - Classifying the data of the image based on the determined boundary contours, in particular based on the determined plurality of boundary contours, preferably (by) screening or eliminating or filtering (filtering out) (to reduce) the data of the image;

[0012] -Determine the pose of the payload held by the gripper based on such classification, in particular the reduced data; and

[0013] -Control the robot and / or the gripper based on the determined pose of the payload.

[0014] One embodiment of the present invention is based on the following recognition: based on the known pose of the robot, the pose of the gripper guided by the robot can be determined, and based on the following concept: based on the image of the gripper with the payload, in or using its environment, the pose of the payload held by the gripper can be determined better, in particular faster and / or more precisely. Preferably, at least a part of the data of the image is determined in the following way: first, at least a part of the image data is screened or filtered as image data that does not belong to the payload according to the pose of the gripper, and the pose of the payload is determined only based on such reduced image (data).

[0015] In one embodiment, one or more virtual boundary contours determined based on the determined gripper pose are the boundary contours of the gripper in the image determined based on the data model of the gripper, and the classification of the data includes screening, removing, or filtering (filtering out) the data of the image that belongs to the gripper, in particular the data located within the gripper boundary contour, based on the gripper boundary contour.

[0016] In one embodiment, the data model of the gripper is determined based on the design data of the gripper, in particular CAD data, or may include such data. In one embodiment, the gripper boundary contour of the gripper corresponds to a preferably simplified (virtual) representation of the gripper in the image, and / or in one embodiment, the gripper boundary contour is determined such that the gripper boundary contour is at least partially consistent with the outer contour of the gripper, in particular the theoretical or data model-based outer contour, within the tolerance caused by the simplification, and preferably located inside the actual outer contour. In one embodiment, due to the pose change of the gripper, such as the pose change of one or more fingers of the gripper, the gripper boundary contour changes accordingly, or the gripper boundary contour of the gripper in the image is also determined based on the commanded or detected pose (gripper pose) of the segments of the gripper relative to each other.

[0017] This is based on the following recognition: there is usually a more precise data model for the gripper, and based on the following idea: the pose of the outer contour of the gripper in the image can be relatively precisely determined based on its determined pose and such a data model or based on its determined virtual boundary contour, and based on this, the image data belonging to the gripper can be screened more quickly, more precisely, and / or more reliably, and thus the pose of the payload can be determined better, in particular faster, more precisely, and / or more reliably according to the (correspondingly simplified) image data.

[0018] Additionally or alternatively, in one embodiment, the (further) virtual boundary contour determined based on the determined gripper pose is the environmental boundary contour of the gripper's environment in the image, which is determined based on the dimensions of the gripper, in particular the theoretical dimensions or target dimensions or actual dimensions or detected dimensions and / or the load held by the gripper, preferably the dimensions of the load whose pose is to be determined, in particular the theoretical dimensions or target dimensions or actual dimensions or detected dimensions, and the classification of the data includes screening or eliminating or filtering (removing) the data of the image belonging to the gripper's environment based on the environmental boundary contour of the gripper, in particular the data located outside the environmental boundary contour (in one embodiment, the gripper's environment includes the robot and / or the environment of the gripper and the robot (collectively), in particular the background or the environment of an object different from the gripper, the robot (and the load)).

[0019] In one embodiment, the environmental boundary contour of the gripper's environment has a virtual, preferably convex, shroud surrounding the load, and in a preferred embodiment, the shroud is in the form of an ellipsoid, in particular a sphere or a cuboid, and / or is determined such that the load is completely within the environmental boundary contour.

[0020] This is based on the recognition that the approximate pose of the load relative to the environment is already known through the pose of the gripper, and on the idea that by hiding the data of the image located outside the boundary contour, which is arranged at or near the determined gripper pose and thus also at or near the pose of the load held by the gripper, and which is therefore dimensioned to reliably or with a corresponding safety margin safeguard the load, the image data that does not actually belong to the load can be screened (more) quickly, (more) precisely, and / or (more) reliably, and thus the pose of the load can be determined better, in particular (more) quickly, (more) precisely, and / or (more) reliably based on the (correspondingly reduced) image data.

[0021] In one embodiment, the classification of the image data based on the determined gripper boundary contour and environmental boundary contour is carried out in multiple stages, where first the image data is reduced by screening based on the environmental boundary contour, and then these already reduced data are (further) reduced by screening based on the gripper boundary contour. In one embodiment, the advantage of doing so is that a significant reduction in data can usually be achieved through screening based on the environmental boundary contour. Additionally, in one embodiment, the environmental boundary contour can have a (simpler) geometric shape (than the gripper boundary contour), enabling the corresponding selection to be carried out (more) quickly, (more) precisely, and / or (more) reliably, and thereby also improving the subsequent screening based on the gripper boundary contour.

[0022] In another embodiment, the classification of the image data based on the determined gripper boundary profile and the environment boundary profile is performed in the reverse order: namely, first the image data is reduced by filtering based on the gripper boundary profile, and then these already reduced data are further reduced by filtering based on the environment boundary profile.

[0023] In another embodiment, the classification of the image data based on the determined gripper boundary profile and the environment boundary profile is performed in parallel, and in an extended scheme it is performed as follows: in one step, not only the data located outside the environment boundary profile is filtered out, but also the data located inside the gripper boundary profile is filtered out. Thereby, the speed of the method can be increased.

[0024] In one embodiment, the data reduced by filtering based on the gripper boundary profile and / or based on the environment boundary profile (which data substantially contains data belonging to the load) is (further or again) filtered before or in order to determine the pose of the load. Thereby, in one embodiment, the pose can also be determined better, in particular (more) quickly, (more) precisely and / or (more) reliably.

[0025] Additionally or alternatively, in one embodiment, the data reduced by filtering based on the gripper boundary profile and / or based on the environment boundary profile (which data correspondingly substantially contains data belonging to the load) is, if necessary, converted into a depth image after further data processing (such as the aforementioned filtering), and based on this a mask is determined respectively, which mask can advantageously be used to segment the load in one or more 2D images. In one embodiment, the pose of the load is determined based on such segmentation on the basis of the classified or reduced image data. Thereby, in one embodiment, the segmentation of the load or the determination of the pose of the load in the 2D image can be performed better, in particular (more) quickly, (more) precisely and / or (more) reliably. In one embodiment, the segmentation of the load is achieved based on the mask, which mask itself is determined based on the classified data, so that the pose of the load held by the gripper is also determined based on the classified data.

[0026] In one embodiment, controlling the robot and / or the gripper based on the determined load pose includes: using the gripper guided by the robot to transport and / or distribute the load.

[0027] In a particularly advantageous application, controlling a robot and / or a gripper based on the determined load pose includes: monitoring and / or correcting the load pose, particularly preferably during the grasping, transporting, and / or distributing of the load by a robot-guided gripper. In one embodiment, preferably during the transportation of the load by the robot, the pose of the load is controlled one or more times, preferably relative to the pose of the gripper, or compared to a reference pose. Here, if an incorrect pose or an undesired pose change of the load is determined or identified, then in one embodiment, a warning is issued accordingly and / or the corresponding movement of the robot and / or the gripper is controlled, such as adjusting the gripper, dropping and re-grasping the load, adjusting the pose of the robot when distributing the load, etc., in response thereto.

[0028] According to one embodiment of the present invention, a system, particularly designed by hardware technology and / or software technology, particularly programming technology, for performing the methods described herein, and / or comprising:

[0029] means for determining the pose of the gripper based on the position of the robot;

[0030] means for determining one or more virtual boundary contours in an image of at least a part of the gripper based on the determined pose in the environment of the gripper, in one embodiment, means for determining the gripper boundary contour based on the determined pose of the gripper and the data model of the gripper; and / or means for determining the environment boundary contour based on the determined gripper pose and the dimensions of the gripper and / or the load held by the gripper;

[0031] means for classifying the image data based on the determined at least one boundary contour, particularly means for filtering out the image data belonging to the gripper based on the gripper boundary contour, and / or means for filtering out the data belonging to the environment of the gripper, particularly the robot and / or the environment of the gripper and the robot, based on the environment boundary contour;

[0032] means for determining the pose of the load held by the gripper based on the classified data; and

[0033] means for controlling the robot and / or the gripper based on the determined pose of the load.

[0034] In one embodiment, the system or its means includes: means for converting the image data reduced by filtering based on the gripper boundary contour and based on the environment boundary contour into a depth image; means for determining at least one mask based on the depth image; and means for segmenting the load in the 2D image based on the mask.

[0035] The system and / or device in the sense of the present invention can be implemented in hardware technology and / or software technology, in particular having: at least one processing unit, preferably data-connected or signal-connected to a storage system and / or a bus system, in particular a digital processing unit, in particular a microprocessor unit (CPU), a graphics card (GPU), etc.; and / or one or more programs or program modules. The processing unit can be designed for this purpose to: process the instructions of a program implemented as stored in the storage system, collect input signals from the data bus, and / or send output signals to the data bus. The storage system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be provided such that it can embody or execute the method described herein, such that the processing unit can execute the steps of the method and thereby, in particular, can control a robot and / or a gripper.

[0036] In one embodiment, a computer program product can have, in particular, a computer-readable, non-volatile storage medium for storing a program or instructions, or a storage medium having a program stored thereon or having instructions stored thereon. In one embodiment, the execution of the program or the instructions causes a system or a controller, in particular a computer or an arrangement of multiple computers, to execute the program or the instructions, such that the system or the controller, in particular one or more computers, execute the method described herein or one or more steps thereof, or the program or the instructions are designed for this purpose.

[0037] In one embodiment, one or more, in particular all, steps of the method are implemented fully or partially by a computer, or one or more, in particular all, steps of the method are executed fully or partially automatically, in particular by the system or its device.

[0038] In one embodiment, the system has a robot and / or a gripper. Description of the Drawings

[0039] More advantages and features are given by the dependent claims and the embodiments. For this purpose, there is partially shown schematically:

[0040] Figure 1 A system for operating a robot guiding a gripper according to an embodiment of the present invention; and

[0041] Figure 2 A method for operating a robot according to an embodiment of the present invention. Detailed Embodiments

[0042] Figure 1A system for operating a six-axis robot 10 is shown, whose (joint) pose is described by joint coordinates q1, …, q6, and a gripper 2 is arranged on its end flange 11, which gripper holds a payload 3. A robot controller 20 communicates with the robot 10 and an imaging device 30, which is used to capture an image of at least a part of the gripper 2 in its environment.

[0043] In step S10, an image in the form of a point cloud is provided, and the pose x of the gripper 2 is determined based on the (joint) pose q = [q1, …, q6] of the robot 10, which pose indicates the three-dimensional position and three-dimensional orientation of the gripper relative to the environment (x = x(q)).

[0044] In step S20, based on the known dimensions of the payload 3 and the pose of the gripper 2 determined in step S10, the diameter and (center point) position of a sphere S are determined such that the held payload 3 can be completely arranged inside the sphere with a preset tolerance. For example, the position [x, y, z] of the center point of the sphere S can be determined by adding the target offset between the pose of the gripper and the center point of the payload to the position of the gripper determined by the pose of the gripper, and the diameter of the sphere S can be chosen large enough such that the payload is completely located inside the sphere S even under the maximum possible deviation of the (still held) payload from this target offset. Preferably, the position of the gripper itself determined by the pose of the gripper can be determined as the position [x, y, z] of the center point of the sphere S, and accordingly the diameter of the sphere S is chosen large enough such that the held payload is always completely located inside the sphere S.

[0045] Now, in step S30, image data located outside the sphere S, such as data belonging to a storage location 4, is filtered out or excluded. This corresponds to filtering data belonging to the gripper environment, in particular the robot 10, and the common environment 4 of the gripper and the robot based on the environmental boundary contour in the form of the sphere S.

[0046] In step S40, based on the data model of the gripper 2 and the pose of the gripper 2 determined in step S10, the gripper boundary contour G of the gripper 2 in the image is determined, which gripper boundary contour is slightly inside the outer contour of the gripper 2 in the pose determined in step S10.

[0047] Subsequently, in step S50, data located inside the gripper boundary contour G in the image data remaining after step S30 is excluded. This corresponds to filtering data belonging to the gripper 2 based on the gripper boundary contour G determined in step S40.

[0048] Through this screening that first based on the environmental boundary profile (step S30) and then on the gripper boundary profile (step S50), the remaining image data is classified as data potentially belonging to the payload 3.

[0049] Now, in step S60, based on this classified data, the pose of the payload 3 relative to the gripper 2 is determined in a known manner (e.g., by pattern recognition or matching, etc.). In one embodiment, in step S60, the image data reduced by screening based on the gripper boundary profile and the environmental boundary profile, preferably after further filtering, is converted into a depth image. Based on this, one or more masks are determined, and these masks are used to segment the payload in the 2D image respectively, where this segmentation or the segmented 2D image can be particularly used to determine the pose.

[0050] In step S70, based on the pose of the payload 3 thus determined, the robot 10 and / or the gripper 2 is controlled, especially during the transportation of the payload 3, to compare its pose relative to the gripper 2 with a reference pose, or to control it, and to make corresponding corrections when an unacceptable deviation (such as slipping) occurs.

[0051] By using the pose of the gripper 2 determined in step S10, the image data belonging to the gripper or the environment can be excluded more quickly, especially improved.

[0052] In addition, accordingly, the errors in determining the payload pose can be advantageously reduced, which are caused by some parts of the gripper or the environment being wrongly confused with some parts of the payload.

[0053] In addition, since the pose of the payload is already roughly known based on the pose of the gripper 2 determined in step S10, the actual pose of the payload 3 can be determined more quickly, especially improved, based on the (classified or reduced) image data.

[0054] Although the exemplary embodiments have been described in the foregoing description, it should be noted that there may be many variations. It should also be noted that the exemplary embodiment is merely an example and should not form any limitation on the scope of protection, application, and construction. On the contrary, through the foregoing description, those skilled in the art can be taught to implement the transformation of at least one exemplary embodiment, where various changes, especially regarding the functions and arrangements of the components described above, can be achieved without departing from the scope of protection of the present invention, for example, can be obtained according to the claims and their equivalent feature combinations.

[0055] List of reference numerals

[0056] 2 Gripper

[0057] 3 Payload

[0058] 4 Storage location

[0059] 10 Robot

[0060] 11 End flange

[0061] 20 Robot controller

[0062] 30 Imaging device

[0063] G Grasper boundary contour

[0064] S Sphere (environmental boundary contour)

[0065] q1 - q6 (Joint) postures of the robot

Claims

1. A method for operating a robot (10) that guides a gripper (2), comprising the following steps: - determining (S10) the pose of the gripper based on the pose of the robot; - in the environment of the gripper, determining (S20; S40) at least one virtual boundary contour (S; G) in an image of at least a part of the gripper based on the determined pose; - classifying (S30; S50) the data of the image based on the determined at least one boundary contour; - determining (S60) the pose of a load (3) held by the gripper based on the classified data; and - controlling (S70) the robot and / or the gripper based on the determined pose of the load.

2. The method according to claim 1, wherein The virtual boundary contour determined based on the determined pose of the gripper is the gripper boundary contour of the gripper determined in the image based on the data model of the gripper, and the classification of the data includes screening the data of the image belonging to the gripper based on the gripper boundary contour.

3. The method according to any one of the preceding claims, characterized in that, The virtual boundary contour determined based on the determined pose of the gripper is the environmental boundary contour of the environment of the gripper in the image, and the environmental boundary contour is determined based on the size of the gripper and / or the load held by the gripper; and the classification of the data includes screening the data of the image belonging to the environment of the gripper, in particular belonging to the robot and / or belonging to the environment of the gripper and the robot, based on the environmental boundary contour.

4. The method according to claims 2 and 3, characterized in that, First, the data of the image is reduced by screening based on the environmental boundary contour, and then the already reduced data is further reduced by screening based on the gripper boundary contour.

5. The method according to any one of the preceding claims 2-4, characterized in that, The data of the image reduced by screening based on the gripper boundary contour and / or based on the environmental boundary contour is converted into a depth image, and based on this, at least one mask is determined, which is particularly used for segmenting the load in a 2D image.

6. The method according to any one of the preceding claims, characterized in that, Controlling the robot and / or the gripper based on the determined pose of the load includes: transporting and / or distributing the load using the gripper guided by the robot, and / or monitoring and / or correcting the pose of the load, especially during gripping, transporting, and / or distributing the load using the gripper guided by the robot.

7. A system for operating a robot that guides a gripper, wherein, The system is designed to execute the method according to any one of the preceding claims, and / or includes: - means for determining the pose of the gripper based on the position of the robot; - means for determining at least one virtual boundary contour in an image of at least a part of the gripper in the environment of the gripper based on the determined pose; - means for classifying the data of the image based on the determined at least one boundary contour; - means for determining the pose of a load held by the gripper based on the classified data; and - means for controlling the robot and / or the gripper based on the determined pose of the load.

8. A computer program or computer program product, wherein, The computer program or computer program product comprises instructions, in particular instructions stored on a computer-readable and / or non-volatile storage medium, which when executed by one or more computers or by the system according to claim 7 cause the one or more computers or the system to carry out the method according to any one of claims 1 to 6.

Images