Operating a robot having a gripper

Abstract

A method for operating a robot (10) which guides a gripper (2) comprises the steps of: determining (S10) a pose of the gripper on the basis of a position of the robot, determining (S20; S40) at least one virtual boundary contour (S; G) in an image of at least part of the gripper in the surroundings of the gripper on the basis of the determined pose; classifying (S30; S50) data of the image on the basis of the determined at least one boundary contour, determining (S60) a pose of a load (3), held by the gripper, on the basis of the classified data; and controlling (S70) the robot and / or the gripper on the basis of the determined pose of the load.

Description

[0001] Description

[0002] Operating a robot with a gripper

[0003] The present invention relates to a method and a system for operating a robot that guides a gripper, as well as a computer program or system.

[0004] Computer program product for carrying out a method described here.

[0005] The object of the present invention is to improve an operation of a robot that guides a gripper.

[0006] This object is achieved by a method having the features of claim 1. Claims 7, 8 represent a system or computer program or.

[0007] A computer program product for carrying out a method described herein is protected. The subclaims relate to advantageous developments.

[0008] According to one embodiment of the present invention, a robot has a robot arm, in particular can be such a robot arm. Additionally or alternatively, the robot, in particular the robot arm, has at least three, in particular at least six, in one embodiment at least seven, joints or (movement) axes, in one embodiment, and at least three, in particular at least six, in one embodiment at least seven, rotary joints or rotary axes.

[0009] According to one embodiment of the present invention, the robot guides or carries a gripper, in one further development a finger gripper with one or more adjustable fingers, a suction gripper, a magnetic, preferably electromagnetic, gripper, or the like. In one embodiment, the gripper is arranged on the robot, in particular the robot arm, preferably on an end flange of the robot, in particular the robot arm, preferably in a manner that is preferably non-destructively removable or replaceable.

[0010] According to one embodiment of the present invention, a method for operating the robot that guides the gripper comprises the step of: determining a pose of the gripper based on a posture of the robot.A pose comprises in one embodiment a one-, two- or three-dimensional position and / or a one-, two- or three-dimensional orientation, a pose of the gripper determined on the basis of a position of the robot in one embodiment a one-, two- or three-dimensional position and / or a one-, two- or three-dimensional orientation of the gripper relative to a robot-fixed reference system, in one embodiment relative to a base of the robot, in particular the robot arm, and / or relative to an environment of the robot, a pose of a load held by the gripper determined here in one embodiment a one-, two- or three-dimensional position and / or a one-, two- or three-dimensional orientation of the load relative to the gripper, relative to a robot-fixed reference system, in one embodiment relative to a base of the robot, in particular the robot arm, or relative to an environment of the robot.In one embodiment, a robot position comprises the positions of the robot's joints or axes, preferably detected or actual positions and / or commanded or target positions of the robot's joints or axes. Thus, in one embodiment, a gripper pose can be determined using forward kinematics or transformation based on the positions of the robot's joints.

[0011] According to one embodiment of the present invention, the method comprises the further step of determining one or more virtual boundary contours in an image of at least a portion of the gripper in or with an environment of the gripper (each) based on the determined pose of the gripper, wherein the or one or more of the boundary contours each preferably have the pose of the gripper or a predetermined translational and / or rotational offset relative to the pose of the gripper. In one embodiment, the method also comprises the step of determining this image; in another embodiment, the determined image can also be provided by an external instance.

[0012] In one embodiment, the image comprises a three-dimensional point cloud and / or a camera image and / or is or will be determined using a 3D recording device, in particular at least one 3D camera, at least two spatially spaced stereographic cameras, at least one 3D scanner, or the like. Data from the image, in one embodiment, indicate three-dimensional positions of points on the gripper or its surroundings, preferably points on the surface of the gripper or its surroundings. According to one embodiment of the present invention, the method comprises the further steps:

[0013] - Classifying, preferably (reducing by) sorting out or eliminating or (filtering away), data of the image on the basis of the determined boundary contour, in particular on the basis of the determined boundary contours;

[0014] - Determining a pose of a load held by the gripper on the basis of the (thus) classified, in particular reduced, data; and

[0015] - Controlling the robot and / or gripper based on this determined pose of the load.

[0016] One embodiment of the present invention is based on the realization that a pose of the robot-guided gripper can be determined based on a known position of the robot, and the idea based thereon of exploiting this to better, in particular more quickly and / or more precisely, determine a pose of a load held by the gripper based on an image of the gripper with the load in or with its environment, preferably by at least part of the data of the image initially being sorted out or filtered out as not being assigned to the load based on a pose of the gripper and the pose of the load is then determined only on the basis of the (data of the) image reduced in this way.

[0017] In one embodiment, the or one of the virtual boundary contour(s) determined on the basis of the determined pose of the gripper is a gripper boundary contour for the gripper in the image, which is determined on the basis of a data model of the gripper, and the classification of data comprises sorting out or eliminating or filtering out data of the image assigned to the gripper on the basis of this gripper boundary contour, in particular data which or whose position lies within the gripper boundary contour.

[0018] The data model of the gripper is or is determined in one embodiment on the basis of design data, in particular CAD data, of the gripper or can include such data. In one embodiment, the gripper boundary contour for the gripper corresponds to a, preferably simplified, (virtual) representation of the gripper in the illustration and / or is determined in one embodiment in such a way that, within the framework of a tolerance caused by the simplification, it at least partially corresponds to an outer contour of the gripper, in particular a theoretical one or one based on the data model, and preferably lies within the actual outer contour. In one embodiment, the gripper boundary contour changes accordingly or as a result of a change in the position of the gripper, for example of one or more fingers of the gripper.the gripper boundary contour for the gripper in the figure is (also) determined on the basis of a commanded or detected pose of the gripper's links relative to each other (gripper position).

[0019] This is based on the realization that more precise data models are often available for the gripper, and the idea is to exploit the fact that the pose of the outer contour of the gripper in the image is determined relatively precisely on the basis of its determined pose and such a data model or a virtual boundary contour determined on the basis thereof, and based on this, the data of the image assigned to the gripper is sorted out more quickly, more precisely and / or more reliably, and as a result the determination of the pose of the load based on the (correspondingly reduced) data of the image can be carried out better, in particular more quickly, more precisely and / or more reliably.

[0020] Additionally or alternatively, in one embodiment, the (further) virtual boundary contour(s) determined on the basis of the determined pose of the gripper is / are an environmental boundary contour for an environment of the gripper in the image, which is determined on the basis of dimensions, in particular theoretical or target dimensions or real or recorded dimensions, of the gripper and / or dimensions, in particular theoretical or target dimensions or real or recorded dimensions, of a load held by the gripper, preferably the load whose pose is determined, and the classification of data comprises sorting out or eliminating or (filtering away) an environment of the gripper (which in one embodiment comprises the robot and / or a (common) environment of gripper and robot, in particular therefore a background orfrom the gripper, the robot (and the load) comprises objects other than the gripper) of the mapping on the basis of this environmental boundary contour, in particular data which or whose position lies outside the environmental boundary contour.

[0021] In one embodiment, the environmental boundary contour for an environment of the gripper has a virtual, preferably convex, shell around the load, in a preferred embodiment in the form of an ellipsoid, in particular a sphere, or a cuboid, and / or is determined such that the load lies completely within the environmental boundary contour.

[0022] This is based on the realisation that the approximate pose of the load relative to its surroundings is already known from the pose of the gripper, and on the idea that by masking out data from the image that lie outside a boundary contour that is located at or near the determined pose of the gripper and thus also at or near the pose of the load it is holding and is dimensioned such that it reliably envelops the load or with an appropriate safety margin, data from the image that reliably do not belong to the load can be sorted out more quickly, more precisely and / or more reliably, and as a result the determination of the load's pose based on the (correspondingly reduced) data from the image can be carried out better, in particular more quickly, more precisely and / or more reliably.

[0023] In one embodiment, the classification of image data based on the determined gripper boundary contour and surrounding boundary contour is carried out in several stages, in that first the image data is reduced by sorting based on the surrounding boundary contour and then this reduced data is (even) further reduced by sorting based on the gripper boundary contour. In one embodiment, this has the advantage that a significant data reduction can often be achieved by sorting based on the surrounding boundary contour. Furthermore, in one embodiment, the surrounding boundary contour can have a simpler geometric shape (than the gripper boundary contour), so that the corresponding sorting can take place more quickly, more precisely and / or more reliably, thus also improving the subsequent sorting based on the gripper boundary contour.

[0024] In another embodiment, the classification of data of the image is carried out on the basis of the determined gripper boundary contour and surrounding boundary contour by conversely first reducing data of the image by sorting on the basis of the gripper boundary contour and then (even) further reducing this reduced data by sorting on the basis of the surrounding boundary contour.

[0025] In another embodiment, the classification of image data based on the determined gripper boundary contour and the surrounding boundary contour occurs in parallel. In a further development, both data that lie outside the surrounding boundary contour and data that lie within the gripper boundary contour are sorted out in a single step. This can accelerate the process.

[0026] In one embodiment, the data reduced by sorting based on the gripper boundary contour and / or based on the surrounding boundary contour, which essentially contain data associated with the load, are filtered (further or after) before or to determine the pose of the load. This allows the pose to be determined even better, in particular more quickly, more precisely, and / or more reliably, in one embodiment.

[0027] Additionally or alternatively, in one embodiment, the data reduced by sorting based on the gripper boundary contour and / or based on the surrounding boundary contour, which data accordingly essentially contain data assigned to the load, are converted into a depth image, if necessary after further data processing, for example the aforementioned post-filtering, and based on this, a mask is determined which can advantageously be used to segment the load in one or more 2D images. In one embodiment, the pose of the load is determined based on this segmentation on the basis of the classified or reduced data of the image. In one embodiment, this makes it possible to improve the segmentation of the load in a 2D image or to determine the pose, in particular to carry out this process more quickly, more precisely and / or more reliably.In one embodiment, by segmenting the load based on the mask(s), which in turn are determined based on the classified data, the pose of the load held by the gripper is determined (also) based on the classified data.

[0028] In one embodiment, controlling the robot and / or gripper based on the determined pose of the load comprises transporting and / or releasing the load with the robot-guided gripper.

[0029] In a particularly advantageous application, controlling the robot and / or gripper on the basis of the determined pose of the load comprises monitoring and / or correcting a pose of the load, particularly preferably during gripping, transporting and / or releasing the load with the robot-guided gripper. In one embodiment, the pose of the load, preferably relative to the gripper, is checked once or repeatedly or compared with a reference pose, preferably while the robot is transporting the load. If an incorrect pose or an undesirable change in the pose of the load is determined or detected, in one embodiment this is responded to by correspondingly issuing a warning and / or correspondingly controlling a movement of the robot and / or gripper, for example adjusting the grip, setting down and re-picking up the load, adjusting the position of the robot for or during the release of the load, or the like.

[0030] According to one embodiment of the present invention, a system, in particular hardware and / or software, in particular program technology, is set up to carry out a method described here and / or comprises:

[0031] Means for determining a pose of the gripper based on a position of the robot; means for determining one or more virtual boundary contours in an image of at least a portion of the gripper in an environment of the gripper based on the determined pose, in one embodiment for determining the gripper boundary contour based on the determined pose and a data model of the gripper and / or for determining the environmental boundary contour based on the determined pose of the gripper and dimensions of the gripper and / or a load held by the gripper; means for classifying data of the image based on the determined at least one boundary contour, in particular for sorting out data of the image associated with the gripper based on the gripper boundary contour and / or for sorting out data associated with an environment of the gripper, in particular the robot and / or an environment of the gripper and robot, based on the environmental boundary contour;

[0032] Means for determining a pose of a load held by the gripper based on the classified data; and

[0033] Means for controlling the robot and / or gripper based on the determined pose of the load.

[0034] In one embodiment, the system or its means comprises: means for converting the data of the image reduced by sorting based on the gripper boundary contour and based on the surrounding boundary contour into a depth image, means for determining at least one mask based on this depth image, and means for segmenting the load in a 2D image based on this mask.

[0035] A system and / or means within the meaning of the present invention can be designed in hardware and / or software, in particular at least one, in particular digital, processing unit, in particular a microprocessor unit (CPU), graphics card (GPU) or the like, preferably connected to a memory and / or bus system for data or signals, and / or one or more programs or program modules. The processing unit can be designed to execute instructions implemented as a program stored in a memory system, to detect input signals from a data bus, and / or to output output signals to a data bus. A memory system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be designed in such a way that it embodies the methods described here orcapable of carrying out such a method, so that the processing unit can carry out the steps of such methods and thus in particular can control the robot and / or gripper.

[0036] In one embodiment, a computer program product can comprise, in particular be, a storage medium, in particular a computer-readable and / or non-volatile one, for storing a program or instructions or with a program or instructions stored thereon. In one embodiment, execution of this program or these instructions by a system or a controller, in particular a computer or an arrangement of multiple computers, causes the system or the controller, in particular the computer(s), to carry out a method described here or one or more of its steps, or the program or the instructions are configured to do so.

[0037] In one embodiment, one or more, in particular all, steps of the method are fully or partially computer-implemented, or one or more, in particular all, steps of the method are fully or partially automated, in particular by the system or its means. In one embodiment, the system comprises the robot and / or gripper.

[0038] Further advantages and features emerge from the subclaims and the exemplary embodiments. The following shows, partly schematically:

[0039] Fig. 1 : a system for operating a robot that guides a gripper according to an embodiment of the present invention; and

[0040] Fig. 2: a method for operating the robot according to an embodiment of the present invention.

[0041] Fig. 1 shows a system for operating a six-axis robot 10, whose (joint) position(s) are specified by joint coordinates qi, ..., q6, and on whose end flange 11 a gripper 2 is arranged with a load 3 held by it. A robot controller 20 communicates with the robot 10 and a recording device 30 for recording an image of at least a portion of the gripper 2 in its surroundings.

[0042] In a step S10, the image is provided in the form of a point cloud and, based on the (joint) position(s) q = [qi,... ,q6] of the robot 10, a pose x of the gripper 2 is determined, which indicates its three-dimensional position and three-dimensional orientation relative to the environment (x = x(q)).

[0043] In a step S20, based on the known dimensions of the load 3 and the pose of the gripper 2 determined in step S10, a diameter and a (center) position of a sphere S are determined such that the held load 3 can be arranged completely within the sphere with a specified tolerance. For example, the position [x, y, z] of the center of the sphere S can be determined as the position of the gripper determined by the pose of the gripper plus a target offset between the gripper base and the load center, and the diameter of the sphere S can be selected to be large enough that the load (still) lies completely within the sphere S even with the maximum possible deviation from this target offset (with the load (still) held).Preferably, the position [x, y, z] of the center of the sphere S can be determined as the position of the gripper itself determined by the pose of the gripper and the diameter of the sphere S can be chosen accordingly larger so that the held load always lies completely within the sphere S.

[0044] Now, in a step S30, those data from the image that lie outside the sphere S are sorted out or eliminated, for example, data assigned to a storage location 4. This corresponds to a sorting out of data assigned to the environment of the gripper, in particular to the robot 10 and a common environment 4 of the gripper and robot, based on an environment boundary contour in the form of the sphere S.

[0045] In a step S40, based on a data model of the gripper 2 and the pose of the gripper 2 determined in step S10, a gripper boundary contour G for the gripper 2 in the image is determined, which lies slightly within the outer contour of the gripper 2 in the pose determined in step S10.

[0046] Then, in step S50, those data of the image remaining after step S30 that lie within the gripper boundary contour G are eliminated. This corresponds to a sorting out of data assigned to gripper 2 based on the gripper boundary contour G determined in step S40.

[0047] By this sorting, first on the basis of the surrounding boundary contour (step S30) and then on the basis of the gripper boundary contour (step S50), the remaining data of the image are classified as data potentially assigned to load 3.

[0048] Now, in a step S60, the pose of the load 3 relative to the gripper 2 is determined on the basis of the data classified in this way in a manner known per se, for example by recognizing or matching patterns or the like. In one embodiment, in step S60, the data of the image reduced by sorting based on the gripper boundary contour and on the basis of the surrounding boundary contour are converted into a depth image, preferably after post-filtering, one or more masks are determined based on this, and these masks are (each) used to segment the load in a 2D image, wherein this segmentation or segmented 2D image(s) can be used in particular to determine the pose. In a step S70, the robot 10 and / or gripper 2 is controlled on the basis of the pose of the load 3 determined in this way, in particular during transport of the load 3, its pose relative to the gripper 2 is compared with a reference pose orchecked and corrected accordingly in the event of an unacceptable deviation, for example slipping.

[0049] By using the pose of the gripper 2 determined in step S10, the elimination of data associated with the gripper or the environment from the image can be improved, in particular carried out more quickly.

[0050] In addition, this can advantageously reduce errors in determining the pose of the load, which are based on parts of the gripper or the environment being mistakenly confused with parts of the load.

[0051] In addition, since the pose of the load is already approximately known due to the pose of the gripper 2 determined in step S10, the determination of the actual pose of the load 3 on the basis of the (classified or reduced data of the) image can be improved, in particular carried out more quickly.

[0052] Although exemplary embodiments have been explained in the preceding description, it should be noted that a multitude of modifications are possible. Furthermore, it should be noted that the exemplary embodiments are merely examples and are not intended to limit the scope of protection, applications, or structure in any way. Rather, the preceding description provides the skilled person with a guide for implementing at least one exemplary embodiment, whereby various changes, particularly with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as it results from the claims and equivalent combinations of features.

[0053] 2 grippers

[0054] 3 Load 4 Storage

[0055] 10 robots

[0056] 11 End flange

[0057] 20 Robot control

[0058] 30 Holding device G Gripper limit contour

[0059] S sphere (environmental boundary contour) qi-q6 (joint) position(s) of the robot

Claims

Patent claims Method for operating a robot (10) which guides a gripper (2), comprising the steps: - determining (S10) a pose of the gripper based on a position of the robot; - determining (S20; S40) at least one virtual boundary contour (S; G) in an image of at least a part of the gripper in an environment of the gripper on the basis of the determined pose; - Classifying (S30; S50) data of the image on the basis of the determined at least one boundary contour; - determining (S60) a pose of a load (3) held by the gripper on the basis of the classified data; and - Controlling (S70) the robot and / or gripper based on the determined pose of the load. The method according to claim 1, characterized in that a virtual boundary contour determined based on the determined pose of the gripper is a gripper boundary contour for the gripper in the image determined based on a data model of the gripper, and classifying data comprises sorting out data from the image associated with the gripper based on this gripper boundary contour.Method according to one of the preceding claims, characterized in that a virtual boundary contour determined on the basis of the determined pose of the gripper is an environmental boundary contour for an environment of the gripper in the image, determined on the basis of dimensions of the gripper and / or a load held by the gripper, and the classification of data comprises sorting out data from the image associated with an environment of the gripper, in particular the robot and / or an environment of the gripper and robot, based on this environmental boundary contour. Method according to claims 2 and 3, characterized in that first, data from the image are reduced by sorting out based on the environmental boundary contour, and then the reduced data are further reduced by sorting out based on the gripper boundary contour. Method according to one of the preceding claims 2-4, characterized in that the image data reduced by sorting based on the gripper boundary contour and / or based on the surrounding boundary contour is converted into a depth image, and based on this, at least one mask is determined, in particular for segmenting the load in a 2D image. Method according to one of the preceding claims, characterized in that controlling the robot and / or gripper based on the determined pose of the load comprises transporting and / or releasing the load with the robot-guided gripper and / or monitoring and / or correcting a pose of the load, in particular during gripping, transporting, and / or releasing the load with the robot-guided gripper. System for operating a robot that guides a gripper, wherein the system is configured to carry out a method according to one of the preceding claims and / or comprises: - means for determining a pose of the gripper based on a position of the robot; - means for determining at least one virtual boundary contour in an image of at least a part of the gripper in an environment of the gripper on the basis of the determined pose; - means for classifying data of the image on the basis of the determined at least one boundary contour; - means for determining a pose of a load held by the gripper based on the classified data; and - Means for controlling the robot and / or gripper based on the determined pose of the load. A computer program or computer program product, wherein the computer program or computer program product contains instructions, in particular stored on a computer-readable and / or non-volatile storage medium, which, when executed by one or more computers or a system according to claim 7, cause the computer(s) or system to perform a method according to one of claims 1 to 6.

Images