Gripping with packaging material

Abstract

A method (20) for operating a gripping robot (2, 3), wherein at least one object (5, 9) to be gripped is at least partially located in packaging material (8) and / or is at least partially covered by the latter, in particular in a container (6) with packaging material (8), wherein the method (20) comprises: determining (S10) scene data of a scene by means of a camera (4), wherein the scene data describe depth information and colour information of a scene (10); determining (S12) classification data by classifying the scene data, wherein the classification data describe an association with packaging material (8); determining (S14) a collision object by filtering, in particular segmenting, the scene data on the basis of the classification data; determining (S16) grip position data and / or motion data on the basis of the collision object, wherein the grip position data describe at least one grip position (11, 11') of the gripping robot (2, 3) on at least one object (5, 9) to be gripped, and wherein the motion data describe at least one motion path for at least part of the gripping robot (2, 3).

Description

[0001] Description

[0002] Gripping with packaging material

[0003] The present invention relates to a method for determining a grip position, a system for operating at least one gripping robot, and a computer program or computer program product.

[0004] In bin-picking applications, a robot equipped with a gripper typically picks objects from a bin and places or drops the picked objects at a target location. To do this, a collision-free path is typically planned for the robot arm to bring the gripper to a target position where it can successfully grasp the object during pickup.

[0005] In certain applications, objects are typically packaged in a container containing deformable packaging material to protect the objects. For example, in the food industry, fresh fish are covered with ice, and in other applications, fragile objects are protected with wood chips or Styrofoam flakes. This can pose a challenge for path planning algorithms, as there may be no collision-free target positions for the gripper that avoid the packaging material while still allowing successful gripping of the object.

[0006] The object of the present invention is in particular to improve this.

[0007] This object is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0008] According to one embodiment of the present invention, a method for operating a gripper robot is provided. In one embodiment, at least one object to be gripped by the gripper robot is located, in particular at least partially, in packaging material and / or is covered by it, in particular at least partially, in particular in a container with packaging material. According to one embodiment of the present invention, the method for operating the gripper robot comprises a step of determining scene data of a scene by means of a camera, in particular an RGBD camera, wherein the scene data describe depth information and color information of the scene. In one embodiment, the depth information and color information of the scene can be determined using means of the camera, in particular by means of a stereo camera configured to record depth information of the scene and a camera configured to record color information of the scene.In one embodiment, means for recording depth information and color information can be configured, in particular combined in one camera, such as in particular in an RGBD camera. According to one embodiment, the method further comprises a step for determining classification data, which is determined in particular by classifying the scene data. In one embodiment, the classification data describes an affiliation with packaging material. In one embodiment, this can be determined by applying known classification methods. In one embodiment, the method further comprises determining a collision object, in particular for motion planning and / or handle position planning, by filtering, in particular segmenting, the scene data based on the classification data.In one embodiment, this makes it possible to ensure that the scene data comprises less data than, for example, before filtering or without detecting a collision object. Furthermore, this can advantageously make it possible for the scene data to be reduced by means of filtering, starting from an at least substantially entire scene, in particular before a gripping position and / or a movement of the robot are planned. In one embodiment, the method further comprises determining gripping position data and / or movement data, wherein the gripping position data in one embodiment describe at least one gripping position for the gripping robot on at least one object to be gripped and the movement data describe at least one movement path or a movement for at least part of the gripping robot.

[0009] The term "scene," as used herein, is to be understood in particular as a snapshot of a (relevant) environment, which includes a scenery with objects, in particular the container, the objects to be grasped and / or the packaging material; dynamic elements, such as in particular at least parts of the robot; the field of view(s) of the camera(s) and / or state(s) of the robot or the camera, as well as the linking of these entities.

[0010] The term "camera," as used herein, is to be understood in particular as a recording device for recording digital and / or three-dimensional images and can in particular comprise at least one 3D camera and / or at least two spatially spaced-apart cameras and / or at least one scanner, preferably for three-dimensional scanning. In one embodiment, the scene data mentioned herein comprise depth information, preferably a point cloud, more preferably a three-dimensional point cloud, and can in particular be such a thing or consist of such a thing, and color information. Depending on the recording device, color information is represented as 2D information in the scene data or is or can be assigned as 3D information to the depth information, in particular to the points of the point cloud.

[0011] This makes it possible in some embodiments for the packaging material to be taken into account in the path planning, in particular by determining a collision object, or to be included, in particular based on the determination of movement data, in particular in contrast to the prior art, in which the packaging material is usually ignored and the path planning is based (solely) on the detected objects and a priori known objects, e.g. the container at a configured position.

[0012] In one embodiment, when handling packaging material as described above, the scene described by the scene data can advantageously be "positively" defined for the path planner and / or collision checker with a collision object described therein. In contrast, the prior art typically starts with an empty scene and then adds known objects to the scene, in particular, detected known objects in the container and / or the container itself, etc.

[0013] Advantageously, by means of embodiments described herein, in comparison with the prior art, the point cloud of the scene described by the scene data can be used as a collision object, since the collision object, in particular the point cloud of the collision object, no longer contains the packaging material, which can lead to more possible grip positions and / or path planning, in particular in comparison with the prior art or in comparison to collision objects that are not determined based on classification data.

[0014] In one embodiment, the method comprises the step of setting the scene in the camera or the like prior to determining scene data.

[0015] Recording device, in particular by appropriate adjustment of the robot arm and / or moving and / or focusing of the recording device or the camera.

[0016] In one embodiment, the method comprises moving the gripping robot based on the determined movement data and / or gripping, in particular with the gripping robot, based on the determined grip position data.

[0017] Advantageously, this allows more (collision-free) handle positions and / or more (collision-free) movement paths to be determined than in the prior art or when avoiding collisions with the packaging material.

[0018] In some embodiments, the invention is based on the approach that the, in particular the entire, scene is or is described by the (determined) scene data, in particular including the, in particular complete, depth information or point cloud, and based thereon, parts of the scene are removed in the scene data which can be assumed not to pose a problem in the event of potential collisions during movement and / or gripping, such as in particular the packaging material in the container.

[0019] The invention is further based in embodiments on the approach that the (remaining parts of the) depth information or point cloud of the collision object (as described herein) can be used for collision checks. This advantageously makes path planning (significantly) more robust in embodiments, since the point cloud better depicts or can depict the real situation than a scene that was (only) set up on the basis of (recognized) previously known or preconfigured objects, such as in particular a CAD model and a position of the container, or incompletely recognized objects, in particular in the container. In one embodiment, determining the collision object comprises removing depth information based on the classification data, in particular where the classification data indicate affiliation with packaging material or where scene data was or is classified as packaging material.

[0020] Advantageously, in some embodiments, this allows objects relevant for path planning or motion planning to be (better) detected or taken into account. Thus, in some embodiments, collision checking can be improved; in particular, in some embodiments, the risk that not all relevant objects are taken into account in the collision check can be reduced, such as objects that are present in the container but not detected by the object detection system, for example, due to partial or partial coverage with packaging material. Advantageously, in some embodiments, fewer collisions with these objects can occur than with prior art methods.

[0021] In one embodiment, the classification is carried out pixel-by-pixel and / or section-by-section based on the scene data, in particular based on the color information of the scene data, wherein the color information of the scene data corresponds to a 2D image of the scene or wherein the color information is assigned to the respective depth information.

[0022] This advantageously makes it easier to identify and classify packaging material. Furthermore, it makes it easier to filter, particularly segment, the scene data.

[0023] Furthermore, the segmentation of the packaging material can advantageously also benefit the object recognition algorithms in some embodiments, which (must) estimate the position of the objects in the container. These algorithms include a final position correction step in some embodiments, which geometrically aligns the object model (CAD and / or point cloud) with the scene represented by the scene data, in particular the point cloud. If, in one embodiment, the objects in the collision object are already segmented from the packaging material, or if the collision object (only) contains scene data that has not been classified as packaging material, this (advantageously) reduces or prevents incorrect point assignments between the model of the known object and the packaging material in the scene.

[0024] In one embodiment, the removal of the depth information is based on a mapping of 2D data to 3D data of the scene data, in particular of 2D color information to 3D depth information or point cloud, in particular if the color information is 2D information or 2D data. In one embodiment, the classification can be carried out pixel by pixel and / or section by section using 2D color information of the scene data. The mapping of 2D data to 3D depth information or point cloud can, in one embodiment, be based on intrinsic parameters of the camera, such as focal length, aperture, field of view, resolution or corresponding camera parameters, and / or extrinsic parameters of the camera, such as position and / or orientation or the like.

[0025] In some embodiments, this makes it possible for packaging material or areas classified as packaging material in the 2D color information to be (advantageously) removed from the depth information of the scene, so that a detected collision object (at least essentially only) has depth information that can be or is assigned to known objects and / or unknown objects.

[0026] In some embodiments, the point cloud may (advantageously) also contain objects that were not intended, such as, in particular, a random object left behind (by someone) in the container. Further advantageously, the method described herein is applicable to various, in particular all, objects in the container, such as, in particular, a wide variety of different fish or other objects with a corresponding variety.

[0027] In one embodiment, color information can be used to filter for packaging material more reliably, so that a collision object, at least essentially, only contains objects that are either known, such as in particular a container in which the objects and the packaging material are located, parts of the robot, such as in particular the gripper, in particular depending on the attachment of the camera, and / or objects to be grasped or objects that are to be grasped but are not (yet) known.

[0028] In one embodiment, this can make it possible to determine a handle position more robustly, especially when there are different objects in the container.

[0029] In one embodiment, the scene data is classified using a convolutional neural network (CNN). This allows for faster classification, particularly because the CNN must (only) be capable of detecting or classifying one class.

[0030] Advantageously, in embodiments, training of the CNN is comparatively simpler because in particular only one class is or must be recognized or classified, namely the packaging material, such as in particular ice, wood chips or plastic material, such as in particular packaging chips such as polystyrene flakes or the like.

[0031] In one embodiment, the determination of grip position data and / or movement data is additionally based on known objects in the scene, in particular on a CAD model of the object to be gripped and / or on a CAD model of the container in which the objects and the packaging material are located.

[0032] This advantageously makes it possible in one embodiment for the grip position data to be determined in a more robust manner.

[0033] In one embodiment, the camera is attached to the gripping robot and / or the camera is attached independently of the gripping robot, in particular with a view of the scene. In one embodiment, a first camera and a second camera, in particular a second camera that is different from the first camera, can be used to acquire the scene data. In one embodiment, the first scene data and second scene data acquired by the first and second cameras can be merged to form scene data, which can then be further processed as described herein.

[0034] This can advantageously make it possible in one embodiment for scene data to be determined in a more robust manner.

[0035] In one embodiment of the present invention, a system for operating at least one robot is provided. In one embodiment, the system is configured to carry out a method described herein. In one embodiment, the system comprises at least one camera, in particular an RGBD camera, and at least one gripper robot. In one embodiment, the system and / or its means further comprise means for determining scene data of a scene. In one embodiment, the system and / or its means comprise means for classifying the scene data, in particular for determining classification data. In one embodiment, the system and / or its means comprise means for determining a collision object. In one embodiment, the system and / or its means comprise means for determining grip position data and / or movement data.

[0036] Advantageously, this makes it possible, in one embodiment, to determine a gripping position that would be classified as collision-prone according to prior art methods. In particular, in one embodiment, a gripper of the gripping robot can advantageously penetrate the packaging material, in particular, a comparatively better gripping position can be determined.

[0037] A system and / or means within the meaning of the present invention can be designed in hardware and / or software, in particular at least one, in particular digital, processing unit, in particular a microprocessor unit (CPU), graphics card (GPU) or the like, preferably connected to a memory and / or bus system for data or signals, and / or one or more programs or program modules. The processing unit can be designed to execute instructions implemented as a program stored in a memory system, to detect input signals from a data bus, and / or to output signals to a data bus. A memory system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be designed in such a way that it embodies the methods described here orcapable of carrying out such a method, so that the processing unit can carry out the steps of such methods and thus in particular can operate the robot.

[0038] In one embodiment, a computer program product can comprise, in particular be, a storage medium, in particular a computer-readable and / or non-volatile one, for storing a program or instructions or with a program or instructions stored thereon. In one embodiment, execution of this program or these instructions by a system or a controller, in particular a computer or an arrangement of multiple computers, causes the system or the controller, in particular the computer(s), to carry out a method described here or one or more of its steps, or the program or the instructions are configured to do so.

[0039] In one embodiment, one or more, in particular all, steps of the method are carried out completely or partially automatically, in particular by the controller or its means.

[0040] Further advantages and features emerge from the subclaims and the exemplary embodiments. The following shows, partly schematically:

[0041] Fig. 1 : a system according to an embodiment of the present invention;

[0042] Fig. 2 scene data according to an embodiment;

[0043] Fig. 3 classification data according to an embodiment;

[0044] Fig. 4 shows a collision object according to an embodiment; and

[0045] Fig. 5: a method in a block diagram representation according to an embodiment of the present invention. Figure 1 schematically shows a system 1 with an exemplary gripping robot 2, 3, which has a gripper 3. The gripper 3 is schematically shown in Figure 1 with two fingers, but in embodiments it can have more fingers or another type of gripping device that is configured for gripping, in particular picking up, objects. Furthermore, a scene 10 is shown, which schematically shows known objects 5 in a container 6. A camera 4 is configured to capture the scene 10, in particular to determine scene data. For this purpose, the camera 4, in the exemplary representation in Figure 1, is mounted independently of the gripping robot 2, 3 and is connected in data communication to a processing unit 7, which in turn is connected in data communication to the robot 2, 3.In some embodiments, the processing unit 7 can be integrated into the camera 4 and / or the robot 2, 3. The objects 5 in the container 6 are embedded in packaging material 8 (not shown here).

[0046] Figure 2 schematically shows a scene 10 in a plan view, as can be recorded in particular by a camera 4, as shown in particular in Figure 1. The container 6 is not shown in Figure 2. The scene data that describe the scene 10 include depth information and color information, as represented here by white-colored objects 5 and black-colored packaging material. Furthermore, Figure 2 shows by way of example that the objects 5 are at least partially covered by packaging material 8 or embedded therein. The packaging material 8 is shown here in a simplified circular shape, but can have or assume any desired and in particular different shapes in embodiments; it can in particular be ice, packaging material made of plastic, packaging material made of natural materials, such as in particular wood, paper, cardboard or cellulose, etc.Furthermore, Figure 2 shows an unknown object 9 which was, for example, accidentally left in the container 6.

[0047] Figure 3 shows the same scene 10 as in Figure 2, with the difference that the packaging material 8 in the scene 10 has been identified, in particular classified. Accordingly, Figure 3 shows classification data that describe an affiliation with packaging material. This is indicated by the dashed lines of the packaging material 8. In embodiments, the classification can be carried out based on the color of the packaging material 8 or based on other criteria that characterize the packaging material 8. The unknown object 9 is not classified as packaging material 8 because, in particular, it does not have the properties, in particular a characteristic property, of the packaging material 8, such as in particular a certain (previously known) color and / or (previously known) shape.

[0048] Figure 4 schematically shows the same scene 10 as in Figure 1 or Figure 2, with the difference that Figure 5 shows a collision object, the filtering, in particular segmentation of the scene data, was determined based on the classification data. Furthermore, a determined handle position 11 is shown, which is or is described by handle position data. Furthermore, a handle position 1T on the unknown object 9 is shown as an example, which was determined based on the collision object. Here, it can be seen by way of example that the handle position 11' is set at a point on the unknown object 9 that is concealed by packaging material 8. Such a handle position 11' would have resulted in a collision with the methods customary in the prior art and would therefore not have been planned.

[0049] Figure 5 schematically shows a method 20 according to one embodiment as a block diagram. The determination of scene data S10 is carried out in particular by means of a camera directed at a scene, wherein the scene in embodiments comprises objects to be grasped with or in packaging material. Based on the determined scene data, classification data is determined S12, which describes an affiliation of the determined scene data to packaging material. S14 represents, by way of example, the determination of a collision object, which is determined based on the determined classification data using the determined scene data. S16 represents the determination of grip position data and / or movement data, which are determined based on the determined collision object.The method 20 may further comprise a step S18, which is shown in dashed lines in Figure 5, wherein S18 exemplifies a gripping based on the determined grip position data and / or a moving based on the determined movement data of the robot.

[0050] Although exemplary embodiments have been explained in the preceding description, it should be noted that numerous modifications are possible. Furthermore, it should be noted that the exemplary embodiments are merely examples and are not intended to limit the scope of protection, applications, or structure in any way. Rather, the preceding description provides the skilled person with a guide for implementing at least one exemplary embodiment, whereby various modifications, particularly with regard to the function and arrangement of the described components, can be made without departing from the scope of protection as it results from the claims and equivalent combinations of features.

[0051] List of reference symbols

[0052] 1 system

[0053] 2 gripper robots

[0054] 3 grippers of the gripping robot

[0055] 4 Camera

[0056] 5 Known objects

[0057] 6 containers

[0058] 7 Processing unit

[0059] 8 Packaging material

[0060] 9 Unknown object

[0061] 10 Scene

[0062] 11 Grip position on the known object

[0063] 11 ' Grip position on the unknown object

[0064] 20 procedures

[0065] S10 Determining scene data

[0066] S12 Determining classification data

[0067] S14 Determining a collision object

[0068] S16 Determining grip position data and / or determining movement data

[0069] S18 Gripping and / or moving the gripper robot

Claims

A method (20) for operating a gripping robot (2, 3), wherein at least one object (5, 9) to be gripped is at least partially located in packaging material (8) and / or is at least partially covered by it, in particular in a container (6) with packaging material (8), the method (20) comprising: - Determining (S10) scene data of a scene by means of a camera (4), wherein the scene data describe depth information and color information of a scene (10); - determining (S12) classification data by classifying the scene data, wherein the classification data describe an affiliation with packaging material (8); - Determining (S14) a collision object by filtering, in particular segmenting, the scene data based on the classification data; - Determining (S16) grip position data and / or movement data based on the collision object, wherein the grip position data describe at least one grip position (11, 1T) for the gripping robot (2, 3) on at least one object (5, 9) to be gripped, and wherein the movement data describe at least one movement path for at least part of the gripping robot (2, 3). Method (20) according to the preceding claim, characterized in that determining (S14) a collision object comprises removing the depth information based on the classification data. Method (20) according to one of the preceding claims, characterized in that the classification is carried out pixel-by-pixel and / or section-by-section based on the scene data, in particular based on the color information of the scene data.Method (20) according to one of the preceding claims 2 or 3, characterized in that the removal of the depth information is based on a mapping of 2D data to 3D data, in particular to the depth information. in particular by means of intrinsic parameters of the camera and / or extrinsic parameters. Method (20) according to one of the preceding claims, characterized in that the classification is carried out pixel-by-pixel using, in particular 2D, color information of the scene data. Method (20) according to one of the preceding claims, characterized in that the classification of the scene data is carried out using a convolutional neural network. Method (20) according to one of the preceding claims, characterized in that the determination (S16) of the grip position data and / or the movement data is additionally based on known objects in the scene (10), in particular on a CAD model of the object to be grasped and / or on a CAD model of the container in which the objects (5, 9) and the packaging material (8) are located.Method (20) according to one of the preceding claims, characterized in that the camera (4) is attached to the gripping robot (2, 3) or that the camera (4) is attached independently of the gripping robot (2, 3) with a view of the scene (10). System (1) for operating at least one gripping robot (2, 3), which is configured to carry out a method (20) according to one of the preceding claims, wherein the system (1) has at least one, in particular RGBD, camera (4) and at least one gripping robot (2, 3). Computer program or computer program product, wherein the computer program or computer program product contains instructions, in particular stored on a computer-readable and / or non-volatile storage medium, which, when executed by one or more computers or a system (1) according to claim 9, the computer(s) or the system (1). to carry out a method (20) according to one of claims 1 to 8.

Images