Gripping with packaging material

Abstract

The invention relates to a method (20) for operating a gripping robot (2, 3), in which at least one object (5, 9) to be gripped is located at least partially in and / or at least partially covered by a packaging material (8), in particular in a container (6) having a packaging material (8), the method (20) comprising: determining (S10) scene data of a scene by means of a camera (4), wherein the scene data describes depth information and color information of the scene (10); determining (S12) classification data by classifying the scene data, wherein the classification data describes an attribution of the packaging material (8); determining (S14) collision objects by filtering, in particular segmenting, the scene data on the basis of the classification data; grasping position data and / or movement data are / is determined (S16) on the basis of the collision object, the grasping position data describing at least one grasping position (11, 11 ') for grasping the robot (2, 3) on the at least one object (5, 9) to be grasped, and the movement data describing at least one movement path for grasping at least part of the robot (2, 3).

Description

Technical Field

[0001] The present invention relates to a method for determining a grasping position, a system for operating at least one grasping robot, and a computer program or computer program product. Background Art

[0002] In "bin-picking" applications, a robot equipped with a gripper typically grasps an object from a container and places or drops the grasped object at a target location. To this end, a collision-free path is usually designed for the robotic arm to bring the gripper to a target position where it can successfully grasp the object by grasping.

[0003] In some application fields, objects are usually packed in containers containing deformable packaging materials for protecting the objects. In the food industry, for example, fresh fish is covered with ice, while in other application fields, fragile objects are protected using wood chips or polystyrene foam. This may pose a challenge to path planning algorithms because there may be no collision-free target position for the gripper that both avoids the packaging material and can successfully grasp the object. Summary of the Invention

[0004] The object of the present invention is particularly to improve this.

[0005] The object of the present invention is achieved by the teachings of the independent claims. Different embodiments and extensions of the present invention are given by the dependent claims.

[0006] According to an embodiment of the present invention, a method for operating a grasping robot is provided. In one embodiment, at least one object to be grasped by the grasping robot is particularly at least partially located in and / or at least partially covered by a packaging material, particularly in a container having a packaging material. According to an embodiment of the present invention, the method for operating the grasping robot includes the following steps: determining scene data of a scene by means of a camera, particularly an RGBD camera, wherein the scene data describes depth information and color information of the scene. In one embodiment, the depth information and color information of the scene can be determined by means of the camera, particularly by a stereo camera designed to capture depth information of the scene and a camera designed to capture color information of the scene. In one embodiment, means designed to capture depth information and color information, particularly can be combined in a camera, for example, particularly combined in an RGBD camera. According to one embodiment, the method further includes the following step: determining classification data, which is particularly determined by classifying the scene data. In one embodiment, the classification data describes the attribution of the packaging material In one embodiment, this can be determined by applying known classification methods. In one embodiment, the method further includes: determining a collision object by filtering, in particular segmenting, the scene data based on classification data, in particular for motion planning and / or grasping position planning. Thereby, in one embodiment, it can be achieved that the data contained in the scene data is less, for example, than before the filtering or when no collision object is determined. Furthermore, advantageously, it can be achieved thereby that starting from an at least substantially completely captured scene, the scene data is reduced by filtering, in particular before planning the grasping position and / or motion of the robot. In one embodiment, the method further includes: determining grasping position data and / or motion data, wherein, in one embodiment, the grasping position data describes at least one grasping position of the grasping robot on at least one object to be grasped, and the motion data describes at least one motion path or motion of at least a part of the grasping robot.

[0007] The term "scene" used herein is in particular understood as a snapshot (Momentaufnahme) of the (relevant) environment, which includes: a scene with objects, in particular containers, objects to be grasped, and / or packaging materials; dynamic elements, in particular at least a part of the robot; one or more fields of view of the camera and / or one or more states of the robot or the camera, and the relationships between these entities.

[0008] The term "camera" used herein is in particular understood as an imaging device for taking digital and / or three-dimensional images, which can have in particular at least one 3D camera and / or at least two spatially separated cameras and / or at least one scanner, preferably for three-dimensional scanning. In one embodiment, the scene data mentioned herein includes: depth information, preferably a point cloud, more preferably a three-dimensional point cloud, which can in particular be such depth information or consist of such depth information; and color information, wherein depending on the imaging device, the color information is represented as 2D information in the scene data, or can be 3D information as depth information, in particular corresponding to the points of the point cloud or corresponding to the points of the point cloud.

[0009] In some embodiments, in this way, during trajectory planning, in particular by determining the collision object, the packaging material can be taken into account, or in particular incorporated according to the determination of the motion data, in particular compared to the prior art, where the packaging material is usually ignored and the trajectory planning is (only) based on the detected objects and the pre-known objects, such as the container at the configured position.

[0010] In one embodiment, when processing the packaging material as described above, the scene described by the scene data can be advantageously defined as "positive" for the trajectory planner and / or the collision detector using the collision objects described herein. In contrast, in the prior art, typically an empty scene is started with and then known objects are added to the scene, in particular known objects identified in the container and / or the container itself, etc.

[0011] Advantageously compared to the prior art, by the embodiment described herein, the point cloud of the scene described by the scene data can be used as a collision object, since the collision object, in particular the point cloud of the collision object, does not (any longer) contain the packaging material, which can result in more possible grasping positions and / or trajectory planning, in particular compared to the prior art or compared to collision objects not determined based on classification data.

[0012] In one embodiment, the method includes, before determining the scene data, the step of adjusting the scene in the camera or the imaging device, in particular by correspondingly adjusting the robotic arm and / or moving and / or focusing the imaging device or the camera.

[0013] In one embodiment, the method includes the steps of moving the grasping robot based on the determined motion data; and / or performing a grasping based on the determined grasping position data, in particular using the grasping robot.

[0014] Advantageously hereby, compared to the prior art or compared to a situation where collisions with the packaging material are considered avoided, more (collision-free) grasping positions and / or more (collision-free) motion paths can be determined.

[0015] In some embodiments, the present invention is based on the following approach: the scene, in particular the entire scene, is described by (the determined) scene data, in particular including especially complete depth information or a point cloud, and the following part of the scene is removed from the scene data, namely the part that is considered not to cause potential collision problems during motion and / or grasping, in particular the packaging material in the container.

[0016] In some embodiments, the present invention is further based on the following approach: the depth information or the point cloud (of the remaining part) of the (collision object described herein) can be used for collision checking. Advantageously hereby, in one embodiment, the trajectory planning is (significantly) (more) robust, since the point cloud better or can better reflect the actual situation than a scene designed based on (only) (recognized) pre-known or pre-configured objects (such as in particular the CAD model and position of the container) or not fully recognized objects (in particular in the container).

[0017] In one embodiment, determining the collision object includes: removing depth information based on classification data, especially where the classification data indicates the attribution of the packaging material or where the scene data is classified as the packaging material.

[0018] Advantageously, in some embodiments, relevant objects for trajectory planning or motion planning can be (better) identified or considered. Thus, in some embodiments, collision checking can be improved. In particular, in some embodiments, the risk of not considering all relevant objects in the collision checking can be reduced. For example, although an object exists in the container but is not recognized by the object recognition device because it is partially or segmentally covered by the packaging material. Therefore, compared with the prior art, collisions with these objects can be advantageously reduced in the embodiments.

[0019] In one embodiment, the classification is performed pixel-by-pixel and / or segmentally according to the scene data, especially according to the color information of the scene data, where the color information of the scene data corresponds to a 2D image of the scene, or where the color information is associated with the corresponding depth information.

[0020] Advantageously, this enables the packaging material to be more easily recognized or classified. In addition, the scene data can also be (more) easily filtered, especially segmented.

[0021] Furthermore, in an embodiment, the segmentation of the packaging material can also be beneficial to the object recognition algorithm, which especially (must) estimate the position of the object in the container. In an embodiment, the algorithm includes a final step of position correction, which geometrically matches the object model (CAD and / or point cloud) with the scene represented by the scene data, especially the point cloud. In one embodiment, if the object in the collision object has been segmented by the packaging material, or the collision object (only) has scene data that is not classified as the packaging material, false point matches between the known object model and the packaging material in the scene can be (advantageously) reduced or avoided.

[0022] In one embodiment, removing the depth information is based on the mapping of the 2D data of the scene data to 3D data (English: “mapping”), especially the mapping from 2D color information to 3D depth information or point cloud, especially when the color information is 2D information or 2D data. In one embodiment, the classification can be performed pixel-by-pixel and / or segmentally according to the 2D color information of the scene data. In one embodiment, the mapping from 2D data to 3D depth information or point cloud can be based on the intrinsic parameters of the camera, such as especially the focal length, aperture, field of view, resolution or corresponding camera parameters, and / or the extrinsic parameters of the camera, such as especially the position and / or orientation, etc.

[0023] Thus, in some embodiments, it is possible to (advantageously) remove regions in the packaging material or 2D color information classified as packaging material from the depth information of the scene, such that the determined collision objects (at least substantially, only) include depth information that can be associated with known and / or unknown objects.

[0024] In some embodiments, the point cloud may also (preferably) contain unexpected objects, such as in particular random objects (left by a person) in a container. More preferably, the method described herein can be applied in embodiments to various, in particular all, objects in a container, such as in particular various fish or other objects with corresponding diversity.

[0025] In one embodiment, the packaging material can be filtered more reliably based on color information, such that the collision objects contain at least substantially only known objects, such as in particular a container containing an object and packaging material, components of a robot, such as in particular a gripper, especially depending on the camera installation, and / or an object to be grasped or an object that has been grasped but (not yet) recognized.

[0026] In one embodiment, this allows for a more robust determination of the grasping position, especially in the case where there are different objects in the container.

[0027] In one embodiment, the classification of the scene data is implemented by means of a convolutional neural network (CNN). Thus, in one embodiment, the classification can be performed more quickly, especially because the CNN only needs to be able to detect or classify one category.

[0028] Advantageously, the training of the CNN in the described embodiment is relatively simpler because in particular only one category needs to be recognized or classified, namely, the packaging material (such as in particular ice, wood chips) or the plastic material (such as in particular packaging pellets, such as polystyrene pellets, etc.).

[0029] In one embodiment, the determination of the grasping position data and / or the motion data is additionally based on known objects in the scene, in particular based on the CAD model of the object to be grasped and / or based on the CAD model of the container containing the object and the packaging material.

[0030] Thus, in one embodiment, it is advantageously possible to allow for a more robust determination of the grasping position data.

[0031] In one embodiment, the camera is fastened to the gripping robot and / or the camera is fastened independently of the gripping robot, in particular overlooking the scene. In one embodiment, a first camera and a second camera, in particular a second camera different from the first camera, can be used to determine the scene data. In one embodiment, the first scene data and the second scene data determined by the first and second cameras can be merged into scene data, which is then further processed in the manner described herein.

[0032] Thereby, in one embodiment, it is advantageously possible to allow for a (more) robust determination of the scene data.

[0033] One embodiment of the invention provides a system for operating at least one robot. In one embodiment, the system is designed to perform the method described herein. In one embodiment, the system includes at least one camera, in particular an RGBD camera, and at least one gripping robot. In one embodiment, the system and / or its devices further include means for determining scene data of the scene. In one embodiment, the system and / or its devices include means for classifying the scene data, in particular for determining classification data. In one embodiment, the system and / or its devices include means for determining collision objects. In one embodiment, the system and / or its devices include means for determining gripping position data and / or motion data.

[0034] Thereby, in one embodiment, it is advantageously possible to allow a position that would be classified as having a collision risk according to the prior art methods to be determined as a gripping position. In one embodiment, the gripper of the gripping robot can particularly advantageously enter the packaging material, and in particular a relatively better gripping position can be determined.

[0035] The system and / or device in the sense of the present invention can be designed in hardware technology and / or software technology, in particular having: at least one processing unit, in particular a digital processing unit, in particular a microprocessor unit (CPU), a graphics card (GPU), etc., preferably data-connected or signal-connected to a storage system and / or a bus system; and / or one or more programs or program modules. The processing unit can be designed for this purpose to: process the instructions of a program implemented in the storage system, collect input signals from the data bus, and / or send output signals to the data bus. The storage system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state, and / or other non-volatile media. The program can be provided such that it can embody or execute the method described herein, such that the processing unit can execute the steps of such method, and thus in particular can operate the robot.

[0036] In one embodiment, a computer program product may have, in particular, a computer-readable, non-volatile storage medium for storing a program or instructions, or a storage medium having a program or instructions stored thereon. In one embodiment, the execution of the program or the instructions causes a system or a controller, in particular a computer or an arrangement of multiple computers, to execute the program or the instructions, such that the system or the controller, in particular one or more computers, executes the method described herein or one or more of its steps, or the program or the instructions are designed for this purpose.

[0037] In one embodiment, one or more, in particular all, steps of the method are performed fully or partially automatically, in particular by the system or its devices. Description of the Drawings

[0038] Further advantages and features are given by the dependent claims and the embodiments. For this purpose, it is schematically shown:

[0039] Figure 1 A system according to an embodiment of the present invention;

[0040] Figure 2 Scene data according to an embodiment;

[0041] Figure 3 Classification data according to an embodiment;

[0042] Figure 4 A collision object according to an embodiment; and

[0043] Figure 5 The method according to an embodiment is shown in block diagram form. Detailed Description

[0044] Figure 1 System 1 is schematically shown, which has exemplary grasping robots 2, 3, and the grasping robot has a gripper 3. The gripper 3 is Figure 1 schematically shown as two fingers in [the figure], but in an embodiment, there may be more fingers, or it may be designed as other types of grasping devices for grasping, in particular picking up objects. In addition, scene 10 is also shown, which schematically shows known object 5 in container 6. Camera 4 is designed to acquire scene 10, in particular for determining scene data. For this purpose, in Figure 1 the exemplary illustration, camera 4 is installed independently of grasping robots 2, 3 and is connected to processing unit 7 in a data communication manner, and the processing unit is in turn connected to robots 2, 3 in a data communication manner. In an embodiment, processing unit 7 may be integrated in camera 4 and / or robots 2, 3. Object 5 in container 6 is wrapped in packaging material 8 (not shown here).

[0045] Figure 2 The scene 10 is schematically shown in a top view, and this scene can be particularly captured by a camera 4, as Figure 1 shown. The container 6 is not shown in Figure 2 . The scene data describing the scene 10 includes depth information and color information, as shown by the white object 5 and the black packaging material here. In addition, as Figure 2 exemplarily shown, the object 5 is at least partially covered by the packaging material 8 or enclosed in the packaging material. The packaging material 8 is simply represented by a circle here, but in an embodiment, it can have or take any shape, especially different shapes, and this packaging material can be particularly ice, plastic packaging material, natural material packaging material, such as especially wood, paper, cardboard, or cellulose, etc. In addition, in Figure 2 , an unknown object 9 is also shown, which is, for example, accidentally left in the container 6.

[0046] Figure 3 Shows the same scene 10 as Figure 2 , with the difference that: the packaging material 8 in the scene 10 has been determined, especially classified. Accordingly, in Figure 3 , the classification data describing the attribution of the packaging material is shown. This is represented by the corresponding shading of the encapsulation material 8. In an embodiment, the packaging material 8 can be classified according to its color or other features characterizing the packaging material 8. The unknown object 9 is not classified as the packaging material 8 because it especially does not have the attributes, especially the characteristic attributes, of the packaging material 8, such as especially a specific (pre-known) color and / or (pre-known) shape.

[0047] Figure 4 Schematically shows the same scene 10 as Figure 1 or Figure 2 , with the difference that: Figure 5 Based on the classification data, the scene data is filtered, especially segmented, to determine the collision object. In addition, the determined grasping position 11 is also shown, and this grasping position is described by the grasping position data. In addition, a grasping position 11' on the unknown object 9 is also exemplarily shown, which is determined based on the collision object. Here, for example, it can be seen that the grasping position 11' is on the part of the unknown object 9 covered by the packaging material 8. In the method according to the prior art, such a grasping position 11' may cause a collision, so it will not be planned.

[0048] Figure 5A flowchart of method 20 according to an embodiment is schematically shown. Determining the scene data S10 is particularly performed by aligning a camera with the scene, which in the embodiment includes an object to be grasped, having a packaging material or within a packaging material. Classification data S12 is determined based on the determined scene data, and the classification data describes the attribution relationship between the determined scene data and the packaging material. S14 exemplarily shows determining a collision object, which is determined based on the determined classification data and according to the determined scene data. S16 describes determining the grasping position data and / or the motion data based on the determined collision object. Method 20 may further include step S18 shown in dashed lines in Figure 5 where S18 exemplarily describes the grasping based on the determined grasping position data and / or the motion of the robot based on the determined motion data.

[0049] Although exemplary embodiments have been set forth in the foregoing description, it should be noted that there may be many variations. It should also be noted that the exemplary embodiment is merely an example and should not form any limitation on the scope of protection, application, and construction. On the contrary, the foregoing description can impart to those skilled in the art the teaching of implementing the conversion of at least one exemplary embodiment, wherein various changes can be made, particularly regarding the functions and arrangements of the components, without departing from the scope of protection of the present invention. For example, various changes can be obtained according to the claims and their equivalent combinations of features.

[0050] List of reference numerals

[0051] 1 System

[0052] 2 Gripping robot

[0053] 3 Gripper of the gripping robot

[0054] 4 Camera

[0055] 5 Known object

[0056] 6 Container

[0057] 7 Processing unit

[0058] 8 Packaging material

[0059] 9 Unknown object

[0060] 10 Scene

[0061] 11 Gripping position on the known object

[0062] 11’ Gripping position on the unknown object

[0063] 20 Method

[0064] S10 Determine scene data

[0065] S12 Determine classification data

[0066] S14 Determine collision objects

[0067] S16 Determine grasping position data and / or determine motion data

[0068] S18 The robot grasps and / or moves.

Claims

1. A method (20) for operating a grasping robot (2, 3), wherein, At least one object to be grasped (5, 9) is at least partially located in the packaging material (8) and / or is at least partially covered by the packaging material, in particular in a container (6) having the packaging material (8), wherein the method (20) comprises: - Determining (S10) scene data of the scene by means of a camera (4), wherein the scene data describes depth information and color information of the scene (10); - Determining (S12) classification data by classifying the scene data, wherein the classification data describes the attribution of the packaging material (8); - Determining (S14) collision objects by filtering, in particular segmenting, the scene data based on the classification data; - Determining (S16) grasping position data and / or motion data based on the collision objects, wherein the grasping position data describes at least one grasping position (11, 11') of the grasping robot (2, 3) on at least one object to be grasped (5, 9), and wherein the motion data describes at least one motion path for at least a part of the grasping robot (2, 3).

2. The method (20) according to the preceding claim, characterized in that, The determining (S14) of the collision objects includes removing the depth information based on the classification data.

3. The method (20) according to any one of the preceding claims, characterized in that, The classification is performed pixel by pixel and / or segment by segment according to the scene data, in particular according to the color information of the scene data.

4. The method (20) according to claim 2 or 3 above, characterized in that, The removal of the depth information is based on a mapping from 2D data to 3D data, in particular to the depth information, in particular by means of the intrinsic parameters and / or extrinsic parameters of the camera.

5. The method (20) according to any one of the preceding claims, characterized in that, The classification is performed pixel by pixel according to the color information of the scene data, in particular 2D color information.

6. The method (20) according to any one of the preceding claims, characterized in that, The classification of the scene data is performed by means of a convolutional neural network.

7. The method (20) according to any one of the preceding claims, characterized in that, The determination (S16) of the grasping position data and / or the motion data is additionally based on known objects in the scene (10), in particular based on the CAD model of the object to be grasped and / or based on the CAD model of the container, wherein the object (5, 9) and the packaging material (8) are located in the container.

8. The method (20) according to any one of the preceding claims, characterized in that, The camera (4) is fastened to the grasping robot (2, 3), and / or the camera (4) is fastened independently of the grasping robot (2, 3) and overlooks the scene (10).

9. A system for operating at least one gripping robot (2, 3), the system being designed to carry out the method (20) according to any one of the preceding claims, wherein, The system (1) has at least one camera, in particular an RGBD camera (4), and at least one grasping robot (2, 3).

10. A computer program or computer program product, wherein, The computer program or computer program product contains instructions, in particular instructions stored on a computer-readable and / or non-volatile storage medium, which when executed by one or more computers or by the system (1) according to claim 9 cause the one or more computers or the system (1) to execute the method (20) according to any one of claims 1 to 8.

Images