METHOD FOR CREATING AN OBJECT MODEL FOR GRIPPING AN OBJECT, COMPUTER-READABLE STORAGE MEDIUM AND ROBOT SYSTEM

Description

[0001] The invention relates to a method for controlling a first and second manipulator of a robot system for gripping an object, a computer-readable storage medium and a robot system.

[0002] Grasping objects and using robotic systems is difficult.

[0003] In particular, when a robot system has to grasp an object in so-called unstructured work environments, errors can occur when grasping the object. An unstructured work environment is characterized by objects being located in unknown locations and with unknown orientations.

[0004] In such an unstructured work environment, where no information about the objects to be grasped is available, so-called model-free methods are typically used. A typical approach involves the use of suction grippers in conjunction with image recognition methods that segment grasping regions from camera images. An RGB camera or a stereo camera can be statically positioned above the objects. However, these methods are error-prone and slow, i.e., computationally intensive.

[0005] Better results are achieved by using so-called model-based methods, in which the geometric data of an object is known. Knowledge of the geometry and appearance of objects is exploited to recognize them, estimate their position, and select grasp points.

[0006] Modern warehouses and distribution centers are increasingly based on robot- and data-based intralogistics solutions for automating material and goods flows, picking customer orders, or inventorying stock levels. An important step in the logistics chain is the automated handling of small parts. The automated handling of small parts is particularly necessary when singling or picking items for a customer order, such as when ordering online. These small parts—hereinafter also referred to as objects—are typically stored in an unstructured manner, for example, in bulk, in storage boxes, which places high demands on automated handling of small parts.

[0007] To enable the most error-free gripping process possible, which is robust against process errors, warehouse management maintains databases with data on the goods to be processed. These contain, for example, information on identification numbers, descriptions, size, shape, weight, texture, and / or other details. However, these databases are often incomplete. Geometric information required for the automated recognition and handling of the warehouse's goods, for example with a multi-axis robot, is usually missing. Such geometric information can be available as 3D models, CAD data, 3D textures, possible gripping points, or similar. In existing systems, such data is often available for less than 1% of the goods. Therefore, in practice, it is often not possible to use model-based gripping methods.Instead, model-free methods are used, which have the disadvantages described above.

[0008] WO 2014 / 188 177 A1 describes a method for improving the gripping of objects using a manipulator. When gripping a training object, sample gripping data is generated, for example, through manual or virtual training.

[0009] EP 1 589 483 A2 describes a model-based grasping method intended to improve bin picking with overlapping objects. EP 1 589 483 A2 describes that, to create an object model, grasping a reference object is first trained. The grasping of the reference object is recorded by a perception system, and corresponding model data is stored for subsequent use.

[0010] The publication DETRY R ET AL: "Learning object-specific grasp affordance densities," IEEE 8TH INTERNATIONAL CONFERENCE ON DEVELOPMENT AND LEARNING, 2009; June 5, 2009, pages 1-7, describes how objects can be grasped using partial models, even when the objects are not fully visible. This involves using a fully known 3D model.

[0011] The publication UNO Y ET AL: "A Computational Model for Recognizing Objects and Planning Hand Shapes in Grasping Movements", NEURAL NETWORKS, ELSEVIER SCIENCE PUBLISHERS, BARKING, UK, Vol. 8, No. 6, 1 January 1995, pages 839-851 describes a method for learning possible grasping poses and, based on these, creating models with which new grasping poses can be generated.

[0012] It is therefore an object of the present invention to improve the gripping of objects when a model-based method for gripping the objects is not applicable.

[0013] This object is achieved by a method according to claim 1, by a computer-readable storage medium according to claim 7 and by a robot system according to claim 8.

[0014] In particular, the object is achieved by a method for controlling a first and second manipulator of a robot system for gripping an object, in particular for a first robot system, comprising the following steps: Receiving a first visual object representation that specifies at least a first object to be grasped; determining a gripping strategy using the first object representation with a model-free gripping method; controlling the first manipulator to grip the first object using the gripping strategy; creating an object model using the first object representation for model-based gripping of the object; determining a quality indication associated with the first object representation that specifies the quality of the gripping using the gripping strategy and includes an indication of the number of visual features of the first object representation and contains an indication of whether the gripping of the object was successfully carried out; storing the object model, the object representation, and the quality indication;Comparing the determined quality indication with a stored quality indication associated with a second stored object representation;

[0015] According to the invention, depending on the comparison, both an updating of the stored object model using the first and second object representations and a control of the second manipulator for gripping at least one second object using the object model are provided according to the invention, likewise according to the invention using a model-based gripping method and using the object model, wherein the first object and the second object are of the same type.

[0016] For the purposes of this application, a manipulator can be understood as a multi-axis robot, in particular a six-axis robot, a seven-axis robot, or a two-axis robot. However, it is also intended that a manipulator refers solely to the end effector of a robot arm. A manipulator can generally refer to a device that can be designed to change the position and / or orientation of an object. This includes, in particular, parallel grippers or vacuum grippers.

[0017] In the context of this application, controlling or activating a manipulator can be understood as controlling the manipulator in such a way that the manipulator changes its position or orientation in a Cartesian coordinate system. Furthermore, controlling or activating a manipulator can be understood as closing or opening a parallel gripper, particularly taking into account the force to be applied. Furthermore, controlling or activating can include activating or deactivating a vacuum gripper, whereby the negative pressure to be generated can also be adjustable.

[0018] The individual method steps of the described method are carried out in the order described above and defined in the claims.

[0019] A core of the invention is that it is initially possible to grasp objects using a model-free method. The data recorded during grasping using the model-free method, i.e., an object representation, is used to create an object model. Once an object model has been created, further grasping can be performed using the object model.

[0020] This makes it possible to begin grasping objects even without an existing object model. Once an object model is created, a safer or more robust grasping method can be used. By using a model-based grasping method whenever possible, computing resources are conserved and process reliability is increased.

[0021] Furthermore, it allows a first manipulator to collect data using a model-free method to create an object model. Once the object model is created, in one embodiment, the object model can be transmitted to a plurality of manipulators so that a plurality of manipulators can make use of the object model.

[0022] This allows the knowledge gathered by a single manipulator to be shared with multiple manipulators. Overall, this can significantly improve the processing of gripping orders in a warehouse.

[0023] An object or small part has dimensions that are less than 1 meter x 1 meter, preferably less than 50 centimeters x 50 centimeters or particularly preferably less than 20 centimeters x 20 centimeters.

[0024] In one embodiment, an object representation is specified by a 3D point cloud. In another embodiment, a 2D representation, for example, a camera image, may represent an object representation.

[0025] According to the invention, the method comprises storing at least the object model, the quality specification, and the object representation in a database. Storing the object model or the object representation in a database enables the data to be stored in a structured manner. Furthermore, a large number of different entities can access the data. Furthermore, it is made possible for the database to be located at a remote location from the manipulator. In one embodiment, the manipulator and the database can be communicatively linked via the Internet. However, it is also possible for the database and the manipulator to be connected directly to one another in an intranet or via a wired or wireless connection. Storing the object representation can be implemented by storing mesh data.The method further comprises updating the stored object model using the object representation.

[0026] Thus, it is not only possible to create an object model from an object representation, but it is also possible to update an object model using an object representation. This has the advantage that an object model can be continuously improved. In particular, it is possible to continuously improve an object model until it is advantageous to use a model-based grasping method. Thus, a switch from a model-free grasping method to a model-based grasping method can be implemented.

[0027] In the preferred embodiment, the first object and the second object are of the same type and have substantially identical geometric dimensions .Furthermore, they may have substantially identical weights. In one embodiment, the first and second objects may have identical appearances. In particular, the first and second objects may have identical visual features. . It is further conceivable that the first and second objects have a color histogram and / or a texture that may be at least substantially identical with respect to a metric.

[0028] For example, the first and second objects can be of the same product type.

[0029] If the first and second objects are of the same type, the object model can be updated each time the manipulator grasps the object, since each grasp can be assigned to the same object type. Complex object recognition is eliminated in this case.

[0030] In the context of this application, the presence of substantially identical geometric dimensions or weights and metrics means that deviations are within a tolerance interval, in particular less than 10%, preferably less than 5%, particularly preferably less than 1%.

[0031] The method further comprises determining a quality indication associated with the first object representation that indicates the quality of the grasping using the grasping strategy, wherein storing comprises storing the quality indication.

[0032] A quality indication can contain an indication of whether a grasp was successfully executed using the grasping strategy. It is thus possible to determine whether a grasping strategy worked well or not. For example, the quality indication can be represented by an integer. In one embodiment, a value of "0" could indicate that the object was dropped during grasping. A value of "10" could indicate that the grasping of the object worked very well. A quality indication can therefore lie within a predefined interval. It is also conceivable to define a confidence in the confidence interval 0 to 1, where a quality indication with a confidence value of 0 could indicate that the object was dropped during grasping, and a confidence value of 1 could indicate that the grasping worked very well.

[0033] In one embodiment, a grasping strategy may include an approach pose and / or at least one grasping point. A grasping point may be represented as a pose, ie, as a six-dimensional data point.

[0034] Furthermore, the method comprises comparing the determined quality information with a stored quality information associated with a second object representation, wherein the updating of the stored object model and the control of the second manipulator are carried out depending on the comparison.

[0035] The described embodiment provides a kind of filter function that allows only those object representations that have led to good results to be saved. This saves storage space and creates a better object model, allowing for better grasping operations using the object model.

[0036] In one embodiment, the method may comprise checking whether a check criterion is met with respect to the object model, wherein the check criterion may in particular indicate whether the object model will lead to better gripping results than the model-free method, wherein the control is carried out with a model-based method if the check criterion is met.

[0037] A test criterion can be used to switch from a model-free grasping method to a model-based grasping method. This test criterion can be met, for example, when no new object representations are created or recorded. Another test criterion can be to check whether grasping can be successfully performed using a model-based method with the object model in a simulation.

[0038] One advantage of the described embodiment is that the model-based grasping method is only used when the generated object model has sufficient quality. This ensures that the model-based grasping method delivers better results than the model-free grasping method.

[0039] In one embodiment, a / the object model may be formed as a CAD model, in particular a CAD model generated by a registration and / or stitching-based method, as a set of individual views and / or as a 3D point cloud.

[0040] There are several different ways the object model can be represented. Particularly good results are achieved when the object model is available as a CAD model. Such a CAD model can be created, for example, by combining a large number of different object representations, so-called registration and / or stitching methods.

[0041] However, it is also possible that the object model simply consists of a multitude of different views. When grasping with a model-based grasping method, a view can be captured and compared with the views stored in the database. If a similar view is already stored, a grasping strategy can be used, similar to the one used when the view stored in the database was created.

[0042] Another option is to store multiple 3D point clouds. It is also possible to store a single 3D point cloud, where the data from a variety of different object representations can be combined into one 3D point cloud.

[0043] The specific quality specification includes an indication of the number of visual features, in particular edges, of the object representation.

[0044] In one embodiment, the first object representation may be stored only if the comparison shows that the first object representation contains more visual features than the second object representation.

[0045] One possible indicator of the quality of an object representation is the number of visual features. A visual feature can be specified, for example, by the number of edges or the number of vertices.

[0046] The quality information contains an indication of whether a grasp of one and / or the object was successfully executed.

[0047] In one embodiment, the method may include calculating a checksum for the first object representation, wherein the checksum may indicate a version of the object representation.

[0048] It is therefore also possible to implement versioning of the object representation. This can be advantageous when multiple systems execute the described procedure in parallel and store their data in a shared database.

[0049] It is also possible that the generated object model has a specific version, which can also be determined by calculating a hash value.

[0050] For example, by versioning an object model, a system that already stores an initial version of the object model locally can query the database to see if a new version of the object model is available. If a new version of the object model is available, this version of the object model can be downloaded and used for model-based grasping.

[0051] In one embodiment, the method may comprise training a classifier, in particular an artificial neural network, using the grasping strategy, the object model and / or the first and / or second object representation.

[0052] All data stored in the database can be used to train a classifier. The data can be used as "labeled" so-called "ground truth" data for training. This allows the object representation, the manipulator used, and the grasping strategy to be used as the classifier's input. This eliminates the need for time-consuming manual labeling of data.

[0053] In a further embodiment, the method may include identifying objects in a target area using the object model and comparing the identified objects with a database.

[0054] With the embodiment described above, the objects in a target area can be compared with the inventory data in a warehouse management system. This allows for a fully automated check to determine whether the data stored in the warehouse management system matches the objects actually present. The object model can be used for this purpose, as it allows objects to be clearly assigned to an object type.

[0055] The object is also achieved in particular by a computer-readable storage medium which contains instructions which cause at least one processor to implement a method as described above when the instructions are executed by the at least one processor.

[0056] Similar or identical advantages arise as those already described with regard to the process.

[0057] The task is also solved in particular by a robot system comprising: at least one perception device for capturing an object representation of an object; a manipulator configured to grasp the object; a control device having a storage device as described above, and a computing device.

[0058] The at least one perception device can be designed, for example, as a 3D camera, an RGB camera, an ultrasonic sensor, a laser sensor, a LIDAR system or even as a radar system.

[0059] Similar or identical advantages arise as those already described with regard to the process.

[0060] Further embodiments emerge from the subclaims.

[0061] The invention is explained in more detail below using exemplary embodiments and figures. These show: Figure 1: A robot grasping objects from a box; Figure 2: A 2D representation of the box contents and the processing of the 2D representation; Figure 3: An example database; Figure 4: A flowchart for switching from model-free to model-based control of the robot; Figure 5: Combining different object representations into an object model; Figure 6: A distributed robot system.

[0062] Figure 1 shows a robot 1, which in the illustrated embodiment is designed as a multi-axis robot. The robot 1 has a vacuum gripper or suction gripper 4. A compressor, not shown, is provided to generate a negative pressure. The robot 1 is arranged next to a box 2 in which the two objects 3, 3' are located.

[0063] A 3D camera system 10, which has two cameras 11, 11', is arranged above the box 2. The 3D camera system 10 is designed to generate a 3D point cloud. The 3D camera system 10 is arranged such that it can generate a 3D point cloud of the box 2. In the illustrated embodiment, the 3D camera system 10 is arranged approximately 1 m above the box 2.

[0064] Objects 3, 3' are present in an unstructured state in box 2. This means that no information about their position or arrangement is known a priori. It is the task of robot 1 to place objects 3, 3' in box 2 onto conveyor belt 5 located next to robot 1.

[0065] The processing of the 3D representation generated by the camera system 10 is simplified into Figure 2 shown.

[0066] The Figure 2shows a 2D representation 21 of the contents of box 2 from above. Thus, in the 2D representation 21, object representations 23, 23' of objects 3, 3' can be recognized.

[0067] To process the 2D representation 21, the 2D representation 21 is first segmented. In doing so, the object representations 23, 23' must be recognized and extracted from the 2D representation 21. Objects can often be distinguished from the background of the box 2 simply by virtue of their different color. General contrast differences and, if 3D information is available, particularly strong changes in the gradient, i.e., sudden and strong changes in distance, can also indicate the transition from object 3, 3' to the box floor. It is also possible to record a first image of the box 2 before a grasping action and another image after the grasping action. Thus, by overlaying the recorded images, it can be determined where the grasped object 3, 3' was before the grasping action and what geometric dimensions it has.Using the two recorded images, the object representation 23, 23' can thus be efficiently extracted from the 2D representation 21.

[0068] During segmentation, so-called bounding boxes 22, 22' are drawn around the object representations 23, 23'. Further processing of the object representations 23' is described below. After the object representation 23' has been extracted, a grasping strategy 24 can be determined for the associated object 3' using a model-free method. This means that a grasping point 25, an approach trajectory, and the suction pressure of the manipulator 4 are determined.

[0069] The object 3' is grasped using the specific grasping strategy 24 and placed on the conveyor belt 5. It is stored whether the grasping with the grasping strategy 24 worked the first time and also how long the grasping and / or placing took.

[0070] Furthermore, the number of visual features contained in the bounding box 22' is determined. The number of edges is determined and counted. Edge determination can be performed using well-known methods, such as a Sobel operator, a Laplace filter, the Canny algorithm, or the Roberts operator.

[0071] A quality score is determined from the number of determined visual features. The bounding box 22', which contains the object representations 23', is stored in database 6 along with the determined quality score. Furthermore, the information regarding the grasping strategy 24 is also stored in database 6.

[0072] The database 6 can be located in a data center remote from the robot 1.

[0073] Figure 3shows an exemplary storage of data in database 6. An example table 8 of database 6 is shown.

[0074] The database table 8 shown stores the grippers 4 used, the recorded representation 21, the quality of the representation and the quality of the gripping strategy Q1, Q2, as well as a used approach pose P1, P2 for specific object types 3. Only two rows are shown in the database table 8 shown. In the illustrated embodiment, the information stored in the two rows of table 8 together form an object model 9.

[0075] The Figure 4 shows a flowchart indicating a method for storing an object representation 23 in the database 6.

[0076] First, a quality specification Q2 of the object representations 23 is determined by a quality determination device 7. Before the object representations 23 can be stored in the database 6 together with the quality specification Q2, a check is first performed to determine whether a better object representation is already stored in the database 6.

[0077] For this purpose, a quality specification Q1 is retrieved from database 6, which is assigned to another object representation of the same object type. The two quality specifications Q1 and Q2 are then compared. Only if the quality specification Q2 is better than the quality specification Q1 is the object representation 23 stored together with the quality specification Q2 in database 6, whereby the quality specification Q1 and the associated object representation are deleted. In the preferred embodiment, the quality specifications Q1 and Q2 indicate the number of visual features in the associated object representations. A simple ">" comparison can therefore be performed, whereby a quality specification is considered better if the number of visual features is greater.

[0078] Figure 5 shows a further example of how an object model 9 can be generated from individual object representations 23, 23'. In the embodiment shown, the Figure 5a plurality of object representations 23, 23' are combined by means of a so-called stitching process in such a way that a program can calculate a CAD model 9.

[0079] Figure 6 shows an embodiment in which a total of three robots 1, 1', 1", 1‴ at different locations carry out the method described above. All robots 1, 1', 1" each send object representations 23, 23', 23" to a single database 6. All participating robots 1, 1', 1", 1‴ grasp objects 3, 3' of the same type, so that an assignment is not possible.

[0080] With the described embodiment, a good object model 9 can be generated in a very short time, so that after a short period of time it is possible to switch from model-free gripping to model-based gripping.

[0081] A fourth robot 1‴ can now download the object model 9 and perform a gripping of objects 3, 3' using a model-based method.

[0082] Figure 6 shows a distributed system in which robots 1, 1', 1" grasp objects using model-free methods. Each of the robots 1, 1', 1" creates an object representation 23, 23', 23", each of which is stored in a database 6. Using the stored object representations 23, 23', 23", an object model 9 is created, as already described in detail above.

[0083] In the embodiment of the Figure 6 another robot 1‴ that is newly added to the system can download the created object model 9 from the database 6 and thus immediately begin to grasp objects using a model-based gripping method.

[0084] The robot 1‴ can be provided at a completely different location than the robots 1, 1', 1". For example, the creation of the object model 9 can be carried out at a first warehouse location. The robot 1‴ can be connected to the database 6 via the Internet and can thus be located at any location.

[0085] At this point, it should be noted that all parts described above and defined in the independent claims are claimed as essential to the invention. Modifications to these are familiar to those skilled in the art. However, the actual scope of the invention is determined solely by the appended claims. List of reference symbols:

[0086] 1, 1', 1", 1‴Robot 2Box 3, 3'Object 4Gripper / Suction gripper 5Conveyor belt 6Database 7Quality determination device 8Database table 9Object model 103D camera system 11, 11'Camera 212D / 3D representation 22, 22'Bounding box 23, 23', 23‴Object representation 24Gripping strategy 25Gripping point Q1, Q2Gripping quality P1, P2Pose

Claims

1. Method for controlling first and second manipulators (1, 1') of a robot system for gripping an object (3, 3'), comprising the following steps of: - receiving a first visual object representation (23) indicating at least a first object (3) to be gripped; - determining a gripping strategy (24) using the first object representation (23) with a model-free gripping method; - controlling the first manipulator (1) to grip the object (3, 3') using the gripping strategy (24); - creating an object model (9) using the first object representation (23) for model-based gripping of the first object (3); - determining a quality indication (Q1) associated with the first object representation (23), which indicates the quality of the gripping using the gripping strategy (24) and includes an indication of the number of visual features of the first object representation (23) and an indication of whether the gripping of the object (3) has been successfully carried out; - storing the object model (9), the object representation (23) and the quality indication (Q1) in a database (6); - comparing the determined quality indication (Q1) with stored quality indication (Q2) associated with a second stored object representation (23'); as a function of the comparison: ∘ updating the stored object model (9) using the first and second object representations (23, 23'); ∘ controlling the second manipulator (1') to grip at least one second object (3') using a model-based gripping method and using the object model (9), wherein the first object (3) and the second object (3') are of the same type and thus have at least substantially identical geometric dimensions and / or the same visual features.

2. Method according to one of the preceding claims, characterized by checking whether a test criterion with respect to the object model (9) is fulfilled, wherein control is carried out with a model-based method if the test criterion is fulfilled.

3. Method according to one of the preceding claims, characterized in that the object model (9) is in the form of a CAD model, as a set of individual views and / or as a 3D point cloud.

4. Method according to one of the preceding claims, characterized by calculating a checksum for the first object representation (21), wherein the checksum indicates a version of the object representation (21).

5. Method according to one of the preceding claims, characterized by training a classifier using the gripping strategy, the object model and / or the first and / or second object representation.

6. Method according to one of the preceding claims, characterized by identifying objects in a target area using the object model and comparing the identified objects with a database.

7. Computer-readable storage medium containing instructions that cause at least one processor of a robot system comprising two manipulators to implement a method according to one of the preceding claims when the instructions are executed by the at least one processor.

8. Robot system, comprising: - a perception device (10) for capturing an object representation (23, 23') of an object (3, 3'); - two manipulators (1, 1') configured to grip the object (3, 3'); - a control device having a memory device and a computing device, wherein the control device is configured to carry out a method according to one of claims 1 to 6.