Picking System

Abstract

To provide a picking system capable of picking even when an object is not previously registered.SOLUTION: A picking system includes: a picking device for gripping an object; an RGB-D camera for acquiring three-dimensional point group data of the object to be picked by the picking device; and a control unit for controlling the picking device based on a detection result of the RGB-D camera. The control unit is so configured as to assemble a basic solid and generate a shape model of the object by referring the three-dimensional point group data, and calculate a gripping position of the object by the picking device based on the shape model.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a picking system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, a picking system including a picking device that grips a workpiece (object) and a control device that controls the picking device is known (see, for example, Patent Document 1).

[0003] The picking system of Patent Document 1 is configured to measure the three-dimensional shape of the workpiece using a distance sensor, and recognize the position and orientation of the workpiece by comparing the measurement result with a 3D CAD model of the workpiece. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-69542 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional picking system described above, a 3D CAD model needs to be registered in advance in order to recognize the workpiece, and there is room for improvement in this regard.

[0006] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a picking system that is capable of picking an object even if the object has not been registered in advance. [Means for solving the problem]

[0007] A picking system according to the present invention includes a picking device that grips an object, a distance sensor that acquires three-dimensional point cloud data of an object picked by the picking device, and a control device that controls the picking device based on the detection result of the distance sensor. The control device generates a shape model of the object by combining basic solids with reference to the three-dimensional point cloud data, A shape model of a plurality of types of objects and a gripping portion in each of the shape models are associated with each other and registered in advance, and the type of the object is identified using image data of the object, and the gripping portion of the registered shape model corresponding to the identified type of object is taken into consideration, The picking device is configured to calculate a gripping position of the object. If the type of object cannot be identified, the grasping position of the object is calculated based on a shape model of the object that is generated by combining the three-dimensional point cloud data with a basic solid. .

[0008] In this way, by generating a shape model of an object and calculating a gripping position, an object can be picked even if it has not been registered in advance.

[0009] In the above picking system, The control device is configured to identify the type of object using two-dimensional image data of the object, and if the type of object fails to be identified, the control device is configured to calculate the center of gravity position of the object when the density of the generated shape model is uniform as the grip position of the object. It may be configured as follows. Effect of the Invention

[0010] According to the picking system of the present invention, even if an object has not been registered in advance, it can be picked. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of a picking system according to an embodiment of the present invention. [Diagram 2] 2 is a diagram for explaining an example of a shape model registered in a control device of the picking system of FIG. 1. [Diagram 3] 13 is a flowchart for explaining a gripping position specifying operation in the picking system of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, one embodiment of the present invention will be described.

[0013] First, the configuration of a picking system 100 according to an embodiment of the present invention will be described with reference to FIGS.

[0014] The picking system 100 is configured to pick up an object (not shown) and perform, for example, automatic sorting or automatic transport. The picking system 100 is provided to pick up one object (object to be grasped) located in a preset predetermined area. As shown in FIG. 1, the picking system 100 includes a picking device 1, RGB-D cameras 2a and 2b, and a control device 3.

[0015] The picking device 1 is provided to grasp an object located in a predetermined area. For example, the picking device 1 includes a robot arm and a hand, which are not shown. The hand is provided at the tip of the robot arm and is configured to grasp an object. The robot arm is capable of controlling the position and posture of the hand by moving the hand.

[0016] The RGB-D cameras 2a and 2b are configured to capture an object located in a predetermined area to obtain an RGB-D image. The RGB-D image includes an RGB image (color image) and a depth image, and has depth information for each pixel in the RGB image. The RGB-D cameras 2a and 2b can convert the RGB-D image into three-dimensional point cloud data. The RGB image is an example of the "image data" of the present invention. The RGB-D camera 2a is an example of the "distance sensor" and "image sensor" of the present invention, and the RGB-D camera 2b is an example of the "distance sensor" and "image sensor" of the present invention.

[0017] The RGB-D cameras 2a and 2b are configured to capture images of an object from different angles. For example, the RGB-D camera 2a is provided to capture an image of an object located in a predetermined area from one side, and the RGB-D camera 2b is provided to capture an image of the object located in the predetermined area from the other side. That is, the two RGB-D cameras 2a and 2b are provided to prevent the outline of an object located in the predetermined area from being captured in a blind spot.

[0018] The control device 3 is configured to control the picking device 1 based on the imaging results of the RGB-D cameras 2a and 2b. The control device 3 includes a calculation unit 31, a storage unit 32, and an input / output unit 33. The calculation unit 31 is configured to execute calculation processing based on a program stored in the storage unit 32. The storage unit 32 stores a program for controlling the operation of the picking device 1. The picking device 1, the RGB-D cameras 2a and 2b, etc. are connected to the input / output unit 33. The input / output unit 33 is configured to output a control signal for controlling the operation of the picking device 1 and to input the imaging results of the RGB-D cameras 2a and 2b.

[0019] Here, the control device 3 is configured to calculate the gripping position of the object by the picking device 1 based on the imaging results of the RGB-D cameras 2a and 2b. This calculation of the gripping position makes it possible to appropriately pick up the object. Note that the storage unit 32 stores a program for calculating the gripping position of the object by the picking device 1, a DB (database) 32a used in the program, a trained model (not shown) described later, and the like.

[0020] In DB32a, an ID indicating the type of object, a shape model of the object, and a gripping location of the shape model are stored in association with each other. That is, in DB32a, the type ID of the object, the shape model, and the gripping location are set as columns (items), and a plurality of records are stored. The records are registered in DB32a in advance, for example, by a user. The shape model of an object is a three-dimensional schematic representation of the outer shape of the object, and is generated by combining a plurality of basic solids. The basic solids include, for example, a rectangular parallelepiped, a sphere, a cylinder, and a cone, and the orientation, dimensions, etc. are variable.

[0021] As a specific example, when the type of object is a "hammer", as shown in FIG. 2, a shape model Mh is generated using two cylinders C1 and C2, and a gripping location Gp in the shape model Mh is specified. The type ID of the object is registered by a user, the shape model Mh is generated by the user, and the gripping location Gp is ​​specified by the user. The gripping location Gp is ​​a portion suitable for the picking device 1 to grip the object, and as an example, it can be set to the center of gravity of the shape model Mh. Such records are registered in advance for a plurality of types of objects.

[0022] As shown in FIG. 1, the control device 3 is configured to store information (external parameters) related to the positions and attitudes of the RGB-D cameras 2a and 2b in advance, and to integrate the three-dimensional point cloud data obtained by the RGB-D camera 2a and the three-dimensional point cloud data obtained by the RGB-D camera 2b. The control device 3 is configured to generate a shape model of the object by combining basic solids with reference to the integrated three-dimensional point cloud data of the object. The basic solids include, for example, a rectangular parallelepiped, a sphere, a cylinder, and a cone, and their orientations and dimensions are variable. In other words, a shape model that approximates the three-dimensional point cloud data is generated by combining the basic solids while fitting them to the three-dimensional point cloud data. This shape model is a three-dimensional schematic representation of the outer shape of the object, and is composed of a plurality of basic solids.

[0023] The control device 3 is configured to identify the type of object using RGB images (two-dimensional image data) obtained by the RGB-D cameras 2a and 2b. The type of object is identified using a known trained model stored in the storage unit 32. When the control device 3 succeeds in identifying the type of object and the type of the object is registered in the DB 32a, the control device 3 is configured to calculate the grip position of the object based on the generated shape model in consideration of the grip position of the shape model of the object registered in the DB 32a. On the other hand, when the control device 3 fails to identify the type of object and when the type of the identified object is not registered in the DB 32a, the control device 3 is configured to calculate the grip position of the object based on the generated shape model.

[0024] The control device 3 is configured to control the picking device 1 so as to grip the object at the calculated gripping position of the object. That is, after the gripping position specifying operation described below is completed, the control device 3 causes the picking device 1 to perform the gripping operation at the gripping position of the object calculated in the gripping position specifying operation. That is, the control device 3 is configured to optimize the gripping position of the object in the picking operation by performing the gripping position specifying operation before the picking operation of the picking device 1 starts.

[0025] - Picking system's grip position identification operation - Next, a gripping position specifying operation in the picking system 100 according to the present embodiment will be described with reference to Fig. 3. This gripping position specifying operation is performed before the picking device 1 starts picking an object located in a predetermined area. Note that the following steps are executed by the control device 3.

[0026] 3, imaging results of the RGB-D cameras 2a and 2b are acquired. That is, an object located in a predetermined area is imaged by the RGB-D cameras 2a and 2b, and the imaging results are input to the input / output unit 33. The imaging results include an RGB image and three-dimensional point cloud data. Then, the three-dimensional point cloud data from the RGB-D camera 2a and the three-dimensional point cloud data from the RGB-D camera 2b are integrated.

[0027] Next, in step S2, the integrated 3D point cloud data is referenced to combine the basic solids to generate a shape model of the object. That is, by combining the basic solids while fitting them to the 3D point cloud data, a shape model that approximates the 3D point cloud data is generated. It is possible to improve the accuracy of the shape model not only by adding basic solids but also by removing basic solids. That is, a shape model is generated by adding or subtracting basic solids whose orientation, dimensions, etc. can be appropriately adjusted.

[0028] Next, in step S3, the type of object is identified using the RGB images (two-dimensional image data) from the RGB-D cameras 2a and 2b. This identification of the type of object is performed using a known trained model. For example, when the RGB image is input to the trained model, the type of object in the image is estimated and the likelihood of the estimation result is calculated.

[0029] Next, in step S4, it is determined whether or not the classification of the object type using the RGB image has been successful. For example, if the likelihood of the estimation result calculated in step S3 is equal to or greater than a predetermined value, it is determined that the classification has been successful. If it is determined that the classification has been successful, the process proceeds to step S5. On the other hand, if it is determined that the classification has failed, the process proceeds to step S7.

[0030] Next, in step S5, it is determined whether the type of the identified object is registered in DB 32a. If it is determined that the type of the identified object is registered in DB 32a, the process proceeds to step S6. On the other hand, if it is determined that the type of the identified object is not registered in DB 32a, the process proceeds to step S7.

[0031] Next, in step S6, the gripping position of the object is calculated based on the generated shape model, taking into consideration the gripping position in the shape model of the object registered in DB32a. For example, the shape model of the object generated in step S2 is compared with the shape model of the object registered in DB32a, and the gripping position of the object is calculated by applying the gripping position in the shape model of the registered object to the generated shape model of the object. That is, the gripping position of the object is calculated by applying the gripping position designated in advance by the user according to the type of object to the generated shape model. As a specific example, when the type of object is identified as "hammer" using an RGB image and the "hammer" is registered in DB32a, the gripping position of the object is calculated based on the shape model generated with reference to the three-dimensional point cloud data of the object, taking into consideration the gripping position Gp (see FIG. 2) in the registered shape model Mh.

[0032] In step S7, the gripping position of the object is calculated based on the generated shape model. For example, the position of the center of gravity when the density of the shape model of the object generated in step S2 is uniform may be calculated as the gripping position of the object. In addition, the center position of the largest of the basic solids constituting the shape model of the object generated in step S2 may be calculated as the gripping position of the object.

[0033] -effect- In this embodiment, as described above, a shape model of an object is generated by combining basic solids with reference to the three-dimensional point cloud data, and a gripping position of the object by the picking device 1 is calculated based on the shape model, so that the object can be picked even if it has not been registered in advance. That is, even if the type of the object fails to be identified and the type of the identified object is not registered in DB32a, the gripping position is calculated based on the shape model generated with reference to the three-dimensional point cloud data, so that the object can be appropriately picked.

[0034] In addition, in this embodiment, if the type of object is registered in advance, the gripping position of the object is calculated taking into account the gripping point in the shape model of the object, thereby improving the accuracy of picking.

[0035] Furthermore, in this embodiment, the type of object is identified using an RGB image (two-dimensional image data), so that the type of object can be easily identified.

[0036] -Other embodiments- It should be noted that the embodiments disclosed herein are illustrative in all respects and are not intended to be limiting. Therefore, the technical scope of the present invention is not interpreted solely by the above-described embodiments, but is defined by the claims. The technical scope of the present invention includes all modifications within the scope and meaning equivalent to the claims.

[0037] For example, in the above embodiment, an example in which two RGB-D cameras 2a and 2b are provided is shown, but the number of RGB-D cameras provided may be any number. For example, only one RGB-D camera may be provided, in which case the RGB-D camera may be attached to a robot arm. In this way, the object is captured from multiple viewpoints by the robot arm moving the RGB-D camera and capturing the image of the object.

[0038] Also 、R Instead of the GB-D camera, a distance sensor may be provided for acquiring three-dimensional point cloud data of an object.

[0039] In the above embodiment, the gripping point Gp is ​​set at the center of gravity of the shape model Mh, but the gripping point may be set at the center of the largest of the basic solids constituting the shape model. In other words, the gripping point can be freely specified by the user.

[0040] In addition, in the above embodiment, an example was shown in which three-dimensional point cloud data is input from the RGB-D cameras 2a and 2b to the control device 3, but this is not limited to the above, and the control device may calculate three-dimensional point cloud data based on the RGB-D images input from the RGB-D cameras.

[0041] In addition, in the above embodiment, the RGB-D cameras 2a and 2b may have an RGB image acquisition unit that acquires an RGB image and a depth image acquisition unit that acquires a depth image integrally provided in a single housing, or may each be provided in a separate housing.

[0042] -Additional Notes- A method for identifying a gripping position performed in a picking system including a picking device that grips an object, a distance sensor that acquires three-dimensional point cloud data of an object picked by the picking device, and a control device that controls the picking device based on a detection result of the distance sensor, comprising: acquiring, using the distance sensor, three-dimensional point cloud data of an object to be picked by the picking device; generating a shape model of an object by combining basic solids with reference to the three-dimensional point cloud data by the control device; and calculating, by the control device, the gripping position of the object by the picking device based on the shape model.

[0043] In the above-mentioned gripping position specifying method, the picking system further includes an image sensor for acquiring image data of an object to be picked by the picking device, and the control device is configured such that shape models of a plurality of types of objects and gripping positions in each shape model are registered in advance, acquiring image data of an object to be picked by the picking device using the image sensor; identifying, by the controller, a type of object using the image data; A method for identifying a gripping position, comprising a step of calculating, by the control device, the gripping position of the object by the picking device, taking into account the gripping point of a registered shape model of the identified object type.

[0044] A picking method comprising the above-mentioned gripping position specifying method.

[0045] A program for causing a computer to execute each step of the above-mentioned grip position specifying method. [Industrial Applicability]

[0046] The present invention can be used in a picking system that includes a picking device that grips an object and a control device that controls the picking device. [Explanation of symbols]

[0047] 1 Picking device 2a RGB-D camera (distance sensor, image sensor) 2b RGB-D camera (distance sensor, image sensor) 3. Control device 100 Picking System

Claims

1. A picking device that grasps an object; A distance sensor for acquiring three-dimensional point cloud data of an object to be picked by the picking device; A picking system including a control device that controls the picking device based on a detection result of the distance sensor, The control device includes: a shape model of an object is generated by combining basic solids with reference to three-dimensional point cloud data, a plurality of types of shape models of the object are associated with a gripping location in each of the shape models and registered in advance, the type of the object is identified using image data of the object, and a gripping position of the object by the picking device is calculated taking into consideration the gripping location of the registered shape model corresponding to the identified type of object, A picking system characterized in that, if the type of object cannot be identified, the system is configured to calculate the grasping position of the object based on a shape model of the object generated by combining the three-dimensional point cloud data with the basic solid.

2. The picking system according to claim 1, The control device includes: using two-dimensional image data of the object to identify a type of the object; A picking system characterized in that, if the type of object cannot be identified, the system is configured to calculate the center of gravity position when the density of the generated shape model is uniform as the grasping position of the object.

Images