Picking robot device

Abstract

To provide a picking robot device that is capable of reducing errors in placing in a placing work where a picked up object is placed at a certain position.SOLUTION: A picking robot device 1 using machine learning comprises two robot arms 2A, 2B. The two robot arms 2A, 2B can shift an object 10 between one robot arm and the other robot arm. The picking robot device 1 comprises: a camera 3 for acquiring a two-dimensional image and point cloud data for the object 10; and a control unit 4. The control unit 4 determines whether there is a path that allows the object to be placed in a posture as required for placing work at a gripping point P1 for picking, on the basis of the two-dimensional image and point cloud data acquired by the camera 3. If there is no such path, the control unit makes the picking robot device learn the gripping point P2 for placing and a path that allows gripping of the gripping point for placing while having the gripping point P1 for picking being gripped.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a picking robot device.

Background Art

[0002] Patent Document 1 describes a technique for determining whether an object can be grasped based on three-dimensional point cloud data included in a hand passage region when grasping a plurality of objects stacked in a box with a hand of a robot arm.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in conventional methods such as Patent Document 1, during the placement operation of placing the object grasped (picked) by the robot arm (picking robot device) at an arbitrary location, the picking robot device cannot change the object being picked in the air. For this reason, in the placement operation, there is room for improvement regarding the situation where the object cannot be placed at a desired location, that is, the occurrence of an error during placement.

[0005] An object of the present disclosure is to provide a picking robot device that can reduce errors during placement in the placement operation of placing a picked object at an arbitrary location.

Means for Solving the Problems

[0006] A picking robot device according to an aspect of an embodiment of the present invention is a picking robot device that picks up an object and performs a placing operation of placing the picked object at an arbitrary location. The picking robot device includes two robot arms each mounted with a hand capable of gripping the object. One of the robot arms grips the object with the hand at a picking grip point, which is a position for gripping the object when picking up the object, and performs picking. The other robot arm is configured to grip the object with the hand at a placing grip point, which is a position for gripping the object when placing the object at the arbitrary location, so that the object can be transferred between the one robot arm and the other robot arm. The picking robot device further includes a camera that acquires a two-dimensional image and point cloud data of the object, and a control unit that determines whether there is a path along which the object can be placed in the posture as instructed by the placing operation at the picking grip point based on the two-dimensional image and the point cloud data acquired by the camera. When there is no such path, the control unit learns the placing grip point and a path along which the placing grip point can be grasped while holding the picking grip point.

Advantages of the Invention

[0007] According to the present disclosure, it is possible to provide a picking robot device capable of reducing errors during placement in a placing operation of placing a picked object at an arbitrary location.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described with reference to the accompanying drawings. To facilitate understanding of the description, the same reference numerals are used for the same components in each drawing as much as possible, and redundant descriptions are omitted.

[0010] FIG. 1 is a schematic configuration diagram of a picking robot device 1 according to an embodiment. The picking robot device 1 is a device that picks (grasps) an object 10 and performs a placement operation of placing the picked object 10 at an arbitrary location.

[0011] As shown in FIG. 1, the picking robot device 1 includes two robot arms 2A and 2B, a camera 3, and a control unit 4.

[0012] The two robot arms 2A and 2B have, for example, six degrees of freedom, and each is equipped with a hand capable of grasping the object 10 as an end effector. The hand is, for example, a two-finger hand, and it is possible to grasp and release an object by changing the distance between the two fingers.

[0013] In addition, the two robot arms 2A and 2B are such that one of the two robot arms 2A and 2B picks the object 10 by grasping the object 10 with the hand at the picking grip point P1, which is the position where the object 10 is grasped when picking the object 10, and the other robot arm places the object 10 at an arbitrary location by grasping the object 10 with the hand at the placement grip point P2, which is the position where the object 10 is grasped when placing the object 10. Thus, the object 10 can be transferred between one robot arm and the other robot arm.

[0014] FIG. 2 is a schematic diagram for explaining the transfer procedure of the object 10 in the present embodiment. As shown in FIG. 2(A), in the initial state of the placement operation, the object 10 is accommodated in, for example, a box 11 or the like. In this initial state, one of the two robot arms 2A and 2B grasps the object 10 at the picking grip point P1 and picks the object 10. In the example of FIG. 2, the picking grip point P1 is set below the object 10 in the drawing.

[0015] Next, when transfer is required during placement, as shown in FIG. 2(B), the object 10 is transferred between two robot arms 2A and 2B. In the example of FIG. 2(B), while one of the two robot arms 2A and 2B maintains the state of gripping the object 10 at the gripping point P1 for picking, the other of the two robot arms 2A and 2B grips the object 10 at the placement gripping point P2 set at a position different from the gripping point P1 for picking. In the example of FIG. 2, the placement gripping point P2 is set above the object 10 in the drawing. At this time, both of the two robot arms 2A and 2B are in the state of gripping the object 10.

[0016] Next, as shown in FIG. 2(C), one of the two robot arms 2A and 2B releases the object 10 from the picking gripping point P1. As a result, only the other of the two robot arms 2A and 2B is in the state of gripping the object 10 at the placement gripping point P2. After that, while the robot arm grips the object 10 at the placement gripping point P2, the object 10 is transported to the target position, and the object 10 is placed at the target position in the desired posture, and then the object is released from the placement gripping point P2. As a result, the placement operation in which the object 10 is placed at the target position is achieved.

[0017] Returning to FIG. 1, the camera 3 is an imaging device that acquires a two-dimensional image and point cloud data of the object 10. The point cloud data is, for example, three-dimensional point cloud data. The camera 3 may be installed, for example, at a position for imaging the entire working area of the two robot arms 2A and 2B from above, or may be installed on the end effector of the two robot arms 2A and 2B or on any part of the six-degree-of-freedom link. The camera 3 may be single or plural.

[0018] The control unit 4 controls the operation of the placement operation by the two robot arms 2A and 2B based on the image information captured by the camera 3.

[0019] Based on the two-dimensional image and point cloud data acquired by the camera 3, the control unit 4 determines whether there exists a path along which the object can be placed in the posture as instructed at the gripping point P1 for picking. If the result of the determination is that there is no such path, the control unit 4 learns the gripping point P2 for placement and a path along which the gripping point P2 for placement can be grasped while holding the gripping point P1 for picking.

[0020] The control unit 4 includes an object recognition unit 41, a gripping point search unit 42, a path generation unit 43, and a deep reinforcement learning unit 44.

[0021] The object recognition unit 41 recognizes the object 10 from the two-dimensional image and three-dimensional point cloud data captured by the camera 3.

[0022] The gripping point search unit 42 calculates the gripping point P1 for picking by using the object recognition result by the object recognition unit 41.

[0023] The path generation unit 43 acquires the posture of the object 10 at the time of placement by using the object recognition result by the object recognition unit 41. The information on the posture at the time of placement is included in, for example, the control command for the placement operation. The control command also includes information on the target position where the object is to be placed.

[0024] In addition, the path generation unit 43 calculates whether there exists a path along which the object 10 can be placed at the target position of the placement operation in the instructed posture while the robot arm holds the gripping point P1 for picking.

[0025] When the path generation unit 43 determines based on the calculation result that there is such a path, it directly executes the placement. That is, the object 10 is moved to the target position and placed while the robot arm holds the gripping point P1 for picking.

[0026] On the other hand, when the path generation unit 43 determines that there is no such path based on the calculation result, it performs learning of the gripping point P2 for placement by using the success of the placement as a reward. More specifically, while one of the two robot arms 2A and 2B holds the gripping point P1 for picking, the paths of the two robot arms 2A and 2B are learned so that the other robot arm can hold the gripping point P2 for placement.

[0027] The positions of these gripping points P2 for placement and the information on the paths of the two robot arms 2A and 2B are derived based on the output of the deep reinforcement learning unit 44. As illustrated in FIG. 1, the deep reinforcement learning unit 44 has a model 44A formed of a deep neural network having an input layer 44A1, a plurality of hidden layers 44A2, and an output layer 44A3. The output of the deep reinforcement learning unit 44 described above is the output value of the output layer 44A3 of this model 44A.

[0028] Also, in the present embodiment, as the input information input to the input layer 44A1 of the model 44A, as illustrated in FIG. 1, the force sensor values of the hands mounted on each of the two robot arms 2A and 2B (information such as gripping force), information such as the angles, angular velocities, respective accelerations, and torques of the joints, the respective paths of the two robot arms 2A and 2B (for example, information such as the three-dimensional positions of the end effectors and the time transition of the change amounts of the respective joints), and information on the ideal posture of the object 10 at the time of placement (for example, six degrees of freedom) are included. Here, as the information on the ideal posture, for example, the information acquired by the above-described path generation unit 43 can be used.

[0029] The path generation unit 43 also performs learning of the model 44A of the deep reinforcement learning unit 44. For example, the path generation unit 43 attempts a placement operation using the gripping point P2 for placement based on the learned model output and the paths of the two robot arms 2A and 2B. As a result of the attempt, if the placement operation is successful, the information on the successful gripping point and path is recorded, and when handling an object having the same shape as the object 10 this time after the next time, the recorded information can be reused. On the other hand, as a result of the attempt, if the placement operation fails, the learning of the model 44A is repeated until the gripping point and path at which the placement is successful are calculated. The learning of the model 44A includes, for example, a process of updating the weights of each connection part connecting the elements of each layer of the model 44A.

[0030] Physically, the control unit 4 can be configured as a computer system including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a RAM (Random Access Memory) and a ROM (Read Only Memory) which are main storage devices, input devices such as a keyboard and a mouse, an output device such as a display, a communication module which is a data transmission / reception device such as a network card, an auxiliary storage device such as a hard disk, and the like. Each function of the control unit 4 shown in FIG. 1 is realized by loading a predetermined computer software onto hardware such as the CPU and the RAM, operating the communication module, the input device, and the output device under the control of the CPU, and reading and writing data in the RAM and the auxiliary storage device.

[0031] FIG. 3 is a flowchart of the placement operation in the present embodiment. Each process of the flowchart shown in FIG. 3 is performed by each part of the control unit 4.

[0032] In step S1, an object (object 10) is recognized from the two-dimensional image and the three-dimensional point cloud data captured by the camera 3 by the object recognition unit 41.

[0033] In step S2, the gripping point search unit 42 calculates the gripping point P1 for picking.

[0034] In step S3, the path generation unit 43 acquires information on the posture of the object 10 at the time of placement from the object recognition result.

[0035] In step S4, the path generation unit 43 calculates whether there is a path along which the object 10 can be placed in the posture as instructed at the gripping point P1 for picking.

[0036] In step S5, based on the calculation result of step S4, the path generation unit 43 determines whether there is a path.

[0037] If it is determined that there is a path (Yes in step S5), the process proceeds to step S6, and the placement operation is directly executed using the path calculated in step S4 by the path generation unit 43. When the processing of step S6 is completed, the processing of this control flow ends.

[0038] On the other hand, if it is determined that there is no path (No in step S5), the learning process of the model of the deep reinforcement learning unit 44 by the path generation unit 43 from step S7 onward is performed.

[0039] In step S7, the gripping point P2 for placement is learned with the success of placement as the reward.

[0040] In step S8, a path that can grip the gripping point P2 for placement while gripping the gripping point P1 for picking is learned.

[0041] In step S9, placement is executed using the learned gripping points and path.

[0042] In step S10, it is determined whether the placement operation was successful.

[0043] If the placement is successful (Yes at step S10), the process proceeds to step S11, where the information on the successful gripping point and path is recorded. When an article with the same shape as the object 10 this time is recognized as an object in a subsequent instance, the recorded information is reused. When the processing of step S11 is completed, the processing of the current control flow ends.

[0044] On the other hand, if the placement fails (No at step S10), the process returns to step S7, and the processing of steps S7 to S10 is repeated until the placement is successful.

[0045] As described above, the picking robot device 1 of this embodiment is a device that performs a placement operation of picking the object 10 and placing the picked object at an arbitrary location, and includes two robot arms 2A and 2B. Each of the two robot arms 2A and 2B is equipped with a hand capable of gripping the object 10. One of the two robot arms grips the object 10 with the hand at the picking gripping point P1, which is the position where the object 10 is gripped when picking the object 10, and performs picking. The other robot arm can transfer the object 10 between one robot arm and the other robot arm so that the object 10 is gripped with the hand at the placement gripping point P2, which is the position where the object 10 is gripped when placing the object 10 at an arbitrary location. Further, the picking robot device 1 includes a camera 3 that acquires a two-dimensional image and point cloud data of the object 10, and a control unit 4. Based on the two-dimensional image and point cloud data acquired by the camera 3, the control unit 4 determines whether there is a path along which the object can be placed in the posture as instructed by the placement operation at the picking gripping point P1. If there is no path, it learns the placement gripping point P2 and a path along which the placement gripping point can be grasped while holding the picking gripping point P1.

[0046] With this configuration, the picking robot device 1 itself can change the gripping point so that it can place the object 10, making it possible to reduce errors during the placement operation of placing the picked object at an arbitrary location. Also, since the picking robot device 1 can change the object 10 in the air, the temporary placement cart that was necessary to temporarily place the object 10 for changing the object 10 as in the prior art becomes unnecessary, improving space efficiency. Further, since temporary placement is unnecessary, the speed of the placement operation of the picking robot device 1 is improved. Since the gripping point and path where the placement is successful can be automatically calculated only by the robot, the placement operation can be automated without human intervention, and the placement operation can be efficiently and quickly performed.

[0047] As described above, the present embodiment has been described with reference to specific examples. However, the present disclosure is not limited to these specific examples. Those obtained by appropriately making design changes by those skilled in the art to these specific examples are also included in the scope of the present disclosure as long as they have the features of the present disclosure. Each element included in each of the above-described specific examples and its arrangement, conditions, shape, etc. are not limited to those illustrated and can be changed as appropriate. Each element included in each of the above-described specific examples can be appropriately combined as long as no technical contradiction occurs.

Explanation of Reference Numerals

[0048] 1 Picking robot device 2A, 2B Robot arms 3 Camera 4 Control unit

Claims

【Claim 1】 A picking robot device that performs a placing operation of picking an object and placing the picked object at an arbitrary location, Two robot arms each equipped with a hand capable of gripping the object, wherein one of the robot arms picks the object by gripping the object with the hand at a picking grip point which is a position for gripping the object when picking the object, and the other robot arm grips the object with the hand at a placing grip point which is a position for gripping the object when placing the object at the arbitrary location, and two robot arms capable of transferring the object between the one robot arm and the other robot arm, A camera that acquires a two-dimensional image and point cloud data of the object, Based on the two-dimensional image and the point cloud data acquired by the camera, it is determined whether there is a path along which the object can be placed in the posture as instructed by the placing operation at the picking grip point. If there is no such path, a control unit is provided that learns the placing grip point and a path along which the placing grip point can be grasped while holding the picking grip point. A picking robot device comprising the above.

Images