Robot control device, robot, robot control system, and robot control method

Abstract

Provided is a control device that addresses the problem where a boundary cannot be estimated by merely moving in a certain direction when picking of a plurality of objects, the control device estimating a boundary between objects closely adhered to each other without a gap from image changes and direction information. This robot control device for controlling a robot having an articulated arm and an end effector comprises: an image acquisition unit for acquiring an image including an object to be moved and an adjacent object; a movement direction acquisition unit for acquiring a movement direction in which the robot has moved the object to be moved; a boundary estimation unit for estimating a boundary between the object to be moved and the adjacent object by using the image and the movement direction; and an operation control unit for controlling the operation of the robot by using the boundary estimated by the boundary estimation unit.

Description

Robot control device, robot, robot control system, and robot control method

[0001] The present invention relates to a robot control device, a robot, a robot control system, and a robot control method that are capable of recognizing boundaries between objects.

[0002] In recent years, the use of robots such as articulated arm robots for transporting objects has been increasing in logistics warehouses, etc. When this type of robot selects and transports one object from among objects mixed on a pallet, for example, it needs to correctly recognize the boundaries between the object being transported and other objects even when they are in close contact with each other with no gaps between them.

[0003] A known prior art technique for recognizing the boundary between closely spaced objects is the workpiece transfer system described in Patent Document 1. For example, the abstract of Patent Document 1 describes the problem of "providing a workpiece transfer system capable of accurately recognizing a workpiece to be transferred," and describes a solution thereto as including "a determination means (S3) for determining whether the boundary of the workpiece to be transferred is clear when the measurement means measures the workpiece to be transferred (S2), an estimation means (S5) for estimating that the boundary of the workpiece to be transferred is unclear, including the unclear portion, a shifting means (S9) for shifting the workpiece to be transferred estimated by the estimation means by a predetermined distance, a remeasurement means (S12, S13) for remeasurement of the workpiece to be transferred shifted by the predetermined distance by the shifting means, and a determination means (S14) for determining that the workpiece to be transferred measured by the remeasurement means is the workpiece to be transferred."

[0004] Furthermore, paragraph 0030 of the same document states, "As shown in Figure 4(b), if the boundary portion K1 between adjacent workpieces Wa is clear (step S3: YES), the CPU 50 stores each of the workpieces Wa of the cut-out portion T shown in Figure 4(b) in the measurement result memory unit 56 as a workpiece Wa to be transferred (step S4)." Paragraph 0031 states, "On the other hand, as shown in Figure 4(c), if the boundary portion K2 between adjacent workpieces Wa is unclear (in the illustration, the unclear portion is indicated by a dashed line) (step S3: NO), the CPU 50 presumes that the unclear portion is the workpiece to be transferred (step S5). In other words, in this embodiment, the cut-out portion T shown in Figure 4(c) is all presumed to be the workpiece Wa1 to be transferred."

[0005] Furthermore, in FIG. 2 of the same document, as the processing after the above-mentioned step S5, a group of processing steps necessary to identify whether the unclear boundary portion K2 is a true boundary, such as "instructing the shifting operation (step S9)," "checking the result of the shifting operation (step S10)," and "did the operation proceed as a single unit? (step S11)," are disclosed.

[0006] Japanese Patent Application Laid-Open No. 2020-121346

[0007] However, when the work transfer system of Patent Document 1 detects an unclear boundary portion K2 as illustrated in Figure 4(c) of the same document, it identifies whether the boundary portion K2 is a true boundary or not by processing steps S5 to S14 of Figure 2 of the same document.Therefore, if it is unable to recognize even an unclear boundary, there is no motivation to try to identify the true boundary, and there is a possibility that part of the actual boundary will not be detected.

[0008] 7 and 8 of the same document, in determining whether or not the unclear boundary portion K2 is a true boundary, the workpiece transfer system performs an operation of shifting the workpiece in a direction perpendicular to the boundary portion K2, and considers whether or not a gap occurs as a result of this operation. Therefore, the boundaries that can be detected by the technology of the same document are limited to boundaries that are created when the boundary portion K2 is separated as a result of shifting the workpiece, and there is a problem in that boundaries other than this cannot be detected.

[0009] In light of the above, an object of the present invention is to provide a robot control device, a robot control system, and a robot control method that can extract the boundaries of multiple objects with high accuracy and correctly recognize the regions of individual objects.

[0010] In order to solve the above problems, the robot control device of the present invention is a robot control device that controls a robot having a multi-joint arm and an end effector, and is equipped with an image acquisition unit that acquires an image including a moving object and an adjacent object, a movement direction acquisition unit that acquires the movement direction in which the robot moves the moving object, a boundary estimation unit that estimates the boundary between the moving object and the adjacent object using the image and the movement direction, and an operation control unit that controls the operation of the robot using the boundary estimated by the boundary estimation unit.

[0011] According to the present invention, it is possible to extract the boundaries of a plurality of objects with high accuracy and correctly recognize the regions of the individual objects.

[0012] FIG. 1 is a hardware configuration diagram of a transport robot system according to a first embodiment. FIG. 1 is a hardware configuration diagram of a control device according to a first embodiment. FIG. 2 is a functional block diagram of the control device according to a first embodiment. FIG. 3 is a flowchart diagram of the control device according to a first embodiment. FIG. 4 is a diagram for explaining a scene in which rectangular objects are closely packed with no gaps according to a second embodiment. FIG. 5 is a diagram for explaining a scene after a rectangular object is moved according to a second embodiment. FIG. 6 is a diagram for explaining an image difference before and after a rectangular object is moved according to a second embodiment. FIG. 7 is a diagram for explaining a method of drawing auxiliary lines that become boundaries of rectangular objects according to a second embodiment. FIG. 8 is a diagram for explaining a scene in which packed objects are closely packed with no gaps according to a third embodiment. FIG. 9 is a diagram for explaining a scene after a packed object is moved according to a third embodiment. FIG. 10 is a diagram for explaining the relationship between a line segment connecting the centers of circular boundaries and a movement direction according to a third embodiment. FIG. 11 is a diagram for explaining a method of drawing auxiliary lines that become boundaries of packed objects according to a third embodiment.

[0013] Hereinafter, an embodiment of a robot control device according to the present invention will be described with reference to the drawings.

[0014] First, a robot control system 100 according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 4. FIG.

[0015] 1 is a schematic diagram showing the environment in which a robot control system 100 according to this embodiment is used. As shown here, the robot control system 100 is a system in which a robot control device 1, a robot 2, and a stereo camera 3 are connected to each other so that they can communicate with each other via wire or wirelessly, and is used to grasp any of various objects 4 (e.g., plastic bottles, toilet paper, cardboard boxes, etc.) placed on a pallet 5 and transfer the object to another pallet. Each component will be described in detail below.

[0016] <Robot 2> The robot 2 is a multi-joint arm robot having a multi-joint arm 21 and an end effector 22 such as a robot hand, and the operation of which is controlled by the robot control device 1. As is clear from Fig. 1, the robot 2 is installed in a location where, by moving the multi-joint arm 21, it can grasp any object 4 on a pallet 5 with the end effector 22 and transfer it to another pallet, for example.

[0017] <Stereo Camera 3> The stereo camera 3 is an imaging device or measurement device that transmits stereo images captured synchronously by the left camera 3L and right camera 3R and processed three-dimensional information to the robot control device 1. As is clear from Fig. 1, the stereo camera 3 is installed in a location where it can capture an image of the entire top surface of the pallet 5. In this embodiment, the stereo camera 3, which is made up of left and right cameras each equipped with an image sensor of 1280 pixels x 1024 pixels (approximately 1.31 million pixels), is installed facing downward at a position approximately 2 m above the pallet 5. However, the specifications and installation position of the stereo camera 3 are not limited to those described above as long as it is possible to measure gaps of approximately 20 mm between objects on the pallet 5 with high accuracy, including in the depth direction.

[0018] <Robot Control Device 1> Figure 2 is a block diagram showing the hardware configuration of the robot control device 1. As shown here, the robot control device 1 is a computer equipped with an arithmetic unit 11 such as a CPU, a storage device 12 such as a semiconductor memory, an input device 13 such as a keyboard or mouse operated by an operator, an output device 14 such as a liquid crystal display that presents information to the operator, a communication interface 15 for communicating with the robot 2 and the stereo camera 3, and a bus 16 that connects them.

[0019] 3 is a functional block diagram of the robot control device 1 realized by the hardware configuration of FIG. 2. As shown here, the robot control device 1 includes an image acquisition unit 15a, a movement direction acquisition unit 15b, an object recognition unit 11a, a boundary estimation unit 11b, and an operation control unit 11c. The image acquisition unit 15a and the movement direction acquisition unit 15b are realized by cooperation of the communication interface 15, the storage device 12, and the like in FIG. 2, and the object recognition unit 11a, the boundary estimation unit 11b, and the operation control unit 11c are realized by cooperation of the arithmetic device 11, the storage device 12, and the like in FIG. Each unit will be described in detail below.

[0020] The image acquisition unit 15a is a functional unit that acquires and stores stereo images of the entire top surface of the pallet 5 from the stereo camera 3. The image acquisition unit 15a can store multiple stereo images taken at different times.

[0021] The movement direction acquisition unit 15b is a functional unit that acquires and stores the movement direction of the end effector 22 when moving the object 4 on the pallet 5. As an example of a method for acquiring the movement direction, the movement direction acquisition unit 15b monitors the angles of each joint of the articulated arm 21 and calculates a change in coordinates of the end effector 22 based on these joint angles, thereby estimating the movement direction.

[0022] The object recognition unit 11a is a functional unit that processes the stereo images acquired by the image acquisition unit 15a, estimates the contours of the objects 4 on the pallet 5, and estimates the shape, position, and orientation of the objects 4 based on the estimated contours. Methods for estimating the contours include, for example, extracting contours from brightness edges or depth edges in the images, matching with stereo image features of the objects 4 acquired in advance, and detecting boundaries of regions of approximately the same height in three-dimensional information. Note that when multiple objects 4 are closely adjacent to each other, the boundaries between the adjacent objects become unclear, and therefore the object recognition unit 11a may estimate a single contour for the multiple closely adjacent objects 4. How to deal with this case will be described later.

[0023] The boundary estimation unit 11b is a functional unit that estimates whether a boundary between objects 4 exists within the contour recognized by the object recognition unit 11a, based on the data acquired by the image acquisition unit 15a and the movement direction acquisition unit 15b. Details of the boundary estimation process will be described later.

[0024] The movement control unit 11c is a functional unit that controls the movement of the robot 2, taking into consideration the boundary estimated by the boundary estimation unit 11b.

[0025] <Processing Flowchart of Robot Control Device 1> Next, the processing procedure in the robot control device 1 having the functional blocks of FIG. 3 will be described with reference to the flowchart of FIG.

[0026] First, in process S1, the image acquisition unit 15a acquires and stores stereo images of the object 4 on the pallet 5 taken from above using the stereo camera 3. Note that, hereinafter, the stereo images acquired in this process are referred to as pre-movement images P1.

[0027] In process S2, the object recognition unit 11a estimates the contours of each object on the pallet 5 based on the pre-movement image P1, and further recognizes the shape and position and orientation of each object based on the estimated contours. Possible objects 4 on the pallet 5 include, for example, cardboard or box-shaped objects and a group of wrapped plastic bottles, as illustrated in FIG. 1 . The general shape of the group of single objects is preferably a rectangle in which the outer edges of adjacent contours are approximately perpendicular to each other. Furthermore, the object recognition unit 11a estimates six-dimensional information, combining three dimensions in the translational direction and three dimensions in the rotational direction, as the position and orientation of the object 4.

[0028] In process S3, the object recognition unit 11a determines whether the contour recognized in process S2 is the contour of a single object. As described above, when objects are closely spaced from one another, the contour estimated by the object recognition unit 11a may contain multiple objects 4 whose individual contours could not be detected. Therefore, in this process, if a single contour contains a brightness edge or depth edge that is not recognized as a boundary and is equal to or greater than a predetermined threshold, or if the size of a single contour is larger than the maximum (known) size of the object to be recognized, the extracted contour is estimated to be not the contour of a single object. Note that the method for estimating that the extracted contour is not the contour of a single object is not limited to the above.

[0029] In process S4, the operation control unit 11c causes the end effector 22 of the robot 2 to grasp the edge of the object 4 and slightly move the object 4. In addition, the movement direction acquisition unit 15b stores the movement direction of the end effector 22 (i.e., the movement direction of the grasped object 4).

[0030] In this process, the direction of movement of object 4 can basically be determined appropriately depending on the gripping position of object 4. However, in cases where object 4 to be moved is placed on other objects and consideration must be given to preventing the load from falling over, or where object 4 to be moved is surrounded by other objects and the direction in which object 4 can be shifted is restricted, the gripping position of object 4 can be determined taking these circumstances into consideration, and then object 4 can be moved in that specific restricted direction.

[0031] Furthermore, the amount of movement of the object 4 in this process is a distance that allows for reliable detection of gaps that occur between objects as a result of moving the object 4 grasped by the end effector 22, and can be set appropriately depending on the specifications and installation position of the stereo camera 3, but is, for example, a distance of 20 mm or more.

[0032] In process S5, the image acquisition unit 15a acquires and stores an image of the object 4 taken from above by the stereo camera 3 after it has been moved by the end effector 22. Note that, hereinafter, the image acquired in this process will be referred to as a post-movement image P2.

[0033] In step S6, the boundary estimation unit 11b estimates the boundary of the object 4 based on the pre-movement image P1, the post-movement image P2, and the movement direction. Details of the boundary estimation method will be described in the second and third embodiments.

[0034] In process S7, the boundary estimation unit 11b re-estimates the position and orientation of the moving object 4 based on the boundary estimated in process S6. The method of estimating the position and orientation may be the same as in process S2, or may be realized by another algorithm that utilizes the moving object and its boundary information.

[0035] In process S8, the operation control unit 11c operates the end effector 22 based on the position and orientation re-estimated in process S7 to grasp the object 4 and execute a desired operation such as transferring it to another pallet, etc. It goes without saying that even if it is determined in process S3 that the object 4 is a single object, it is possible to grasp the object 4 based on the position and orientation estimated in process S2.

[0036] The robot control system of this embodiment described above can correctly recognize the regions of individual objects even in a situation where a variety of objects are mixed together and there are no gaps between the objects.

[0037] 5A to 5D, a robot control system 100 according to a second embodiment of the present invention will be described. This embodiment describes a method for estimating an object boundary when the object 4 to be grasped by the robot 2 is a rectangular object 41, and a duplicate description of points common to the first embodiment will be omitted.

[0038] 5A is an example of a pre-movement image P1 in this embodiment, showing a scene in which six rectangular objects 41 are closely packed with no gaps, as viewed from directly above. The outline O shown by the solid line is an object boundary estimated by the object recognition unit 11a based on differences in height and geometric characteristics from the surroundings. The boundary B shown by the dotted line is actually present, but cannot be recognized as a boundary by the object recognition unit 11a because the rectangular objects 41 are closely packed with no gaps. Therefore, the object recognition unit 11a mistakenly assumes that the six rectangular objects 41 are a single object 4 surrounded by the outline O, and erroneously estimates its shape, position, and orientation.

[0039] FIG. 5B is an example of a post-movement image P2 in this embodiment after the above-described steps S4 and S5 are performed, and shows the image P2 after the end effector 22 is used to move the object 41 out of the six rectangular objects 41. Ａ The scene is viewed from directly above after the moving object 41 is moved by a predetermined amount (for example, 20 mm) in the direction of movement D to generate a gap. Ａ Since the direction of movement of the rectangular object 41 cannot be in a direction where another rectangular object 41 exists, the direction of movement is determined to be a direction in which no rectangular object 41 exists (to the right in FIG. 5B). Furthermore, this direction is not limited to the situation of the other rectangular objects 41, but is also limited by space constraints, prevention of collapse of the load, and other circumstances. In FIG. 5B, the movement direction D is to the right, and this direction is stored in the storage device 12 in a vector format or the like.

[0040] 5C is an image in which the difference between the post-shift image P2 in FIG. 5B and the pre-shift image P1 in FIG. 5A is indicated by a solid line. Ａ is a straight line that is approximately perpendicular to the moving direction D, and the moving object 41 Ａ On the other hand, the short side of the contour O Ｂ is a straight line substantially parallel to the moving direction D, and the moving object 41 Ａ Therefore, at this point, the moving object 41 Ａ The entire long side of the original contour could not be extracted.

[0041] FIG. 5D shows an outline O that is approximately parallel to the moving direction D. Ｂ Extend the auxiliary line L ＳThe outline O Ｂ The determination of whether the contour O is substantially parallel to the moving direction D is performed by, for example, Ｂ From the coordinates of both ends of the contour O Ｂ This can be achieved by calculating the inclination vector of the auxiliary line L and considering the dot product with the direction vector of the movement direction D. Ｓ The moving object 41, which was difficult to extract in FIG. 5C, Ａ Since the boundary estimation unit 11b includes the entire long side of the contour of the auxiliary line L Ｓ The moving object 41 Ａ The object recognition unit 11a can estimate the boundary of the moving object 41 based on the boundary estimated by the boundary estimation unit 11b. Ａ The movement control unit 11c can then generate the next movement of the end effector 22 based on the shape, position, and orientation estimated by the object recognition unit 11a.

[0042] 6A to 6D, a robot control system 100 according to a third embodiment of the present invention will be described. This embodiment describes a method for estimating an object boundary when the object 4 to be grasped by the robot 2 is a packaged object 42 made up of six objects with a circular outline C, such as plastic bottles, packaged together in a package such as transparent plastic wrap. Explanations of points common to the above embodiments will be omitted.

[0043] 6A is an example of a pre-movement image P1 in this embodiment, showing a scene in which two packed objects 42 are closely packed together with no gaps between them, as viewed from directly above. A line segment L shown by a solid line is a straight line connecting the centers of the circular contours C that are close to each other in the pre-movement image P1. A method for extracting the circular contours C includes, for example, extracting a circular pattern from the pre-movement image P1. The object recognition unit 11a estimates the entire area connected by the line segment L as a single object 4. However, in FIG. 6A , because the centers of the circular contours C included in adjacent packed objects 42 are close to each other, the object recognition unit 11a erroneously estimates the 12 circular contours C as a single packed object 42.

[0044] FIG. 6B is an example of the post-movement image P2 in this embodiment, in which the moving object 42 of the two packaging objects 42 is moved by the end effector 22. Ａ The image P2 shows a scene viewed from directly above after the moving object 42 has been moved in the moving direction D. Ａ By moving the line segment, the slope of the line segment L did not change. Ａ and the line segment L whose slope has changed Ｂ Therefore, the robot control device 1 calculates the distance between these line segments L Ａ , L Ｂ , and the inclination of the moving direction D is stored in the storage device 12 in vector format.

[0045] FIG. 6C is a diagram showing the line L Ａ , L Ｂ , and only the moving direction D is extracted and shown. As is clear from the figure, the line segment L shown by the solid line is approximately parallel or approximately perpendicular to the moving direction D. Ａ The other line segments are the dotted line segments L Ｂ Line segment L Ｂ is neither approximately parallel nor approximately perpendicular to the moving direction D, it can be determined that it is not a line segment connecting circles inside a single packaged object 42.

[0046] FIG. 6D shows an auxiliary line L approximately parallel to the movement direction D with respect to the post-movement image P2. Ｓ The auxiliary line L Ｓ is the line segment L Ｂ The auxiliary line L is a straight line that passes through the center point of the Ｓ represents the boundary between the two packed objects 42, which was difficult to detect from the pre-movement image P1. Therefore, the boundary estimation unit 11b uses the auxiliary line L Ｓ Referring to the moving object 42 Ａ The object recognition unit 11a can estimate the boundary of the moving object 42 estimated by the boundary estimation unit 11b. Ａ Based on the contour of the moving object 42 ＡThe movement control unit 11c can then generate the next movement of the end effector 22 based on the shape, position, and orientation estimated by the object recognition unit 11a.

[0047] Next, a robot control system 100 according to a fourth embodiment of the present invention will be described with reference to the functional block diagram of Fig. 7. This embodiment assumes a case where the object boundary cannot be correctly estimated even when the movement direction D of the end effector 22 in step S4 is taken into consideration, and a duplicated description of points common to the above-described embodiments will be omitted.

[0048] As shown in FIG. 7 , the robot control device 1 of this embodiment includes a display 14a that presents the estimation results of the boundary estimation unit 11b to the operator as an image. When executing a robot 2 motion command generated by the motion control unit 11c, the end effector 22 may not be able to properly grasp an object. This may be due to an incorrect object boundary estimated by the boundary estimation unit 11b. Therefore, the boundary estimation unit 11b presents an image of the estimated object boundary to the operator via the display 14a and requests the operator to determine whether the estimated object boundary is valid. The operator, upon receiving the request, checks the object boundary on the presented image, and if it is invalid, manually registers a valid object boundary or manually specifies a valid grasping point, thereby assisting the robot 2 to properly grasp the object 4.

[0049] 100 Robot control system 1 Robot control device 11 Arithmetic device 11a Object recognition unit 11b Boundary estimation unit 11c Operation control unit 12 Storage device 13 Input device 14 Output device 14a Display 15 Communication interface 15a Image acquisition unit 15b Movement direction acquisition unit 16 Bus 2 Robot 21 Articulated arm 22 End effector 3 Stereo camera 3L Left camera 3R Right camera 4 Object 41 Rectangular object 42 Packed object 5 Pallet O Contour B Boundary D Movement direction L Ｓ Auxiliary line C Circular contour L, L Ａ , L Ｂ line segment

Claims

1. A robot control device that controls a robot having a multi-joint arm and an end effector, comprising: an image acquisition unit that acquires an image including a moving object and an adjacent object; a movement direction acquisition unit that acquires a movement direction in which the robot moves the moving object; a boundary estimation unit that estimates a boundary between the moving object and the adjacent object using the image and the movement direction; and a movement control unit that controls the movement of the robot using the boundary estimated by the boundary estimation unit.

2. A robot control device according to claim 1, wherein the boundary estimation unit estimates a straight line that is approximately parallel to the direction of movement as the boundary between the moving object and the adjacent object.

3. A robot control device according to claim 1, further comprising an object recognition unit that estimates the position and orientation of the moving object based on the image acquired by the image acquisition unit and determines whether it is a single object, and the operation control unit controls the robot to move the moving object that is not a single object.

4. A robot control device according to claim 3, wherein the moving object is a rectangular object, and the boundary estimation unit estimates the boundary as a linear difference between a pre-movement image and a post-movement image among the images.

5. A robot control device according to claim 3, wherein the object to be moved is a packed object made up of multiple objects each having a circular boundary, the object recognition unit connects the circular boundaries with line segments, and the boundary estimation unit estimates the boundary so that it passes through a line segment that is not approximately perpendicular or approximately parallel to the direction of movement in the post-movement image of the image.

6. A robot control device according to claim 1, wherein the movement direction acquisition unit calculates a change in coordinates of the end effector from the joint angles of the articulated arm and estimates the movement direction.

7. A robot control device according to claim 1, further comprising an output device that depicts the estimated boundary on an image and outputs the image.

8. A robot control device according to claim 7, wherein the output device requests an operator to determine whether the drawn boundary is valid.

9. A robot control device according to claim 1, wherein the movement direction is determined according to the position at which the end effector grasps the object to be moved.

10. A robot control device according to claim 1, wherein the movement direction is a direction in which the load does not collapse even when the object to be moved is moved.

11. A robot control device according to claim 1, wherein the movement direction is a direction in which the movement of the object to be moved is not obstructed by the adjacent object.

12. A robot control device according to claim 1, wherein the amount of movement in the movement direction is about 20 mm.

13. A robot incorporating the robot control device according to claim 1, and comprising an articulated arm and an end effector whose movements are controlled by the movement control unit.

14. A robot control system comprising: a robot control device according to any one of claims 1 to 12; a stereo camera that transmits images to the image acquisition unit; and a robot controlled by the robot control device.

15. A robot control method for controlling a robot having a multi-joint arm and an end effector, comprising: an image acquisition process for acquiring an image including a moving object and an adjacent object; a movement direction acquisition process for acquiring a movement direction in which the robot has moved the moving object; a boundary estimation process for estimating a boundary between the moving object and the adjacent object using the image and the movement direction; and a motion control process for controlling the motion of the robot using the estimated boundary.

Images