Control device, robot system, and robot control method

The robot control system addresses the challenge of interference by using a searcher and decider to determine optimal hand positions and orientations, enabling accurate and efficient workpiece placement.

WO2024247135A1PCT designated stage expired Publication Date: 2024-12-05KAWASAKI JUKOGYO KK
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/020184
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional robot control systems face difficulties in accurately picking and placing workpieces due to interference between the robot arm or hand and other objects, which can lead to improper placement.

Method used

A control device and robot system that includes a robot arm and hand, equipped with a searcher to find candidate positions and postures for the hand relative to the workpiece, and a decider to select the optimal position and posture, ensuring the robot can pick and place workpieces without interference.

Benefits of technology

The system effectively picks and places workpieces by determining the best relative position and orientation of the hand, reducing interference and improving the accuracy of the pick-and-place process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023020184_05122024_PF_FP_ABST
    Figure JP2023020184_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A main control device 3 has a robot arm 12 and a hand 14 attached to the robot arm 12, and controls a robot 1 that performs pick-and-place processing of a workpiece. The main control device 3 comprises a searcher 41 that searches a plurality of candidates for the relative position and posture of the hand 14 with respect to a workpiece placed at a placement position candidate, and a determiner 42 that determines the relative position and posture of the hand 14 with respect to a workpiece W to be picked from a plurality of candidates for the relative position and posture of the hand at the time of placement.
Need to check novelty before this filing date? Find Prior Art

Description

Control device, robot system, and robot control method

[0001] The technology disclosed herein relates to a control device, a robot system, and a method for controlling a robot.

[0002] Conventionally, control devices for controlling robots that perform picking have been known. For example, a control device described in Patent Literature 1 controls a robot arm and hand to pick a workpiece, transport it to another location, and place it there. This control device changes the gripping mode depending on the state of the workpiece to be picked, thereby realizing gripping of the workpiece by the robot arm and hand without interfering with other objects.

[0003] Japanese Patent Application Laid-Open No. 2004-230513

[0004] When picking a workpiece, one target workpiece is often selected from multiple candidates, so interference between the robot arm and hand and other objects tends to be taken into consideration when selecting the target workpiece.

[0005] However, if only interference during picking is taken into consideration, it may become difficult to place the workpiece appropriately.

[0006] The technology disclosed herein has been made in consideration of these points, and its purpose is to appropriately pick and place workpieces.

[0007] The control device disclosed herein is a control device that controls a robot that has a robot arm and a hand attached to the robot arm and performs pick-and-place processing of a workpiece, and is equipped with a searcher that searches for multiple candidates for the relative position and orientation of the hand with respect to the workpiece to be placed at a candidate place position, and a determiner that determines the relative position and orientation of the hand with respect to the workpiece to be picked from multiple candidates for the relative position and orientation of the hand at the time of placement.

[0008] The robot system of the present disclosure includes a robot having a robot arm and a hand attached to the robot arm, and the control device.

[0009] The robot control method disclosed herein is a method for controlling a robot that has a robot arm and a hand attached to the robot arm and performs pick-and-place processing of a workpiece, and includes searching for multiple candidates for the relative position and orientation of the hand with respect to the workpiece to be placed at a candidate place position, and determining the relative position and orientation of the hand with respect to the workpiece to be picked from the multiple candidates for the relative position and orientation of the hand at the time of placement.

[0010] According to the control device, the picking and placing of the workpiece can be performed appropriately.

[0011] According to the robot system, it is possible to appropriately pick and place the workpiece.

[0012] According to the robot control method, the workpiece can be picked and placed appropriately.

[0013] FIG. 1 is a schematic diagram showing the configuration of a robot system. FIG. 2 is a diagram showing a schematic hardware configuration of a robot control device and a main control device 3. FIG. 3 is a block diagram showing the configuration of a control system of a processor. FIG. 4 is a diagram showing the interior of a first container as viewed from above. FIG. 5 is an exemplary perspective view showing the interior of a second container. FIG. 6 is a schematic plan view of the interior of the second container for explaining a search for position candidates. FIG. 7 is an explanatory diagram showing a hand when picking up a target workpiece. FIG. 8 is a schematic diagram showing the relative position and posture of the hand when viewed in the direction of a reference axis. FIG. 9 is a flowchart of position planning. FIG. 10 is a flowchart of a subroutine for place position planning. FIG. 11 is a flowchart of a subroutine for pickup position planning. FIG. 12 is a timing chart of place position planning, pickup position planning, and robot operation. FIG. 13 is a flowchart of position planning according to a modified example. FIG. 14 is a flowchart of a subroutine for place position correction. FIG. 15 is a timing chart of place position planning, pickup position planning, and robot operation according to a modified example.

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An exemplary embodiment will now be described in detail with reference to the accompanying drawings. 1 is a schematic diagram showing the configuration of a robot system 100. As shown in FIG.

[0015] The robot system 100 includes a robot 1 and a main control device 3 that controls the robot 1. The main control device 3 controls the robot 1 to cause the robot 1 to execute a process. In this example, the process performed by the robot 1 is a pick-and-place process. For example, the robot 1 picks up a target workpiece from a first container 91 in which multiple workpieces are piled up randomly, and transfers the target workpiece to a second container 92. The robot 1 arranges the target workpieces in a regular pattern within the second container 92. The robot system 100 arranges the multiple workpieces in the first container 91 within the second container 92 by repeating the pick-and-place process. The target workpiece has, for example, a substantially rectangular parallelepiped shape.

[0016] The robot system 100 may be equipped with a camera 51 that acquires images of the workpieces. The camera 51 may include a first camera 51A that acquires images of the inside of the first container 91 and a second camera 51B that acquires images of the inside of the second container 92. The first camera 51A is fixedly disposed above the first container 91. The first camera 51A captures images of the inside of the first container 91 from above. The first camera 51A acquires images including a plurality of workpieces in the first container 91. The second camera 51B is fixedly disposed above the second container 92. The second camera 51B captures images of the inside of the second container 92 from above. The second camera 51B acquires images including a plurality of workpieces in the second container 92. Hereinafter, when there is no need to distinguish between the first camera 51A and the second camera 51B, they will be simply referred to as cameras 51.

[0017] Here, the image is a two-dimensional image or a three-dimensional image. The three-dimensional image may be point cloud data, an RGB-D image, an RGB image, a depth image, a voxel image, or the like. That is, the camera 51 may be a three-dimensional camera, i.e., an RGB-D camera that outputs an RGB-D image, a stereo camera that acquires an RGB image, or a three-dimensional vision sensor that acquires point cloud data.

[0018] -Robot- The robot 1 has a robot arm 12 and a hand 14 attached to the robot arm 12. In this example, the robot 1 is an industrial robot. The hand 14 is a so-called end effector. A base coordinate system with three orthogonal axes is defined in the space in which the robot 1 is placed.

[0019] The robot arm 12 is configured to move three-dimensionally. Specifically, the robot arm 12 is configured to perform movements including translation with at least three degrees of freedom. In this example, the robot arm 12 is a vertical multi-joint robot arm. The robot arm 12 is supported by a base 10. The robot arm 12 has a plurality of links, a plurality of joints connecting the plurality of links, and servo motors that rotate the plurality of joints.

[0020] In detail, the robot arm 12 has a first link 12a connected to the base 10, a second link 12b connected to the first link 12a, a third link 12c connected to the second link 12b, a fourth link 12d connected to the third link 12c, and a fifth link 12e connected to the fourth link 12d.

[0021] Specifically, the base 10 and the first link 12a are connected to each other via a first joint 13a that is rotatable about an axis extending in the vertical direction. The first link 12a and the second link 12b are connected to each other via a second joint 13b that is rotatable about an axis extending in the horizontal direction. The second link 12b and the third link 12c are connected to each other via a third joint 13c that is rotatable about an axis extending in the horizontal direction. The third link 12c and the fourth link 12d are connected to each other via a fourth joint 13d that is rotatable about the axis of the fourth link 12d (i.e., the direction in which the fourth link 12d extends). The fourth link 12d and the fifth link 12e are connected to each other via a fifth joint 13e that is rotatable about an axis perpendicular to the axis of the fourth link 12d.

[0022] The robot arm 12 has servo motors 16 (see FIG. 2) that rotate and drive each joint. Each servo motor 16 has an encoder 16a (see FIG. 2).

[0023] The robot arm 12 configured in this manner is configured to perform translational movements in the directions of three orthogonal axes and rotational movements around each of the three orthogonal axes.

[0024] The hand 14 includes a suction device 15 that is disposed at a position offset in the radial direction around a predetermined reference axis T of the robot arm 12. The suction device 15 is disposed at a position eccentric to the reference axis T. For example, the hand 14 has a base 14a that is attached to the robot arm 12. The base 14a extends in the radial direction around the reference axis T. The suction device 15 is attached to the end of the base 14a that is farther from the reference axis T via a plate, a block, or the like.

[0025] Furthermore, the suction device 15 is configured to be able to change its position relative to the robot arm 12. For example, the hand 14 is attached to the robot arm 12 so as to be rotatable about a reference axis T. Specifically, the hand 14 is connected to the tip of the robot arm 12, i.e., the fifth link 12e. The fifth link 12e and the hand 14 are connected to each other via a sixth joint 13f so as to be rotatable about the reference axis T. The suction device 15 changes its position relative to the robot arm 12 by rotating the hand 14 about the reference axis T.

[0026] The hand 14 sucks the workpiece W using a suction device 15. The suction device 15 has a suction pad. An air hose connected to a negative pressure generator is connected to the suction device 15. A solenoid valve serving as an actuator 15a (see FIG. 2) is provided on the air hose. By controlling the solenoid valve, the suction device 15 can switch between suction and release. The suction device 15 sucks the surface of the target workpiece W. A pressure sensor 15b (see FIG. 2) is provided on the air hose. The pressure sensor 15b detects the magnitude of the negative pressure in the air hose.

[0027] 2 is a diagram showing the general hardware configuration of the robot controller 2 and the main controller 3. The main controller 3 transmits and receives signals, information, etc. to and from the robot controller 2. The main controller 3 controls the robot 1 via the robot controller 2. The main controller 3 outputs commands to the robot controller 2. The robot controller 2 controls the servo motor 16 of the robot arm 12 in response to commands from the main controller 3.

[0028] Images from the first camera 51A and the second camera 51B are input to the main control device 3. The main control device 3 detects the target workpiece W based on the images. In addition, the main control device 3 generates a target path for the robot 1 based on the detected target workpiece W, and outputs a command corresponding to the generated target path to the robot control device 2. An actuator 15a and a pressure sensor 15b are connected to the main control device 3. The main control device 3 controls suction by the hand 14 via the actuator 15a and the pressure sensor 15b. The main control device 3 is an example of a control device.

[0029] The robot control device 2 has a processor 21 , a storage device 22 , and a memory 23 .

[0030] The processor 21 controls the entire robot control device 2. The processor 21 performs various types of arithmetic processing. For example, the processor 21 is formed of a processor such as a CPU (Central Processing Unit). The processor 21 may also be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0031] The storage unit 22 stores various data and programs executed by the processor 21. The storage unit 22 is formed of a non-volatile memory, a hard disk drive (HDD), a solid state drive (SSD), etc. The memory 23 temporarily stores data, etc. For example, the memory 23 is formed of a volatile memory.

[0032] The processor 21 drives the servo motor 16 based on a command from the main control device 3. At this time, the robot control device 2 feedback controls the current supplied to the servo motor 16 based on the detection result of the encoder 16a.

[0033] The main control device 3 has a processor 31 , a storage device 32 , and a memory 33 .

[0034] The processor 31 controls the entire main control device 3. The processor 31 performs various types of arithmetic processing. For example, the processor 31 is formed of a processor such as a CPU (Central Processing Unit). The processor 31 may also be formed of an MCU (Micro Controller Unit), an MPU (Micro Processor Unit), an FPGA (Field Programmable Gate Array), a PLC (Programmable Logic Controller), a system LSI, or the like.

[0035] The storage unit 32 stores various data and programs executed by the processor 31. The storage unit 32 is formed of a nonvolatile memory, a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0036] For example, the memory 32 stores an object detection program for detecting an object, a position and orientation determination program for determining the position and orientation of the hand 14, and a path generation program for generating a target path for the robot 1. The various programs cause the main control device 3 to realize various functions.

[0037] The memory 33 temporarily stores data etc. For example, the memory 33 is formed of a volatile memory. The memory 33 stores images from the camera 51.

[0038] The processor 31 causes the robot 1 to perform a picking operation in which the hand 14 moves to the target workpiece W and causes the suction device 15 to suck the target workpiece W, and a placing operation in which the target workpiece W sucked by the hand 14 moves to a target place position and causes the suction device 15 to release the suction of the target workpiece W. The processor 31 causes the robot 1 to perform a placing operation after the picking operation.

[0039] Prior to a picking operation, the processor 31 determines the relative position and orientation of the hand 14 with respect to the workpiece at the time of picking. Prior to a placing operation, the processor 31 determines the target placement position of the target workpiece W at the time of placing. Hereinafter, the relative position and orientation of the hand 14 with respect to the workpiece will be simply referred to as the "relative position and orientation of the hand 14."

[0040] FIG. 3 is a block diagram showing the configuration of the control system of the processor 31. The processor 31 realizes various functions by reading and storing a program from the storage device 32 into the memory 33. Specifically, the processor 31 functions as a searcher 41 that searches for candidates for the relative position and orientation of the hand 14 with respect to a workpiece to be placed at a candidate placement position in the storage space, and a determiner 42 that determines the relative position and orientation of the hand 14 with respect to a target workpiece to be picked. Furthermore, the processor 31 functions as an operation controller 43 that causes the robot arm 12 and the hand 14 to execute robot operations including a picking operation for picking the target workpiece W and a placing operation for placing the target workpiece W at the placement position. The processor 31 can also function as an imaging controller 44 that causes the camera 51 to acquire images, a workpiece detector 45 that detects the workpiece based on the images, a height detector 46 that detects height information of the storage space based on the images, and a path generator 47 that generates a target path for the robot arm 12 and the hand 14.

[0041] The photography controller 44 controls the first camera 51A and the second camera 51B to capture images from each of the first camera 51A and the second camera 51B. The photography controller 44 causes the first camera 51A to perform picking photography to capture images of the inside of the first container 91. The photography controller 44 causes the second camera 51B to perform place photography to capture images of the inside of the second container 92. The photography controller 44 stores the images from the cameras 51A in the memory 33.

[0042] The workpiece detector 45 detects workpieces from the image through image recognition. The image may contain multiple workpieces. The workpiece detector 45 performs plane detection using, for example, RANSAC (Random Sample Consensus) to detect the surface of the workpiece (hereinafter referred to as the "target surface S"). That is, the workpiece detector 45 detects the workpiece by detecting a plane of a predetermined shape contained in the image. FIG. 4 is a top view of the inside of the first container 91. The workpiece detector 45 can detect multiple target surfaces S, i.e., multiple workpieces. When detecting the target surface S, the workpiece detector 45 determines at least the center position of the target surface S, the normal direction of the target surface S, the vertical and horizontal orientation of the target surface S, and the size (e.g., vertical and horizontal dimensions) of the target surface S. The size of the target surface S may be stored in the memory 32 in advance.

[0043] The height detector 46 detects height information of the storage space in which the target workpiece W will be stored. In this example, the storage space is the inside of the second container 92. The height information is information about the height of each position in the storage space, specifically, height information about each position within the second container 92. In areas within the second container 92 where the bottom surface is exposed, the height information is the height of each position on the bottom surface. In areas within the second container 92 where workpieces are present, the height information is the height of each position on the top surface of the workpieces. The height detector 46 detects the height information of the storage space based on an image of the inside of the second container 92. The height detector 46 obtains the height information of the storage space, for example, by performing preprocessing such as discretization or smoothing on point cloud data of a three-dimensional image. The height detector 46 outputs the height information of the storage space as a height map.

[0044] The path generator 47 generates a target path for the robot arm 12 and the hand 14. The target path is an ordered set of target positions and target postures of the robot arm 12 and the hand 14. That is, the target path is a time-dependent change in the target position and target posture of the robot arm 12 and the hand 14. The target path may also be an ordered set of target rotation angles of each joint of the robot arm 12 (including the sixth joint 13f) that realizes the time-dependent change in the target position and target posture of the robot arm 12 and the hand 14. The target path uniquely defines the position and posture of the robot arm 12 and the hand 14. The target path also defines a target rotation position of the suction cup 15 around the reference axis T. For example, the path generator 47 generates a target path for the robot arm 12 and the hand 14 for a picking operation. The path generator 47 generates a target path for the robot arm 12 and the hand 14 for a placing operation.

[0045] For example, the path generator 47 executes path planning using a probabilistic roadmap method (PRM) or a rapidly-exploring random tree (RRT). In these methods, the path generator 47 also executes an interference check, i.e., checks whether the robot arm 12 and the hand 14 interfere with other objects. The interference check may be performed using point cloud data or CAD data, for example.

[0046] The searcher 41 searches for candidates for the relative position and orientation of the hand 14 with respect to a workpiece to be placed at a place position candidate in the storage space. The searcher 41 searches for multiple candidates for the relative position and orientation of the hand 14. The place position candidates corresponding to the multiple candidates for the relative position and orientation of the hand 14 may be the same as each other, or may be different from each other. In other words, the searcher 41 may search for multiple different candidates for the relative position and orientation of the hand 14 with respect to one position candidate. Alternatively, the searcher 41 may search for multiple candidates for the relative position and orientation of the hand 14 with respect to different position candidates. In this case, the candidates for the relative position and orientation of the hand 14 may be different from each other, or may be the same as each other.

[0047] Specifically, the searcher 41 searches for a plurality of place candidates, which are combinations of position candidates and candidates for the relative position and orientation of the hand 14 with respect to the workpiece placed at the position candidates. Each of the plurality of place candidates has a different combination of position candidate and candidate for the relative position and orientation of the hand 14. At least some of the position candidates included in the plurality of place candidates may be the same as each other. At least some of the candidates for the relative position and orientation of the hand 14 included in the plurality of place candidates may be the same as each other.

[0048] The searcher 41 may search not only for candidates for the relative position and orientation of the hand 14, but also for candidate placement positions. For example, the searcher 41 detects a space within the storage space where a workpiece can be placed (hereinafter referred to as a "placeable space"). The searcher 41 detects a plane P within the storage space where a workpiece can be placed (hereinafter referred to as a "placeable surface") based on the height map, and determines the space above the placeable surface P as the placeable space. FIG. 5 is an exemplary perspective view showing the inside of the second container 92. In FIG. 5, the dotted surface is the placeable surface P. The placeable surface P is a plane exposed upward and is a region with a uniform height in the height map. The placeable surface P is an exposed portion of the bottom surface of the second container 92. If the top surface of the existing workpiece W0 is also exposed, it can also be the placeable surface P. In addition, when multiple existing workpieces W0 are lined up, the top surfaces of the existing workpieces W0 are exposed upward, and the heights of the top surfaces of the existing workpieces W0 are the same, the top surfaces of the existing workpieces W0 become a single placeable surface P. In the example of Figure 5, the top surfaces of the two existing workpieces W0 are each treated as a separate placeable surface P.

[0049] The searcher 41 determines whether the workpiece can be placed at each position while changing the position of the workpiece on the placement surface P. FIG. 6 is a schematic plan view inside the second container to explain the search for position candidates. A position determined to be placement-possible becomes a position candidate. Whether or not a workpiece can be placed is determined based on whether the workpiece can be placed without interfering with other objects. At this time, the searcher 41 sets the size of the workpiece for search (see dashed line) to be the actual size of the workpiece (see dashed line) plus a predetermined margin. Specifically, the searcher 41 enlarges the bottom surface of the workpiece by the predetermined margin. The size of the workpiece may be stored in advance in the memory 32. Alternatively, the size of the workpiece may be identified when the workpiece detector 45 detects the workpiece.

[0050] In this way, the searcher 41 searches for positions in the accommodation space where the workpiece can be placed as candidate placement positions.

[0051] The searcher 41 finds candidates for the relative position and orientation of the hand 14 with respect to the workpiece to be placed at the candidate position.

[0052] In this example, the relative position and orientation of the hand 14 is determined by the position of the suction device 15 with respect to the workpiece, the position of the reference axis T, and the orientation of the reference axis T. FIG. 7 is an explanatory diagram showing the hand 14 when suctioning a target workpiece W. Because the suction device 15 is eccentric with respect to the reference axis T, even if the position of the suction device 15 with respect to the workpiece, i.e., the suction position, is determined, the reference axis T can be positioned at any position within 360 degrees around the suction device 15. In this example, the position of the suction device 15 and the orientation of the reference axis T among the relative position and orientation of the hand 14 are determined. The suction position of the suction device 15 is the center of the target surface S. The reference axis T is approximately perpendicular to the target surface S. The position of the reference axis T among the relative position and orientation of the hand 14 can be set to any angular position within 360 degrees around the suction device 15.

[0053] In this way, since the position of the reference axis T can be set arbitrarily, the hand 14 or the robot arm 12 may interfere with other objects when placing the workpiece. In particular, when the workpiece is placed in a space that includes a corner within the storage space, there is a risk that the hand 14 or the robot arm 12 may interfere with the side wall of the second container 92 or the side surface of the existing workpiece W0, which form the corner. When the workpiece is placed in such a position candidate, a relative position and posture of the hand 14 in which the reference axis T is away from the side wall of the second container 92, etc., is appropriate.

[0054] For example, the memory 32 stores relative positions and postures of a plurality of hands 14 having different angular positions of the reference axis T centered on the suction device 15. Fig. 8 is a schematic diagram showing the relative positions and postures of the hand 14 when viewed in the direction of the reference axis T. For example, as shown in Fig. 8, the memory 32 may store 12 types of relative positions and postures of the hand 14 having different angular positions of the reference axis T centered on the suction device 15 at intervals of 30 degrees. The searcher 41 generates a place candidate by combining one of the plurality of relative positions and postures of the hand 14 with one of the plurality of position candidates.

[0055] The searcher 41 creates, as place candidates, combinations of position candidates and relative positions and orientations of the hand 14 that enable the hand 14 and robot arm 12 to hold a workpiece without interfering with other objects. Specifically, the searcher 41 determines whether or not a target path for the hand 14 and robot arm 12 corresponding to the place candidate can be generated. In other words, the searcher 41 sets, as place candidates, combinations of position candidates and relative positions and orientations of the hand 14 that enable generation of a target path for the hand 14 and robot arm 12 that does not interfere with other objects.

[0056] Specifically, the path generator 47 generates a target path for the hand 14 and the robot arm 12 to move the hand 14 to a state corresponding to the position candidate and the candidate for the relative position and orientation of the hand 14. In the placing operation, an intermediate position through which the robot arm 12 and the hand 14 pass is set, and therefore the path generator 47 generates a target path from the intermediate position. At this time, the path generator 47 performs an interference check of the target path to confirm whether or not there is interference. In other words, the path generator 47 generates a target path without interference. If there is interference, the path generator 47 does not generate a target path. When a target path is generated, the searcher 41 sets a combination of the position candidate and the candidate for the relative position and orientation of the hand 14 at that time as a place candidate.

[0057] The searcher 41 registers the place candidates. That is, the searcher 41 stores combinations of the position candidates and the relative position and orientation of the hand 14 in the memory 33 or the storage 32. The searcher 41 further associates the target route with the place candidates and stores the combinations in the memory 33 or the storage 32.

[0058] In this way, the searcher 41 searches for a plurality of place candidates.

[0059] Furthermore, the searcher 41 assigns priorities to multiple place candidates. Specifically, the searcher 41 assigns priorities to position candidates included in the place candidates. For example, the searcher 41 assigns priorities to multiple position candidates from the perspective of workpiece storage efficiency. The searcher 41 may assign a high priority to a position candidate corresponding to a corner space in the storage space from among the multiple position candidates. When storing workpieces in the second container 92, it is preferable from the perspective of workpiece storage efficiency to pack the workpieces into a space including a corner. Also, from the perspective of workpiece storage efficiency, it is preferable to have the side of the workpiece, for example, the side surface, come into contact with the side wall of the second container 92 or the side surface of the existing workpiece W0 when storing the workpiece in the second container 92. Therefore, the searcher 41 may assign a high priority to a position candidate where the side of the workpiece comes into contact with another object from among the multiple position candidates.

[0060] The searcher 41 may assign priorities to multiple position candidates from the viewpoint of storage stability. For example, the searcher 41 may assign a higher priority to a position candidate that is lower in height from the bottom surface of the second container 92. Placing the work on the bottom surface of the second container 92 allows the work to be placed more stably than placing the work on top of the existing work W0.

[0061] In this way, the searcher 41 searches for a plurality of place candidates, which are combinations of position candidates for the place position and candidates for the relative position and orientation of the hand 14 corresponding thereto, and assigns priorities to the plurality of place candidates.

[0062] The determiner 42 determines the relative position and orientation of the hand 14 with respect to the workpiece to be picked from among multiple relative position and orientation candidates of the hand 14 at the time of placing. Specifically, the determiner 42 determines the relative position and orientation of the hand 14 that allows the hand 14 and the robot arm 12 to hold the workpiece without interfering with other objects. Specifically, the determiner 42 determines whether or not a target path for the hand 14 and the robot arm 12 to pick the workpiece can be generated at the relative position and orientation of the hand 14. In other words, the determiner 42 determines the relative position and orientation of the hand 14 that allows the generation of a target path for the hand 14 and the robot arm 12 that does not interfere with other objects from among multiple relative position and orientation candidates of the hand 14 included in multiple place candidates.

[0063] Specifically, the determiner 42 provisionally selects one relative position and orientation of the hand 14 from among the relative positions and orientations of the multiple hands 14. The path generator 47 generates a target path for the hand 14 and the robot arm 12 to pick the target workpiece at the selected relative position and orientation of the hand 14. Since intermediate positions through which the robot arm 12 and the hand 14 pass are set in the picking operation, the path generator 47 generates a target path from the intermediate positions. At this time, the path generator 47 performs an interference check on the target path to confirm whether or not there is interference. That is, the path generator 47 generates a target path without interference. If there is interference, the path generator 47 does not generate a target path. If a target path is generated, the determiner 42 determines the provisional relative position and orientation of the hand 14 as the relative position and orientation of the hand 14 at the time of picking. If a target path cannot be generated, the determiner 42 selects another relative position and orientation of the hand 14, and the path generator 47 attempts to generate a target path for the selected relative position and orientation of the hand 14.

[0064] When there are multiple options for the target workpiece, that is, when selecting the target workpiece from multiple workpieces, the determiner 42 also determines the target workpiece from the multiple works when determining the relative position and orientation of the hand 14. When determining the relative position and orientation of the hand 14 that can pick up the target workpiece from the relative positions and orientations of the multiple hands 14, the more options there are for the target workpiece, the wider the range of choices for the relative position and orientation of the hand 14 becomes.

[0065] By determining the relative position and orientation of the hand 14 during picking, a candidate position defined by a place candidate corresponding to the determined relative position and orientation of the hand 14 is determined as a place target position. In other words, the determiner 42 determines the relative position and orientation of the hand 14 during picking and also determines a place target position. Furthermore, as described above, the determiner 42 also generates a target path for picking.

[0066] The motion controller 43 causes the robot arm 12 and the hand 14 to perform robot motions including a picking motion for picking up the target workpiece W and a placing motion for placing the target workpiece W at a target placement position. The motion controller 43 outputs a command according to the target path to the robot control device 2. Specifically, the motion controller 43 outputs a command according to the target rotation angle of each joint of the robot arm 12 to the robot control device 2. As described above, the robot control device 2 drives the servo motor 16 based on the command, causing the robot arm 12 and the hand 14 to move along the target path.

[0067] The operation controller 43 switches between suction by the suction device 15 and release of the suction by controlling the actuator 15a of the hand 14. The operation controller 43 controls the actuator 15a to generate negative pressure in the suction device 15, thereby causing the suction device 15 to suction the workpiece. The operation controller 43 controls the actuator 15a to release the negative pressure in the suction device 15, thereby causing the suction device 15 to release the suction of the workpiece W. The operation controller 43 determines that suction of the target workpiece W by the suction device 15 has been completed based on the detection result of the pressure sensor 15b. The operation controller 43 determines that suction has been completed when the negative pressure detected by the pressure sensor 15b is equal to or greater than a predetermined threshold.

[0068] Next, the operation of the robot system 100 configured as described above will be described. In the pick-and-place process of the robot system 100, position planning and robot operation are performed in parallel. Position planning includes place position planning and pickup position planning. Robot operation includes picking operation and placing operation. Position planning will be described with reference to the flowchart in FIG. 9. FIG. 9 is a flowchart of position planning.

[0069] First, in step S101, the photography controller 44 causes the second camera 51B to perform place photography. The place photography is photography of the inside of the second container 92. As a result, an image of the inside of the second container 92 is acquired.

[0070] Next, in step S102, the searcher 41 executes place position planning. The searcher 41 searches for a plurality of place candidates. At this time, a target path of the robot arm 12 and the hand 14 for each of the place candidates is also generated.

[0071] Meanwhile, in parallel with steps S101 and S102, in step S103, the photography controller 44 causes the first camera 51A to perform picking photography. The picking photography is photography of the inside of the first container 91. As a result, an image of the inside of the first container 91 is acquired. Note that step S103 may be performed before or after step S101. However, if the size of the workpiece W identified by the workpiece detector 45 is used to perform placement position planning, steps S103 and S104 are performed before step S102.

[0072] Next, in step S104, the workpiece detector 45 detects a workpiece from within the image of the inside of the first container 91. The detected workpiece will be simply referred to as a "detected workpiece."

[0073] Next, in step S105, the determiner 42 executes a pickup position plan. The determiner 42 determines the target workpiece W to be picked, and determines the relative position and orientation of the hand 14 with respect to the target workpiece W. By determining the relative position and orientation of the hand 14, a position candidate defined by the corresponding place candidate is determined as the target placement position. Furthermore, in the pickup position plan, a target path of the robot arm 12 and the hand 14 for picking up the target workpiece W is also generated.

[0074] In this way, the position plan determines the target workpiece W, the relative position and orientation of the hand 14, the target placement position, the target path for picking, and the target path for placing.

[0075] The motion controller 43 causes the robot arm 12 and the hand 14 to move along a target path, thereby executing the picking motion and the placing motion.

[0076] Specifically, the motion controller 43 controls the robot arm 12 and the hand 14 so as to pick up the target workpiece W in the relative position and posture of the hand 14. Specifically, the motion controller 43 moves the robot arm 12 and the hand 14 along the target picking path, and moves the suction device 15 to the target workpiece W. The motion controller 43 causes the suction device 15 to pick up the target workpiece W.

[0077] Thereafter, the motion controller 43 controls the robot arm 12 and the hand 14 so that the target workpiece W is placed at the target placement position. Specifically, the motion controller 43 moves the robot arm 12 and the hand 14 along the target placement path, and moves the target workpiece W to the target placement position. When the target workpiece W reaches the target placement position, the motion controller 43 releases the suction by the suction device 15. In this way, the placement of the target workpiece W at the target placement position is completed. This completes the pick-and-place process.

[0078] When the main control device 3 completes the pick-and-place process for one target workpiece W, it executes the pick-and-place process for another target workpiece W. By repeating this control, the main control device 3 sequentially transfers the target workpieces W in the first container 91 into the second container 92.

[0079] =Place Position Planning= Next, the place position planning will be described in detail. Fig. 10 is a flowchart of the place position planning subroutine.

[0080] In step S201, the height detector 46 creates a height map of the interior of the receiving space, i.e., the second container 92.

[0081] Next, in step S202, the searcher 41 searches for position candidates for the place position. Specifically, the searcher 41 creates multiple position candidates for the workpiece within the second container 92.

[0082] Next, in step S203, the searcher 41 assigns priorities to the location candidates. As described above, the searcher 41 assigns priorities to each of the multiple location candidates, for example, from the perspective of accommodation efficiency or accommodation stability.

[0083] In step S204, the searcher 41 reads out one relative position and orientation from the plurality of relative positions and orientations of the hand 14 stored in the storage unit 32. In step S205, the searcher 41 reads out the position candidate with the highest priority from the plurality of position candidates.

[0084] In step S206, the path generator 47 generates a target path for placing the workpiece at the read position candidate with the hand 14 in the read relative position and orientation holding the workpiece. At this time, the path generator 47 checks for interference with the target path and generates a target path that does not interfere. If interference is detected, the path generator 47 does not generate a target path.

[0085] In step S207, the searcher 41 determines whether or not a target path has been generated. If a target path has been generated, the searcher 41 sets the current relative position and orientation as a candidate for the relative position and orientation of the hand 14. In step S208, the searcher 41 registers a combination of the position candidate and the candidate for the relative position and orientation of the hand 14 as a place candidate.

[0086] On the other hand, failure to generate a target path means that the workpiece cannot be placed appropriately using the combination of the current position candidate and relative position and orientation. Therefore, if a target path is not generated, the searcher 41 determines in step S209 whether or not it has attempted to generate target paths for all position candidates using the current relative position and orientation. If it has not yet attempted to generate target paths for all position candidates, the searcher 41 returns to step S205 and reads out the next-highest-priority position candidate from the multiple position candidates. The path generator 47 then attempts to generate a target path for the new position candidate (step S206), and the searcher 41 determines whether or not a target path has been generated (step S207).

[0087] In this way, the searcher 41 searches for position candidates where the workpiece can be appropriately placed in the current relative position and orientation, in descending order of priority. When the searcher 41 finds a position candidate where the workpiece can be appropriately placed, the searcher 41 associates the position candidate with the current relative position and orientation.

[0088] However, if target path generation has already been attempted for all position candidates in step S209, the multiple position candidates do not include a position where the workpiece can be appropriately placed at the current relative position and orientation. In this case, no place candidate is registered for the current relative position and orientation. The searcher 41 returns to step S204 and searches for a place candidate for another relative position and orientation. Specifically, the searcher 41 reads another relative position and orientation from the multiple relative positions and orientations of the hand 14 stored in the memory 32, and repeats the processing from step S205 onwards.

[0089] After registering the place candidates in step S208, the searcher 41 determines whether or not the search for place candidates for all relative positions and orientations is complete in step S210. If the search for place candidates for all relative positions and orientations is not complete, the searcher 41 returns to step S204 and searches for place candidates for another relative position and orientation.

[0090] When the search for place candidates has been completed for all relative positions and orientations, the searcher 41 ends the place position planning.

[0091] By such a place position planning, a plurality of place candidates are generated, which are combinations of position candidates and candidates for the relative position and orientation of the hand 14. In other words, a set of place candidates is generated by the place position planning.

[0092] =Pickup Position Planning= Next, the pickup position planning will be described in detail. Fig. 11 is a flowchart of the pickup position planning subroutine.

[0093] In step S301, the determiner 42 reads out one candidate for the relative position and orientation of the hand 14 from among the multiple candidates for the relative position and orientation of the hand 14 included in the multiple place candidates. Specifically, the determiner 42 reads out the candidate for the relative position and orientation of the hand 14 defined by the place candidate with the highest priority among the multiple place candidates.

[0094] Next, in step S302, the determiner 42 associates the relative position and orientation candidates with each detected workpiece. Specifically, the determiner 42 determines the absolute or control position and orientation of the hand 14 when the hand 14 is positioned with respect to each detected workpiece using the relative position and orientation candidates. For example, this hand position and orientation is not a relative position and orientation with respect to the workpiece, but the position and orientation of the hand 14 in a coordinate system (e.g., a robot coordinate system) used to control the robot 1. In step S302, combinations of the positions and orientations of the detected workpieces and the hand 14 (this combination will be simply referred to as a "combination of detected workpieces and hand 14") are created according to the number of detected workpieces.

[0095] Next, in step S303, the determiner 42 assigns a priority to each combination of the detected workpiece and the hand 14. For example, the determiner 42 assigns a high priority to a combination including a position and posture of the hand 14 in which the amount of movement from the current position of the hand 14 is small.

[0096] In step S304, the determiner 42 reads out one combination from among a plurality of combinations of the detection workpiece and the hand 14. At this time, the determiner 42 reads out the combination with the highest priority.

[0097] Then, in step S305, the path generator 47 generates a target path for the robot arm 12 to the position and orientation of the hand 14, which is defined by the combination read from the current position of the hand 14. At this time, the path generator 47 checks for interference with the target path and generates a target path that is free of interference. Depending on the status of surrounding objects, the path generator 47 may not be able to generate a target path that is free of interference.

[0098] In step S306, the determiner 42 determines whether an interference-free target path has been generated. If an interference-free target path has not been generated, in step S307, the determiner 42 determines whether an attempt has been made to generate target paths for all of the multiple combinations of detected workpieces and hands 14. If the generation of target paths has not been attempted for all of the combinations, the determiner 42 returns to step S304 and reads out the combination with the next highest priority from the multiple combinations. Thereafter, the path generator 47 attempts to generate a target path for the new combination (step S305), and the determiner 42 determines whether a target path has been generated (step S306).

[0099] In this way, the determiner 42 searches for detected workpieces, in descending order of priority, for which an appropriate target path can be generated using the candidate relative position and orientation of the hand 14 set in step S301. When an appropriate target path is generated, the determiner 42 determines the target path as the target path for picking in step S308. In other words, the determiner 42 determines the detected workpiece when the appropriate target path is generated as the target workpiece W, and determines the candidate relative position and orientation of the hand 14 at that time as the relative position and orientation of the hand 14 for picking. When the relative position and orientation of the target workpiece W and the hand 14 for picking are determined, the determiner 42 terminates the pickup position planning. Furthermore, by determining the relative position and orientation of the hand 14 for picking, the position candidate specified by the corresponding place candidate is determined as the place target position.

[0100] However, if the generation of a target path has already been attempted for all combinations in step S307, there is no detection workpiece that can be appropriately picked with the currently set candidate relative position and orientation of the hand 14. In this case, in step S309, the determiner 42 determines whether the search for a detection workpiece that can be appropriately picked with the candidate relative position and orientation of the hand 14 defined in all place candidates has been completed. If the search for a detection workpiece with the candidate relative position and orientation of the hand 14 defined in all place candidates has not been completed, the determiner 42 returns to step S301 and reads out the candidate relative position and orientation of the hand 14 defined in the place candidate with the next highest priority among the multiple place candidates. Then, the processing from step S302 onwards is repeated for the newly read candidate relative position and orientation. In other words, the determiner 42 searches for a detection workpiece that can be appropriately picked with the candidate relative position and orientation of the hand 14 that has been changed.

[0101] On the other hand, if the search for the detected workpiece has been completed for the relative positions and orientations of the hand 14 defined in all the place candidates, the determiner 42 ends the pick-and-place process in step S310.

[0102] In this way, the determiner 42 determines the relative position and orientation of the hand 14 with respect to the target workpiece W to be picked from the candidates for the relative position and orientation of the hand 14 included in the place candidates. Furthermore, the determiner 42 determines the target workpiece W from among the multiple workpieces.

[0103] In addition, the relative position and posture of the hand 14 during picking is determined, and thus the target placement position of the target work W is also determined. In the placement position plan, the target paths for each of the placement candidates have already been created. Therefore, once the target placement position is determined, the target path for the placement operation is also determined accordingly.

[0104] Next, the parallel processing of the place position plan, the pick-up position plan, and the robot operation will be described with reference to Fig. 12. Fig. 12 is a timing chart of the place position plan, the pick-up position plan, and the robot operation.

[0105] For the place position planning, first, place photography is performed (S101), thereby acquiring an image of the inside of the second container 92. Next, the searcher 41 performs place position planning based on the image of the inside of the second container 92 (S102), thereby obtaining multiple place candidates.

[0106] After the placement position planning is completed, the gripping position planning is executed (S105). Prior to the gripping position planning, picking photography (S103) and workpiece detection (S104) are executed. Specifically, an image of the inside of the first container 91 is acquired by the picking photography. The workpiece is detected based on the image of the inside of the first container 91. The gripping position planning is executed based on the image of the inside of the first container 91, the detected workpiece, and the place candidate. The gripping position planning determines the target workpiece W and also determines the relative position and orientation of the hand 14 with respect to the target workpiece W.

[0107] After the gripping position plan is completed, the picking operation is executed. The operation controller 43 causes the robot 1 to pick up the target workpiece W at the relative position and posture of the hand 14 determined by the gripping position plan.

[0108] After the picking operation, the placing operation is executed. The operation controller 43 operates the robot 1 according to the target path determined by the placing position plan and the gripping position plan. As a result, the target workpiece W is placed at the placing target position.

[0109] In this way, one cycle of place position planning, pick position planning, picking operation, and place operation is completed.

[0110] In such pick-and-place processing, some processes are performed in parallel with other processes. For example, place photography can be performed in parallel with pick photography, workpiece detection, gripping position planning, or picking operation. Place position planning can be performed in parallel with pick photography, workpiece detection, gripping position planning, picking operation, or placing operation. Pick photography can be performed in parallel with place photography, place position planning, or placing operation. Grip position planning can be performed in parallel with place photography, picking operation, or placing operation.

[0111] In this example, the place position planning and the grip position planning for the next pick-and-place process are executed in parallel with the robot operation. Specifically, the place photography is executed in parallel with the picking operation in the previous pick-and-place process. Because the situation inside the second container 92 does not change during the picking operation, the next place photography can be executed during the picking operation. The pick photography is executed in parallel with the place operation in the previous pick-and-place process. Because the situation inside the first container 91 does not change during the placing operation, the pick photography can be executed.

[0112] The place position planning and the pick position planning are each performed in parallel with the picking and placing operations in the previous pick-and-place process. Because the place position planning and the pick position planning are each performed based on an image, the place position planning and the pick position planning can be performed during the robot operation.

[0113] In this way, several processes can be performed in parallel, thereby reducing the cycle time of the pick-and-place process.

[0114] However, the place position planning is performed based on an image captured before the completion of the placing operation of the previous pick-and-place process. In other words, the place position planning is performed based on an image of the state before the workpiece has been placed in the previous placing operation. Therefore, in the place position planning, the place target position of the previous pick-and-place process is excluded from the place candidate.

[0115] <Modification> Next, a modification of the pick-and-place process will be described. Fig. 13 is a flowchart of position planning according to the modification.

[0116] As described above, the place position planning is performed based on the image captured before the completion of the previous placing operation. Therefore, the place position planning determines the placement candidate, excluding the previous target position of the previous placing operation. However, there is a risk that the workpiece actually placed in the previous placing operation may be displaced from the target position of the placing operation. Alternatively, there is a risk that the previous placing operation may cause the existing workpiece to move. Therefore, the nth placing operation is performed after the n-2th (n is a natural number greater than or equal to 3) placing operation and before the n-1th placing operation. Whether or not the target position of the n-1th placing operation needs to be corrected is determined based on a comparison between the image captured by the nth placing operation and the image captured by the n-1th placing operation. Because the n-1th placing operation has not yet been performed at the time of the nth placing operation, the target position of the n-1th placing operation can be corrected.

[0117] The processes of steps S101, S102, S103, S104, and S105 in FIG. 13 are the same as the processes of steps S101, S102, S103, S104, and S105 in FIG.

[0118] More specifically, first, in step S101, the photography controller 44 causes the second camera 51B to perform place photography.

[0119] Next, in step S401, the searcher 41 determines the difference between the image of the current, i.e., nth, place photograph and the image of the previous, i.e., n-1th, place photograph. Specifically, the image difference is determined based on the height map created by the height detector 46. The searcher 41 detects the difference between the image of the nth place photograph and the image of the n-1th place photograph by the difference between the height map of the image of the nth place photograph and the height map of the image of the n-1th place photograph. The n-1th place photograph is performed before the n-2th place operation. The nth place photograph is performed after the n-2th place operation but before the n-1th place operation. Therefore, the difference in the height map corresponds to a change in the situation inside the second container 92 due to the n-2th place operation.

[0120] Next, in step S402, the searcher 41 determines the need to correct the placement target position of the previous, i.e., the (n-1)th, placing operation based on the image difference. Note that the (n-1)th placing operation has not yet been performed. The position of the workpiece placed in the (n-2)th placing operation can be determined from the difference between the image photographed in the nth placing operation and the image photographed in the (n-1)th placing operation. Furthermore, the movement of other workpieces due to the influence of the (n-2)th placing operation can also be determined from the difference between the image photographed in the nth placing operation and the image photographed in the (n-1)th placing operation.

[0121] For example, if the deviation between the target position for the (n-2)th placing operation and the position of the workpiece actually placed by the (n-2)th placing operation exceeds a predetermined range, the searcher 41 determines that the target position for the (n-1)th placing operation needs to be corrected. Alternatively, if another workpiece has entered the target position for the (n-1)th placing operation, the searcher 41 determines that the target position for the (n-1)th placing operation needs to be corrected.

[0122] If the searcher 41 determines that the (n-1)th place target position does not need to be corrected, the process proceeds to step S102, where the current, i.e., nth, place position plan is executed. The place position plan is as described above.

[0123] On the other hand, if the searcher 41 determines that the (n-1)th place target position needs to be corrected, then in step S403, the searcher 41 executes place position correction to correct the (n-1)th place target position. The operation controller 43 executes the (n-1)th place operation based on the corrected place target position. After executing the place position correction, the searcher 41 executes the nth place position plan in step S102.

[0124] In addition, since the place position correction corrects the target position for the (n-1)th place operation in accordance with the change in the second container 92 caused by the (n-2)th place operation, the processes of steps S401, S402, and S403 are not executed until the second position planning. Step S401 is executed from the third position planning onwards.

[0125] Next, the place position correction will be described in detail with reference to Fig. 14, which is a flowchart of the place position correction subroutine.

[0126] First, in step S501, the searcher 41 searches for position candidates for the place position. Specifically, the searcher 41 creates multiple position candidates for the workpiece within the second container 92. The searcher 41 searches for the position candidates using the height map created in step S401. Next, in step S502, the searcher 41 assigns priorities to the position candidates.

[0127] Steps S501 and S502 are similar to steps S202 and S203, respectively, except that these steps are for correcting the previous, i.e., n-1th, place target position, but are based on the image captured during the nth place photograph.

[0128] Thereafter, in step S503, the searcher 41 reads out the location candidate with the highest priority from the plurality of location candidates. This process is similar to step S205.

[0129] Then, in step S504, the path generator 47 generates a target path for placing the workpiece at the read-out position candidate. When this place position correction is being performed, a picking operation is being performed, and the relative position and orientation of the hand 14 cannot be changed. Therefore, the path generator 47 generates a target path for placing the workpiece at the position candidate with the already determined relative position and orientation of the hand 14. At this time, the path generator 47 performs an interference check of the target path to confirm whether or not there is interference. In other words, the path generator 47 generates a target path without interference. If there is interference, the path generator 47 does not generate a target path.

[0130] In step S505, the searcher 41 determines whether or not a target route has been generated. This process is the same as in step S207. If a target route has been generated, in step S506, the searcher 41 determines the position candidate as the previous, i.e., the (n-1)th, place target position, and sets the generated target route as the previous, i.e., the (n-1)th, place operation target route.

[0131] On the other hand, if a target route has not been generated, the searcher 41 determines in step S507 whether or not an attempt has been made to generate a target route for all location candidates. This process is the same as step S209. If an attempt has not been made to generate a target route for all location candidates, the searcher 41 returns to step S503 and reads out the location candidate with the next highest priority from the multiple location candidates. Thereafter, the route generator 47 attempts to generate a target route for the new location candidate (step S504), and the searcher 41 determines whether or not a target route has been generated (step S505).

[0132] If the searcher 41 has attempted to generate target paths for all position candidates, the searcher 41 ends the pick-and-place process in step S508.

[0133] In this way, the searcher 41 searches for a new place target position from among the position candidates in descending order of priority without changing the relative position and orientation that has already been determined. When the searcher 41 finds a position candidate where the workpiece can be placed appropriately, it corrects the place target position of the (n-1)th robot operation, i.e., the placing operation in the robot operation currently in progress, to the new position candidate.

[0134] This place position correction is completed before the start of a place operation in the robot operation currently in progress. After the place position correction, a place position plan (S102) for the robot operation next to the robot operation currently in progress is executed.

[0135] Such parallel processing of position planning will be described with reference to Fig. 15. Fig. 15 is a timing chart of the place position planning, the pickup position planning, and the robot operation according to the modified example.

[0136] In the example of Figure 15, a third place photograph is taken in parallel with the second picking operation. After the third place photograph, the searcher 41 calculates the difference between the image taken by the third place photograph and the image taken by the second place photograph (S401) and determines whether or not the place target position for the second place operation needs to be corrected (S402). The second place photograph is taken before the first place operation, and the third place photograph is taken after the first place operation. Therefore, by comparing the image taken by the third place photograph with the image taken by the second place photograph, changes in the situation inside the second container 92 caused by the first place operation can be determined.

[0137] When the searcher 41 determines that the second place target position needs to be corrected, it executes the place position correction (S403). The second place target position is corrected by the place position correction, and a target path is generated accordingly. The place position correction is completed before the second place operation. The operation controller 43 executes the second place operation according to the corrected place target position and target path.

[0138] After completing the place position correction, the searcher 41 executes the third place position planning.

[0139] In this way, when position planning and robot operation are performed in parallel, the next place position plan can be executed based on an image of the storage space before the completion of the place operation. Therefore, the nth place photograph is executed before the n-1th place operation, and the need to correct the target position for the n-1th place operation is determined using the image photographed for the nth place operation. If the target position for the place operation needs to be corrected, the target position for the place operation is corrected to a position where the workpiece can be appropriately placed with the current relative position and orientation of the hand 14. This allows the cycle time of the pick-and-place process to be shortened while appropriately placing the workpiece.

[0140] In this pick-and-place process, multiple candidates for the relative position and orientation of the hand 14 corresponding to candidate positions for placing the workpiece are created, and the relative position and orientation of the hand 14 at the time of picking is determined from the multiple candidate relative position and orientation of the hand 14. As a result, the workpiece is picked at a relative position and orientation of the hand 14 that enables the workpiece to be placed at the candidate position. As a result, the method for picking the workpiece can be determined after taking the workpiece placement into consideration in advance. Furthermore, multiple candidates for the relative position and orientation of the hand 14 corresponding to candidate positions for placing the workpiece are created, but the relative position and orientation of the hand 14 at the time of picking is determined from among them, so there is no need to calculate multiple relative positions and orientations of the hand 14 at the time of picking. In other words, the amount of calculation required to determine the relative position and orientation of the hand 14 at the time of picking can be reduced.

[0141] Furthermore, when there are multiple position candidates, priorities are assigned to the multiple position candidates. Then, it is determined whether or not the workpiece can be picked up, starting with the relative position and orientation of the hand 14 corresponding to the position candidate with the highest priority. Therefore, the relative position and orientation of the hand 14 at the time of picking can be made to correspond to the position candidate with the highest priority possible. As a result, it is possible to place the workpiece at a position candidate with a relatively high priority.

[0142] Furthermore, when determining the relative position and orientation of the hand 14 during picking, the target workpiece W to be picked is simultaneously determined from among multiple workpieces. In other words, it is possible to test multiple workpieces to see whether one of multiple candidates for the relative position and orientation of the hand 14 during placing can be used to properly pick the workpiece. In other words, it is possible to increase the number of options available when determining whether a workpiece can be picked using a candidate for the relative position and orientation of the hand 14.

[0143] Other Embodiments As described above, the above-described embodiments have been described as examples of the technology disclosed in the present application. However, the technology of the present disclosure is not limited to these embodiments and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above-described embodiments can be combined to create new embodiments. Furthermore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately determining that these non-essential components are essential.

[0144] For example, the robot system 100 incorporating the main control device 3 is not limited to one that transfers the target workpiece W from the first container 91 to the second container 92. For example, the robot 1 may perform devanning. The destination of the target workpiece W is not limited to the second container 92. For example, the destination of the target workpiece W may be a conveyor, a shelf, a pallet, or the like.

[0145] The robot arm 12 is not limited to a vertical articulated robot arm, but may be a horizontal articulated robot arm, a parallel link robot arm, a rectangular coordinate robot arm, a polar coordinate robot arm, or the like.

[0146] The suction device 15 does not have to rotate around the reference axis T relative to the robot arm 12. Even if the suction device 15 cannot rotate around the reference axis T, it only needs to be disposed at a position offset in the radial direction from the reference axis T of the robot arm 12, that is, at a position eccentric to the reference axis T. The suction device 15 may attract the workpiece by magnetic force or the like instead of negative pressure.

[0147] Furthermore, the holding of the workpiece by the hand 14 is not limited to suction, but may be gripping or the like.

[0148] The camera 51 does not have to include the first camera 51A and the second camera 51B. One camera 51 may capture an image of the inside of the first container 91 and an image of the inside of the second container 92. The camera 51 may not be fixedly disposed, but may be attached to the robot arm 12 and moved by the robot arm 12.

[0149] The robot system 100 may not be equipped with the camera 51. In that case, an image may be input to the main control device 3 from outside. The method of acquiring the image does not matter. The image input from outside may be stored in the memory 33 or the storage device 32.

[0150] The target workpieces to be picked may not be piled up randomly but may be aligned. The target workpieces are not limited to rectangular parallelepiped workpieces. For example, the target workpieces may have a substantially triangular prism shape. The target workpieces may also be bags filled with powder or granular material such as fertilizer, lime, or gravel.

[0151] The processes of the operation controller 43, the photography controller 44, and the workpiece detector 45 are merely examples. For example, the workpiece detector 45 can detect a workpiece from an image using various methods.

[0152] The processes of the searcher 41 and the determiner 42 are merely examples. For example, the searcher 41 may search for place location candidates using any method. Alternatively, the searcher 41 may receive place location candidates from an external source.

[0153] The searcher 41 assigns a position candidate to each of the candidates for the relative position and orientation of the hand 14, but is not limited to this. For example, the searcher 41 may assign a candidate for the relative position and orientation of the hand 14 to each of the position candidates.

[0154] The searcher 41 may create multiple place candidates by assigning multiple candidates for the relative position and orientation of the hand 14 to a single position candidate. Alternatively, the searcher 41 may create multiple place candidates by assigning multiple position candidates to a single candidate for the relative position and orientation of the hand 14.

[0155] When creating a place candidate, the searcher 41 determines whether the hand 14 and the robot arm 12 can hold the workpiece without interfering with other objects depending on whether a target path can be generated, but is not limited to this. For example, the searcher 41 may only determine whether the hand 14 and the robot arm 12 will interfere with other objects when the hand 14 holds the workpiece placed in the position candidate.

[0156] When determining the relative position and orientation of the hand 14 with respect to the target workpiece W, the determiner 42 determines whether the hand 14 and the robot arm 12 can hold the workpiece without interfering with other objects depending on whether a target path can be generated, but is not limited to this. For example, the determiner 42 may only determine whether the hand 14 and the robot arm 12 will interfere with other objects when the target workpiece is held by the hand 14.

[0157] When the determiner 42 determines the relative position and orientation of the hand 14, the target workpiece W may already be determined.

[0158] The control device does not have to be a single device like the main control device 3. The control device may include a plurality of separate devices. For example, the workpiece detector 45, the determiner 42, and the motion controller 43 may each be realized by a separate device.

[0159] The flowchart is merely an example. Steps in the flowchart may be changed, replaced, added, omitted, etc. as appropriate. The order of steps in the flowchart may also be changed, and serial processing may be performed in parallel. For example, the placement photography in step S101 in FIG. 9 may be performed consecutively with the picking photography in step S103 or the workpiece detection in step S104.

[0160] Additionally, position planning and robot movement may be performed sequentially rather than in parallel.

[0161] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a Central Processing Unit (CPU), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered a circuit or processing circuit. A processor may also be a programmable processor that executes a program stored in a memory.

[0162] In this specification, a circuit, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0163] If the hardware is a processor considered to be a type of circuitry, the circuit, means, or unit is a combination of hardware and software used to configure the hardware and / or processor.

[0164] The technology of the present disclosure can be summarized as follows.

[0165] [1] The main control device 3 (control device) is a control device that controls a robot 1 that has a robot arm 12 and a hand 14 attached to the robot arm 12 and performs pick-and-place processing of a workpiece, and is equipped with a searcher 41 that searches for multiple candidates for the relative position and orientation of the hand 14 with respect to a workpiece to be placed at a candidate place position, and a determiner 42 that determines the relative position and orientation of the hand 14 with respect to a workpiece W to be picked from multiple candidates for the relative position and orientation of the hand 14 at the time of placement.

[0166] According to this configuration, first, multiple candidates for the relative position and orientation of the hand 14 corresponding to the candidate placement positions are created, and then the relative position and orientation of the hand 14 with respect to the workpiece W to be picked is determined from the multiple candidates for the relative position and orientation of the hand 14. Because multiple candidates are provided when determining the relative position and orientation of the hand 14 during picking, it becomes easier to achieve appropriate holding of the workpiece during picking. Furthermore, because the multiple candidates correspond to the candidate placement positions, the workpiece picked at the relative position and orientation of the hand 14 determined from the multiple candidates can be appropriately placed at the candidate position. As a result, both picking and placing of the workpiece can be performed appropriately.

[0167] [2] In the main control device 3 described in [1], the searcher 41 searches for a plurality of the position candidates and also searches for a plurality of candidates for the relative position and posture of the hand 14 corresponding to the plurality of the position candidates, and the determiner 42 determines the position candidate corresponding to the candidate for the relative position and posture of the hand 14 at the time of placement, which is determined as the relative position and posture of the hand 14 with respect to the target workpiece W, as the target position for placing the workpiece.

[0168] According to this configuration, the target placement position is determined by determining the relative position and posture of the hand 14 during picking. That is, the relative position and posture of the hand 14 during picking and the target placement position are determined in relation to each other. As a result, both picking and placing of the workpiece can be performed appropriately.

[0169] [3] In the main control device 3 described in [1] or [2], the searcher 41 assigns priorities to multiple position candidates, and the determiner 42 determines whether multiple candidates for the relative position and orientation of the hand 14 at the time of placement are the relative position and orientation of the hand 14 with respect to the target workpiece W, in order of the priority of the corresponding position candidates.

[0170] According to this configuration, multiple candidates for the relative position and orientation of the hand 14 corresponding to multiple position candidates are created, and priorities are assigned to the multiple position candidates. Then, whether or not the multiple candidates for the relative position and orientation of the hand 14 become the relative position and orientation of the hand 14 with respect to the target workpiece W is determined in order of the priority of the corresponding position candidates. This makes it more likely that a candidate for the relative position and orientation of the hand 14 corresponding to a position candidate with a relatively high priority will be determined as the relative position and orientation of the hand 14 during picking. As a result, the workpiece is placed at a position candidate with a relatively high priority.

[0171] [4] In the main control device 3 described in any one of [1] to [3], when determining the relative position and posture of the hand 14 with respect to the target work W to be picked from among multiple candidates for the relative position and posture of the hand 14 at the time of placement, the determiner 42 determines the target work W from among multiple works.

[0172] According to this configuration, when there are multiple options for the target workpiece, the determiner 42 also determines the target workpiece from among the multiple works when determining the relative position and orientation of the hand 14. Since the options for the target workpiece increase, the range of choices for determining the relative position and orientation of the hand 14 that can pick up the target workpiece is broadened.

[0173] [5] The main control device 3 described in any one of [1] to [4] further includes an operation controller 43 that causes the robot arm 12 and the hand 14 to execute robot operations including a picking operation for picking the target work W and a placing operation for placing the target work W, and the searcher 41 and the determiner 42 search for candidates for the relative position and posture of the hand 14 at the time of placing for the next picking operation and the next placing operation, and determine the relative position and posture of the hand 14 with respect to the target work W to be picked, in parallel with the robot operation by the operation controller 43.

[0174] According to this configuration, the robot operation including the picking operation and the placing operation, the search for candidates for the relative position and orientation of the hand 14 during placing, and the determination of the relative position and orientation of the hand 14 during picking are performed in parallel, thereby shortening the cycle time of the pick-and-place process.

[0175] [6] The main control device 3 described in any one of [1] to [5] further includes a photography controller 44 that causes a first camera 51A to perform picking photography to obtain an image of the work before it is picked, and causes a second camera 51B to perform place photography to obtain an image of the space where the work is to be placed, wherein the searcher 41 searches for the position candidates and candidates for the relative position and posture of the hand 14 based on the image obtained by the place photography, the determiner 42 determines the relative position and posture of the hand 14 with respect to the target work W based on the image obtained by the pick photography, the photography controller 44 causes the second camera 51B to perform the n-th place photography after the n-2th place operation (n is a natural number of 3 or more) and before the n-1th place operation, and the searcher 41 corrects the place target position of the n-1th place operation based on the image obtained by the nth place photography.

[0176] According to this configuration, the searcher 41 searches for position candidates and candidates for the relative position and orientation of the hand 14 based on the image captured during placement photography. The determiner 42 determines the relative position and orientation of the hand 14 with respect to the target workpiece W based on the image captured during picking photography. The nth placement photography is performed after the n-2th placement operation and before the n-1th placement operation. In other words, the n-1th placement photography is performed before the n-2th placement operation. The target placement position for the n-1th placement operation is basically determined based on the image captured before the n-2th placement operation. Therefore, the completion of the n-2th placement operation may change the situation of the space where the placement is scheduled. Since the nth placement photography is performed after the n-2th placement operation, an image of the space where the placement is scheduled after the completion of the n-2th placement operation is obtained. Furthermore, the nth placement operation is performed before the start of the n-1th placement operation. Therefore, the searcher 41 corrects the target position of the (n-1)th placing operation based on the image captured during the nth placing operation, thereby enabling the (n-1)th placing operation to be performed more appropriately.

[0177] [7] The robot system 100 comprises a robot 1 having a robot arm 12 and a hand 14 attached to the robot arm 12, and a main control device 3 described in any one of [1] to [6].

[0178] According to this configuration, first, multiple candidates for the relative position and orientation of the hand 14 corresponding to the candidate placement positions are created, and then the relative position and orientation of the hand 14 with respect to the workpiece W to be picked is determined from the multiple candidates for the relative position and orientation of the hand 14. Because multiple candidates are provided when determining the relative position and orientation of the hand 14 during picking, it becomes easier to achieve appropriate holding of the workpiece during picking. Furthermore, because the multiple candidates correspond to the candidate placement positions, the workpiece picked at the relative position and orientation of the hand 14 determined from the multiple candidates can be appropriately placed at the candidate position. As a result, both picking and placing of the workpiece can be performed appropriately.

[0179] [8] A control method for a robot 1 having a robot arm 12 and a hand 14 attached to the robot arm 12 and performing pick-and-place processing of a workpiece includes searching for multiple candidates for the relative position and orientation of the hand 14 with respect to a workpiece to be placed at a candidate placement position, and determining the relative position and orientation of the hand 14 with respect to a workpiece W to be picked from multiple candidates for the relative position and orientation of the hand 14 at the time of placement.

[0180] According to this configuration, first, multiple candidates for the relative position and orientation of the hand 14 corresponding to the candidate placement positions are created, and then the relative position and orientation of the hand 14 with respect to the workpiece W to be picked is determined from the multiple candidates for the relative position and orientation of the hand 14. Because multiple candidates are provided when determining the relative position and orientation of the hand 14 during picking, it becomes easier to achieve appropriate holding of the workpiece during picking. Furthermore, because the multiple candidates correspond to the candidate placement positions, the workpiece picked at the relative position and orientation of the hand 14 determined from the multiple candidates can be appropriately placed at the candidate position. As a result, both picking and placing of the workpiece can be performed appropriately.

[0181] 100 Robot system 1 Robot 12 Robot arm 14 Hand 3 Main control device (control device) 41 Searcher 42 Deciding device 43 Motion controller 44 Photography controller 51A First camera 51B Second camera W Target work

Claims

1. A control device for controlling a robot having a robot arm and a hand attached to the robot arm, which performs pick-and-place processing of a workpiece, comprising: a searcher that searches for multiple candidates for the relative position and orientation of the hand with respect to a workpiece to be placed at a candidate place position; and a determiner that determines the relative position and orientation of the hand with respect to the workpiece to be picked from multiple candidates for the relative position and orientation of the hand at the time of placement.

2. A control device as described in claim 1, wherein the searcher searches for a plurality of the position candidates and also searches for a plurality of candidates for the relative position and posture of the hand corresponding to the plurality of the position candidates, and the determiner determines the position candidate corresponding to the candidate for the relative position and posture of the hand at the time of placement, which has been determined as the relative position and posture of the hand with respect to the target workpiece, as the placement target position of the workpiece.

3. A control device as described in claim 2, wherein the searcher assigns priorities to a plurality of position candidates, and the determiner determines whether a plurality of candidates for the relative position and posture of the hand at the time of placement are the relative position and posture of the hand with respect to the target workpiece in order of the priority of the corresponding position candidates.

4. A control device as described in claim 1, wherein the determiner determines the target work from among a plurality of works when determining the relative position and posture of the hand with respect to a work to be picked from among a plurality of candidates for the relative position and posture of the hand at the time of placing.

5. A control device as described in claim 1, further comprising an operation controller that causes the robot arm and the hand to execute robot operations including a picking operation for picking the target work and a placing operation for placing the target work, and the searcher and the determiner search for candidates for the relative position and posture of the hand at the time of placing for the next picking operation and placing operation, in parallel with the robot operation performed by the operation controller, and determine the relative position and posture of the hand with respect to the target work for picking.

6. A control device as described in claim 5, further comprising an imaging controller which causes a first camera to perform picking photography to obtain an image of the work before it is picked, and causes a second camera to perform place photography to obtain an image of the space in which the work is to be placed, wherein the searcher searches for the position candidates and candidates for the relative position and posture of the hand based on the images taken by the place photography, and the determiner determines the relative position and posture of the hand with respect to the target work based on the images taken by the pick photography, the imaging controller causes the second camera to perform the nth place photography after the n-2th (n is a natural number equal to or greater than 3) place operation and before the n-1th place operation, and the searcher corrects the place target position of the n-1th place operation based on the image taken by the nth place photography.

7. A robot system comprising: a robot having a robot arm and a hand attached to the robot arm; and a control device according to any one of claims 1 to 6.

8. A method for controlling a robot having a robot arm and a hand attached to the robot arm, which performs pick-and-place processing of a workpiece, the method including: searching for multiple candidates for the relative position and orientation of the hand with respect to a workpiece to be placed at a candidate place position; and determining the relative position and orientation of the hand with respect to the workpiece to be picked from the multiple candidates for the relative position and orientation of the hand at the time of placing.

Citation Information

Patent Citations

  • Robot setting device, robot system, robot setting method, robot setting program and recording medium readable by computer as well as equipment with the same recorded

    JP2018144159A

  • Picking facility

    JP2019042828A

  • Picking system, control device, picking method, program and storage medium

    JP2023072410A

  • Action generation device, robot system, action generation method, and action generation program

    WO2022153373A1

  • Robot system, movement path generation device and movement path generation method

    WO2022162969A1