Robot system, picking method, and computer program

Abstract

A robot system 100 comprises: a robot 1 having a hand 13 which performs a picking operation including a grasping operation for grasping a target workpiece from among a plurality of workpieces W; a camera 2 for imaging the plurality of workpieces W; and a control device 4. The control device 4 includes: a generating unit 431 which, when the hand 13 is performing the picking operation, generates a planned route of the hand 13 for the next picking operation to be performed with respect to the next target workpiece, on the basis of a first captured image captured by the camera 2 before the start of the picking operation; a determining unit 432 which, when the grasping operation is complete, performs an interference determination for determining whether the hand 13 will interfere with the workpieces W on the planned route, on the basis of a difference between the first captured image and a second captured image captured by the camera 2 after the grasping operation is complete, and sets the planned route determined not to interfere as a determined route; and a control unit 434 which causes the hand 13 to perform the next picking operation in accordance with the determined route.

Description

[Technical Field]

[0001] The technology disclosed herein relates to a robot system, a picking method, and a computer program. [Background technology]

[0002] BACKGROUND ART Conventionally, as disclosed in Patent Document 1, for example, a robot that manually picks a target workpiece from a plurality of randomly piled workpieces has been known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-69542 Summary of the Invention

[0004] In the robot described above, a hand path for the next picking operation is generally generated each time a picking operation by the hand is completed. The hand then performs the next picking operation along the generated path. This picking method has the problem of lengthening the cycle time of the picking operation.

[0005] The technology disclosed herein has been made in consideration of the above points, and its purpose is to shorten the cycle time of picking work.

[0006] The robot system disclosed herein includes a robot having a hand that performs a picking operation, which includes a gripping operation to grip a target workpiece from among multiple workpieces and a transport operation to transport the target workpiece to a predetermined position after the gripping operation; a camera device that photographs the multiple workpieces; and a control device that controls the robot. The control device includes a generation unit, a determination unit, and a control unit. While the hand is performing the picking operation, the generation unit generates a planned path for the hand for the next picking operation to be performed on the next target workpiece based on a first captured image captured by the camera device before the start of the picking operation. When the gripping operation in the picking operation is completed, the determination unit performs an interference determination to determine whether the hand will interfere with the workpiece on the planned path based on the difference between a second captured image captured by the camera device after the gripping operation is completed and the first captured image. The determined planned path is designated as the determined path. The control unit causes the hand to perform the next picking operation according to the determined path.

[0007] The picking method disclosed herein is a method for performing a picking operation in which a hand of a robot performs a gripping operation to grip a target workpiece from among a plurality of workpieces and a transport operation to transport the target workpiece to a predetermined position after the gripping operation. This picking method includes: while the hand is performing the picking operation, generating a planned path for the hand for a next picking operation to be performed on the next target workpiece based on a first captured image of the plurality of workpieces taken before the start of the picking operation; when the gripping operation in the picking operation is completed, performing an interference determination to determine whether the hand will interfere with the workpiece on the planned path based on a difference between a second captured image of the plurality of workpieces taken after the gripping operation is completed and the first captured image; determining the planned path if it is determined that no interference will occur as a determined path; and having the hand perform the next picking operation according to the determined path.

[0008] The computer program disclosed herein causes a computer to realize a function of performing a picking operation in which a robot hand performs a gripping operation to grip a target workpiece from among multiple workpieces and a transport operation to transport the target workpiece to a predetermined position after the gripping operation. This computer program causes the computer to realize the following functions: while the hand is performing the picking operation, generate a planned path for the hand for a next picking operation to be performed on the next target workpiece based on a first captured image of the multiple workpieces taken before the picking operation begins; when the gripping operation in the picking operation is completed, perform an interference determination to determine whether the hand will interfere with the workpiece on the planned path based on a difference between a second captured image of the multiple workpieces taken after the gripping operation is completed and the first captured image, and set the planned path where it is determined that there will be no interference as a determined path; and cause the hand to perform the next picking operation according to the determined path.

[0009] The robot system disclosed herein can shorten the cycle time of the picking operation.

[0010] According to the picking method disclosed herein, the cycle time of the picking work can be shortened.

[0011] The computer program disclosed herein can shorten the cycle time of picking work. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a robot system. [Figure 2] FIG. 2 is a block diagram showing a schematic configuration of the control device and its peripheral devices. [Figure 3] FIG. 3 is a functional block diagram of the processing unit of the control device. [Figure 4] FIG. 4 is a flowchart showing various processes in the processing unit. [Figure 5] FIG. 5 is a time chart showing an example of a picking cycle. [Figure 6] FIG. 6 is a flowchart showing the position determination process in the processing unit. [Figure 7] FIG. 7 is a flowchart showing the processing of collision detection in the processing unit. [Figure 8] FIG. 8 is a diagram illustrating an example of depth information of the first captured image. [Figure 9] FIG. 9 is a diagram showing an example of depth information of the second captured image. [Figure 10] FIG. 10 is a diagram showing the difference in depth information between the first captured image in FIG. 8 and the second captured image in FIG. [Figure 11] FIG. 11 is a diagram illustrating an example of depth information of the first captured image. [Figure 12] FIG. 12 is a diagram illustrating an example of depth information of the second captured image. [Figure 13] FIG. 13 is a diagram showing the difference in depth information between the first captured image in FIG. 11 and the second captured image in FIG. [Figure 14] FIG. 14 is an enlarged view of the target workpiece and its surroundings in FIG. [Figure 15] FIG. 15 is a diagram showing an example of depth information of a second captured image when the depth information of the first captured image is that shown in FIG. [Figure 16] FIG. 16 is a diagram showing the difference in depth information between the first captured image in FIG. 11 and the second captured image in FIG. [Figure 17] FIG. 17 is a diagram showing an example of depth information of the first captured image. [Figure 18] FIG. 18 is a diagram illustrating an example of depth information of the second captured image. [Figure 19] FIG. 19 is a diagram showing the difference in depth information between the first captured image in FIG. 17 and the second captured image in FIG. [Figure 20] FIG. 20 is a time chart showing an example of a picking cycle. DETAILED DESCRIPTION OF THE INVENTION

[0013] Exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0014] Fig. 1 is a diagram showing the schematic configuration of a robot system 100. The robot system 100 includes a robot 1, a camera 2, a robot control device 3, and a control device 4. The robot system 100 is a picking system in which the robot 1 picks a target workpiece from a plurality of workspieces W randomly piled in a container C, for example, and transports the target workpiece to a predetermined location. The target workpiece is a workpiece to be picked.

[0015] The robot 1 has a base 11, an arm 12 connected to the base 11, and a hand 13 connected to the arm 12. The arm 12 is a so-called vertically articulated arm, and has multiple links connected to each other so that they can rotate freely in the vertical direction. The base end of the arm 12 is connected to the base 11 so that it can rotate freely. The hand 13 is connected to the tip of the arm 12 and grips a target workpiece. In other words, the hand 13 moves by the arm 12 and performs a picking operation by gripping the target workpiece. In this example, the hand 13 is a suction-type hand that grips the target workpiece by suction.

[0016] In this example, the "picking operation" includes an approaching operation, a grasping operation, a transporting operation, and a returning operation. The approaching operation is an operation in which the hand 13 moves from a predetermined start position to the position of the target workpiece and approaches the target workpiece. The grasping operation is an operation in which the hand 13 grasps the target workpiece after the approaching operation. The transporting operation is an operation in which the hand 13 transports the target workpiece to a predetermined position after the grasping operation. In other words, the transporting operation is an operation in which the hand 13 transports the grasped target workpiece to a predetermined transport position. The returning operation is an operation in which the hand 13 returns to the start position after the transporting operation. In this way, the picking operation involves the approaching operation, grasping operation, transporting operation, and returning operation, in that order. In short, the picking operation is a series of operations in which the hand 13 moves in the space above multiple workpieces W to approach the target workpiece, grasps the target workpiece, and then moves in the space above multiple workpieces W to return to the start position.

[0017] Camera 2 photographs multiple workpieces W inside container C. Specifically, camera 2 is positioned above the multiple workpieces W, i.e., above container C. In other words, camera 2 is a fixed-point camera. Camera 2 photographs three-dimensional information of the multiple workpieces W. In other words, camera 2 is a camera with a depth sensor, and photographs depth information in the Z-axis direction in addition to planar information, which is two-dimensional information. The depth information is distance information to the surfaces of the multiple workpieces W. Camera 2 outputs the photographed images of the multiple workpieces W to control device 4. Camera 2 is an example of an imaging device. Note that instead of camera 2, a laser scanner, for example, may be used as the imaging device.

[0018] The control device 4 controls the robot 1. Specifically, the control device 4 controls various operations of the arm 12 and the hand 13 via the robot control device 3. The control device 4 and the robot control device 3 are capable of communicating with each other via wire or wirelessly. The control device 4 generates a path for the hand 13 to pick up the target work, i.e., a picking path for the hand 13, based on the captured image output from the camera 2, and causes the arm 12 and the hand 13 to perform the picking operation according to the picking path.

[0019] 2 is a block diagram showing a schematic configuration of the control device 4 and its peripheral devices. The control device 4 includes an input unit 41, a storage unit 42, and a processing unit 43.

[0020] The input unit 41 receives an input operation from a user and outputs an input signal corresponding to the input operation to the processing unit 43. The input unit 41 is, for example, a touch panel, a pointing device such as a mouse, or a keyboard.

[0021] The storage unit 42 is a computer-readable storage medium that stores various programs and various data. The storage unit 42 is formed of a magnetic disk such as a hard disk, an optical disk such as a CD-ROM or DVD, or a semiconductor memory.

[0022] Specifically, the storage unit 42 stores a computer program 421 and the like. The computer program 421 is a program for causing a computer, i.e., the processing unit 43, to realize various functions for performing a picking operation in which the hand 13 of the robot 1 picks up a target workpiece. The computer program 421 is read and executed by the processing unit 43.

[0023] 3 is a functional block diagram of the processing unit 43 of the control device 4. The processing unit 43 has various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and / or a DSP (Digital Signal Processor), and various semiconductor memories such as a RAM (Random Access Memory) and / or a ROM (Read Only Memory). The processing unit 43 reads and executes a computer program 421 etc. from the storage unit 42.

[0024] Specifically, the processing unit 43 has a generating unit 431, a determining unit 432, a correcting unit 433, and a control unit 434 as functional blocks.

[0025] While the hand 13 is performing a picking operation, the generation unit 431 generates a planned path of the hand 13 for the next picking operation to be performed on the next target workpiece, based on the first captured image captured by the camera 2 before the start of the picking operation. The planned path is the path from which the hand 13 moves from a predetermined start position to its return to the start position, that is, the path of the hand 13 when the hand 13 performs a series of operations from the approach operation to the return operation. In other words, the generation unit 431 does not generate a path for the next picking operation after the picking operation has ended and before the next picking operation starts, but generates a path for the next picking operation while the picking operation is being performed.

[0026] The first captured image is an image of the workpieces W in the container C before the picking operation, i.e., before the approaching operation of the picking operation. Specifically, the first captured image is an image capturing three-dimensional information of a plurality of workpieces W. The generation unit 431 generates a planned path of the picking operation for causing the hand 13 to pick up the target workpiece, based on the three-dimensional information of the target workpiece included in the first captured image, i.e., the planar information and depth information of the target workpiece.

[0027] More specifically, the generation unit 431 generates multiple planned paths when the hand 13 is performing a picking operation. In this example, the multiple planned paths are paths for picking up each of multiple different target works. The generation unit 431 sequentially generates each of the multiple planned paths.

[0028] When the gripping operation in the picking operation is completed, the determination unit 432 performs an interference determination to determine whether or not the hand 13 will interfere with the workpiece W on the planned path based on the difference between the second captured image and the first captured image taken by the camera 2 after the gripping operation is completed, and sets the planned path for which it is determined that there will be no interference as the determined path. In other words, the interference determination is a determination of whether or not the hand 13 will interfere with a workpiece W other than the target workpiece when the hand 13 performs the picking operation according to the planned path. In this example, the determination unit 432 performs the interference determination based on the difference between the second captured image and the first captured image taken by the camera 2 after the return operation is completed, i.e., after the picking operation is completed.

[0029] The second captured image is an image of the workpieces W in the container C after the gripping operation, more specifically, after the return operation, i.e., after the picking operation. Like the first captured image, the second captured image is an image capturing three-dimensional information of the multiple workpieces W.

[0030] Specifically, the determination unit 432 performs interference determination based on the difference between the depth information of the first captured image and the depth information of the second captured image. More specifically, the determination unit 432 determines that interference occurs when an area where the height of the surfaces of the plurality of workpieces W in the second captured image is higher than the height of the surfaces of the plurality of workpieces W in the first captured image exists on the planned route, and determines that interference does not occur when the area does not exist on the planned route.

[0031] The determination unit 432 performs interference determination for each of the multiple planned routes generated by the generation unit 431. If there are multiple planned routes determined not to interfere, the determination unit 432 determines the planned route having the shortest estimated time required for the picking operation from among the multiple planned routes determined not to interfere as the determined route. In other words, the determination unit 432 selects one planned route that is estimated to have the shortest movement time for the hand 13 from among the multiple planned routes determined not to interfere.

[0032] Hereinafter, the time estimated to be required for the picking operation may be simply referred to as the "estimated time of the picking operation." For example, the estimated time of the picking operation can be calculated by assuming that the movement speed of the hand 13 is constant and dividing the movement distance of the hand 13 on the planned path by the constant speed. For example, the constant speed may be several tens of percent of the maximum movement speed of the hand 13.

[0033] Furthermore, the determination unit 432 in this example performs position determination to determine whether the position of the target workpiece has changed based on the difference between the first captured image and the second captured image, and if it determines that the position of the target workpiece has not changed, performs the above-mentioned interference determination. In other words, the determination unit 432 performs position determination and interference determination in this order.

[0034] The correction unit 433 corrects the planned path when the determination unit 432 determines that the position of the target workpiece has changed and the amount of change in the position of the target workpiece is equal to or less than a predetermined value. The determination unit 432 performs interference determination for the planned path corrected by the correction unit 433.

[0035] The control unit 434 causes the hand 13 to perform the next picking operation according to the determined path. Specifically, the control unit 434 outputs control signals to the robot control device 3 to cause the arm 12 and the hand 13 to perform the next picking operation. The robot control device 3 controls the actuators (not shown) of the arm 12 and the hand 13 based on the control signals from the control unit 434.

[0036] <Picking work> The picking operation by the robot system 100 described above will now be described. In the picking operation, the following picking method is performed by the control device 4. Specifically, the processing unit 43 performs the picking method by reading and executing the computer program 421 from the storage unit 42. FIG. 4 is a flowchart showing various processes in the processing unit 43. FIG. 5 is a time chart showing an example of a picking cycle. In FIG. 5, two patterns, (A) and (B), are illustrated for the second picking cycle. In FIG. 5, the first picking cycle of pattern (B) is not shown, but is the same as pattern (A). In addition, in the two patterns of the second picking cycle in FIG. 5, the same reference numerals are used to designate common operations and processes.

[0037] First, in step S1, the processing unit 43 acquires a first captured image. Specifically, as shown in Fig. 5, in the first picking cycle, which is the first picking cycle to be performed, the first captured image is captured by the photographing operation m1 of the camera 2. The camera 2 outputs the captured first captured image to the processing unit 43. In this way, the processing unit 43 acquires the first captured image.

[0038] In the following step S2, the generation unit 431 generates a determined path. That is, as shown in FIG. 5, the generation unit 431 performs generation process m2, and this generation process m2 generates a determined path. Specifically, the generation unit 431 generates a determined path based on the first captured image. More specifically, the generation unit 431 determines the position and orientation of the target work based on the three-dimensional information of the first captured image. The generation unit 431 generates a determined path for causing the hand 13 to pick up the target work based on the position and orientation of the target work. The determined path is a path for picking operation m3 performed in the first picking cycle. Because this first picking cycle is the first picking cycle, a determined path is generated before picking operation m3 as a path for the hand 13 performing picking operation m3.

[0039] In the following step S3, the control unit 434 causes the hand 13 to perform a series of picking operations according to the determined path. That is, as shown in FIG. 5, when the generation process m2 of the generation unit 431 is completed, the hand 13 performs a picking operation m3. Specifically, the hand 13 moves from a predetermined start position to the position of the target work by an approach operation, and then grasps the target work by a grasping operation. Next, the hand 13 transports the target work to a predetermined transport position by a transport operation, and then returns to the start position again by a return operation. As a result, the target work is transported from the container C to the predetermined transport position.

[0040] In the following step S4, the generation unit 431 generates multiple planned paths. That is, as shown in FIG. 5, while the picking operation m3 of the first picking cycle is being performed, the generation unit 431 performs the generation process m4. In other words, the picking operation m3 of step S3 and the generation process m4 of step S4 are performed in parallel. The generation unit 431 completes the generation process m4 before the picking operation m3 is completed. In the generation process m4, multiple planned paths of the hand 13 for the next picking operation, that is, for example, the picking operation m7 of the second picking cycle shown in FIG. 5A, are generated.

[0041] Specifically, the generation unit 431 generates multiple planned paths based on the first captured image. More specifically, the generation unit 431 determines the position and orientation of the target work to be picked in the next picking operation based on the three-dimensional information of the first captured image. The generation unit 431 generates a planned path for causing the hand 13 to pick the target work based on the position and orientation of the target work. The generation unit 431 sequentially generates a planned path for each of multiple different target workpieces.

[0042] In this way, since the planned path for the next picking operation is generated while the hand 13 is performing the picking operation m3, it is not necessary to reserve additional time for generating the path in the next second picking cycle, and therefore the cycle time CT2 of the second picking cycle, i.e., the next picking operation, can be shortened.

[0043] When the picking operation m3 is completed, that is, when the hand 13 transports the target work and returns to the predetermined start position, the first picking cycle is completed and the process proceeds to step S5, where the second picking cycle is started.

[0044] In step S5, the processing unit 43 acquires the second captured image. Specifically, as shown in FIG. 5, in the second picking cycle, the second captured image is captured by the camera 2's shooting operation m5. The camera 2 outputs the captured second captured image to the processing unit 43. In this way, the processing unit 43 acquires the second captured image. Note that this second captured image is used as the first captured image in the generation process m8 that is performed later. Therefore, in step S5, it can also be said that the processing unit 43 acquires the first captured image.

[0045] In the subsequent steps S6 to S10, the determination unit 432 performs a determination process. That is, as shown in, for example, pattern (A) in FIG. 5, a determination process m6 is performed before the picking operation m7 in the second picking cycle. In this determination process m6, before the picking operation m7, it is determined whether or not the planned path generated in the first picking cycle is an appropriate path, and a determined path is set as the path when the hand 13 actually performs the picking operation m7. Specifically, the determination process by the determination unit 432 includes a position determination process, an interference determination process, and a determined path setting process.

[0046] 6 is a flowchart showing the position determination process in the processing unit 43. Specifically, in step S6, the determination unit 432 performs the position determination process. This position determination process is performed sequentially for each of the multiple planned routes.

[0047] In step Sa1 of the position determination process, a position determination is performed to determine whether the position of the target workpiece on the planned path has changed. That is, it is determined whether the position of the target workpiece has changed before and after picking operation m3 of the first picking cycle. Specifically, the determination unit 432 determines whether the position of the target workpiece has changed based on the difference between the first captured image and the second captured image. This is because a change in the position of the target workpiece will result in a difference between the first captured image and the second captured image.

[0048] More specifically, the determination unit 432 determines whether there is a difference between the planar information of the first captured image and the planar information of the second captured image and whether the position of the target workpiece is included in the difference area. That is, the determination unit 432 determines that the position of the target workpiece has changed if the position of the target workpiece is included in the difference area, and determines that the position of the target workpiece has not changed if the position of the target workpiece is not included in the difference area. Furthermore, the determination unit 432 determines that the position of the target workpiece has not changed if there is no difference between the planar information of the first captured image and the planar information of the second captured image.

[0049] Thus, if it is determined in step Sa1 that the position of the target workpiece has not changed, the process proceeds to step Sa2. In step Sa2, the determination unit 432 extracts the planned route for which it has determined that the position of the target workpiece has not changed as a candidate for the determined route. When the position determination process has been performed for all planned routes, step S6 ends and the process proceeds to step S7. In step S7, the determination unit 432 determines whether there is one or more planned routes extracted as candidates for the determined route in step S6. If one or more planned routes have been extracted, the process proceeds to step S8. Note that steps Sa3 and subsequent steps in FIG. 6 will be described later.

[0050] 7 is a flowchart showing the interference determination process in the processing unit 43. In step S8, the determination unit 432 performs the interference determination process. This interference determination process is performed sequentially for all of the planned routes extracted in step S6.

[0051] Fig. 8 is a diagram showing an example of depth information of a first captured image, Fig. 9 is a diagram showing an example of depth information of a second captured image, and Fig. 10 is a diagram showing the difference between the depth information of the first captured image in Fig. 8 and the depth information of the second captured image in Fig. 9.

[0052] In step Sb1 of the interference detection process, it is determined whether or not there is an area where the surface height of multiple workpieces W inside the container C is elevated. Specifically, the determination unit 432 makes this determination based on, for example, the difference between the depth information of the first captured image in FIG. 8 and the depth information of the second captured image in FIG. 9. As shown in FIGS. 8 and 9, in the first captured image and the second captured image, the internal space of the container C and the space above the container C are virtually partitioned into multiple areas R. This also applies to FIGS. 11 to 16, which will be described later. In this example, for ease of explanation, the description will be based on two-dimensional depth information in, for example, the YZ plane, rather than three-dimensional depth information.

[0053] The second captured image in FIG. 9 shows the state after the target workpiece Wa has been picked by picking operation m3 of the first picking cycle in the first captured image in FIG. 8. The target workpiece Wb is an example of a target workpiece to be picked in the second picking cycle. As can be seen from this second captured image, in the second picking cycle, when the target workpiece Wa is picked by the first picking cycle, the other workpieces W, including the target workpiece Wb, are not displaced. Therefore, as shown in FIG. 10, when the depth information of the first captured image is compared with the depth information of the second captured image, there is no region R where the height of the surface Fb of the workpiece W in the second captured image is higher than the height of the surface Fa of the workpiece W in the first captured image. Therefore, in this case, the process proceeds from step Sb1 to step Sb2.

[0054] In step Sb2, the determination unit 432 determines that the planned route for which it has determined that there is no region R with a high surface height is a non-interfering route, and extracts it as a candidate for the determined route. In this way, when there is no region R with a high surface height, all of the planned routes extracted in step S6 are determined to be non-interfering routes, and are extracted as candidates for the determined route.

[0055] As described above, the hand 13 performs a picking operation by moving in the space above the multiple workpieces W. The absence of an area R where the surface of the workpieces W is elevated means that the space above the workpieces W, which is the movement space of the hand 13, has not been restricted since the planned path was generated. Therefore, even if the hand 13 performs a picking operation according to the planned path, it can be assumed that there is no risk of the hand 13 interfering with other workpieces W, etc.

[0056] Conversely, in this example, there is an area R1 in the first photographed image where the surface height of the workpiece W is lowered, as shown in Fig. 10. If such an area R1 exists, it can be assumed that the space above the workpiece W, which is the movement space of the hand 13, has increased, and the risk of the hand 13 interfering with other workpieces W or the like is further reduced.

[0057] Fig. 11 is a diagram showing an example of depth information of a first captured image. Fig. 12 is a diagram showing an example of depth information of a second captured image. Fig. 13 is a diagram showing the difference between the depth information of the first captured image in Fig. 11 and the depth information of the second captured image in Fig. 12. Fig. 14 is an enlarged view of the target workpiece and its surroundings in Fig. 13.

[0058] The second photographed image in Fig. 12 shows the state after the target workpiece Wa has been picked by picking operation m3 of the first picking cycle in the first photographed image in Fig. 11. Specifically, the second photographed image in Fig. 12 shows the state in which, when the target workpiece Wa is picked, the workpiece Wc collapses and hits the target workpiece Wb, causing the target workpiece Wb to be displaced. In this example, the target workpiece Wb is displaced in the Y-axis direction.

[0059] In this case, in step Sa1 in the position determination process, the determination unit 432 determines that the position of the target workpiece Wb has changed, and the process proceeds to step Sa3. Specifically, as shown in Fig. 12 etc., there is an area R2 where there is a difference between the planar information of the first captured image and the planar information of the second captured image, and the position of the target workpiece Wb is included in the area R2 where there is a difference, so the determination unit 432 determines that the position of the target workpiece Wb has changed.

[0060] In step Sa3, the determination unit 432 determines whether the amount of change in the position of the target workpiece Wb is equal to or less than a predetermined value. If the amount of change in the position of the target workpiece Wb is equal to or less than the predetermined value, the process proceeds to step Sa4, assuming that the change in the position of the target workpiece Wb is a small change.

[0061] In step Sa4, the correction unit 433 corrects the planned path. As shown in FIG. 12, the correction unit 433 corrects the planned path Q to a planned path Qa to match the position of the target workpiece Wb after displacement. More specifically, the correction unit 433 does not correct the entire planned path Q, but only partially corrects the planned path Q. In other words, the planned path Q is corrected so that it becomes a smooth curved or straight path from a midpoint on the planned path Q to the position of the target workpiece Wb after displacement. In the subsequent step Sa2, the corrected planned path Qa is extracted as a candidate for the determined path.

[0062] In this way, even if the position of the target workpiece Wb has changed, if the amount of change is small, it is possible to have the hand 13 pick it up by correcting the planned path Q, and such a corrected planned path Qa is also extracted as a candidate for the determined path.

[0063] Furthermore, if it is determined in step Sa3 that the amount of change in the position of the target workpiece Wb is greater than a predetermined value, the process proceeds to step Sa5, where the planned route for which the amount of change in the position of the target workpiece Wb is determined to be greater than a predetermined value is excluded from the candidates for the determined route.

[0064] In this way, when the amount of change in the position of the target workpiece Wb is large, there is a high risk that the hand 13 will not reach the position of the target workpiece Wb during the picking operation, and such a planned path is excluded from the candidates for the determined path. By excluding such a planned path, it is possible to prevent picking failures, such as the hand 13 being unable to access the target workpiece Wb.

[0065] In this way, the corrected planned path Qa extracted as a candidate for the determined path in step S6 is also subjected to interference determination processing in step 8. In this case, in step Sb1, the determination unit 432 determines that there is an area R2 where the surface height of the multiple workpieces W in the container C is higher, and the process proceeds to step Sb3. That is, as shown in Figures 13 and 14, there is an area R2 where the height of the surface Fb of the target workpiece Wb in the second captured image is higher than the height of the surface Fa of the target workpiece Wb in the first captured image due to displacement of the target workpiece Wb.

[0066] Subsequently, in step Sb3, the determination unit 432 determines whether the planned route is the corrected planned route Qa. In this case, since the planned route is the corrected planned route Qa, the process proceeds to step Sb4.

[0067] In step Sb4, it is determined whether or not the region R2 with a higher surface height exists in any location other than the position of the target workpiece Wb. In this case, as shown in Fig. 13, the region R2 with a higher surface height exists only in the position of the target workpiece Wb, so the process proceeds to step Sb2. Then, in step Sb2, the determination unit 432 determines that the corrected planned path Qa is a path that does not interfere, and extracts it as a candidate for the determined path.

[0068] In this way, even if there is an area R2 where the surface height is higher, if the planned path is a corrected planned path Qa and area R2 exists only at the position of the target work Wb, the risk of interference due to the presence of area R2 is eliminated by correcting the planned path, and such corrected planned path Qa is also extracted as a candidate for the determined path.

[0069] Fig. 15 is a diagram showing an example of depth information of a second captured image in which the depth information of the first captured image is that of Fig. 11. Fig. 16 is a diagram showing the difference between the depth information of the first captured image in Fig. 11 and the depth information of the second captured image in Fig. 15.

[0070] The second captured image in Fig. 15 shows the state after the target workpiece Wa has been picked by picking operation m3 of the first picking cycle in the first captured image in Fig. 11. Specifically, the second captured image in Fig. 15 shows the state in which the workpiece Wc has collapsed when the target workpiece Wa has been picked. In this example, when the depth information of the first captured image is compared with the depth information of the second captured image, there is an area where the height of the surface Fb of the workpiece W in the second captured image is higher than the height of the surface Fa of the workpiece W in the first captured image, as shown by hatching in Fig. 16.

[0071] In this case, in the interference determination process, the process proceeds from step Sb1 to step Sb3, where it is determined whether the planned route Q is a corrected planned route. If the planned route Q is not a corrected planned route, the process proceeds to step Sb5.

[0072] In step Sb5, the determination unit 432 sets the area where the surface height is increased as a no-passage area RS. The no-passage area RS is an area where the hand 13 may interfere with a workpiece W other than the target workpiece Wb when the hand 13 performs a picking operation. Next, in step Sb5, the determination unit 432 determines whether the no-passage area RS exists on the planned route Q. In this example, as shown in FIG. 16, the no-passage area RS does not exist on the planned route Q. Therefore, in this case, the process proceeds from step Sb5 to step Sb2, and the planned route Q is extracted as a candidate for the determined route.

[0073] Also, if it is determined in step Sb4 that the area with the elevated surface height exists other than the position of the target workpiece Wb, the process proceeds to step Sb5. In this case, in step Sb5, the area with the elevated surface height other than the area existing at the position of the target workpiece Wb is set as the no-passage area RS. In other words, the area with the elevated surface height existing at the position of the target workpiece Wb is excluded from the no-passage area RS.

[0074] In this way, even if there is an area with a higher surface height, if that area is not on the planned path Q, there is no risk of the hand 13 interfering with work W other than the target work Wb when the hand 13 performs the picking operation, and such planned path Q is also extracted as a candidate for the determined path.

[0075] If it is determined in step Sb5 that a no-passage area RS exists on the planned route, the process proceeds to step Sb6, where the planned route that is determined to have a no-passage area RS is determined to be an interfering route and is excluded from the candidates for the determined route.

[0076] In this way, if a no-passage area RS exists on the planned route, there is a high risk that the hand 13 will interfere with other workpieces W during the picking operation, and such a planned route is excluded from the candidates for the determined route. By excluding such a planned route, it is possible to prevent picking failures, such as the hand 13 interfering with other workpieces W, before they occur.

[0077] When the interference detection process has been performed for all planned routes in this manner, step S8 ends and the process proceeds to step S9. In step S9, the determination unit 432 determines whether or not one or more planned routes have been extracted as candidates for the determined route in step S8. If one or more planned routes have been extracted, the process proceeds to step S10.

[0078] In step S10, the determination unit 432 performs a process of setting a determined route. Specifically, if there is one planned route extracted in step S8, i.e., one planned route determined not to interfere, the determination unit 432 sets that planned route as the determined route. Furthermore, if there are multiple planned routes extracted in step S8, the determination unit 432 selects one planned route from the multiple planned routes and sets it as the determined route.

[0079] Specifically, the determination unit 432 selects the planned path with the shortest estimated time for the picking operation from the extracted multiple planned paths. The estimated time for the picking operation is the sum of the time it takes for the hand 13 to move along the planned path and the time it takes for the hand 13 to grasp the target workpiece Wb. By selecting the planned path with the shortest estimated time for the picking operation in this way, the cycle time CT2 of the second picking cycle can be shortened. Once the determined path is set in this way, step S10 ends, and the series of determination processes by the determination unit 432 ends.

[0080] In the following step S11, the control unit 434 causes the hand 13 to perform a picking operation according to the determined path set in step S10. That is, as shown in FIG. 5, when the determination process m6 of the determination unit 432 is completed, the hand 13 performs a picking operation m7. As in the picking operation m3 of the first picking cycle, the hand 13 moves from the start position to the target workpiece Wb by an approach operation and grasps the target workpiece Wb by a grasping operation. Next, the hand 13 transports the target workpiece Wb by a transport operation, and then returns to the start position by a return operation. As a result, the target workpiece Wb is transported from the container C.

[0081] In step S12, the generation unit 431 determines whether or not a new planned path can be generated. This determination is made based on whether or not there is a pickable workpiece W in the container C. Specifically, the generation unit 431 confirms the presence of a pickable workpiece W based on the first captured image acquired in step S5. If there is a pickable workpiece W, the generation unit 431 determines that the planned path can be generated, and if there is no pickable workpiece W, the generation unit 431 determines that the planned path cannot be generated. For example, if there is no workpiece W in the container C, if there is no workpiece W in an appropriate position, if there is no workpiece W with an appropriate posture, etc., it determines that the planned path cannot be generated.

[0082] If it is determined in step S12 that a planned route can be generated, the process proceeds to step S13.

[0083] In step S13, the generation unit 431 generates multiple planned paths. That is, as shown in FIG. 5, the generation unit 431 performs a generation process m8 while the picking operation m7 of the second picking cycle is being performed. In other words, the picking operation m7 of step S11 and the generation process m8 of step S13 are performed in parallel. The generation unit 431 completes the generation process m8 before the picking operation m7 is completed. In the generation process m8, multiple planned paths of the hand 13 for the next picking operation, i.e., the picking operation of the third picking cycle (not shown), are generated. That is, the generation unit 431 sequentially generates multiple planned paths based on the first captured image, similar to step S4 described above.

[0084] When the picking operation m7 is completed, that is, when the hand 13 transports the target workpiece Wb and returns to the predetermined start position, the second picking cycle is completed and the process returns to step S5. When the process returns to step S5, the third picking cycle starts. In this manner, the picking cycle is repeated.

[0085] Thus, in the second picking cycle shown in FIG. 5A, before the hand 13 performs the picking operation m7, a determination process m6 is performed on the planned path generated during the previous first picking cycle. The determination process m6 extracts a planned path that prevents the hand 13 from interfering with other workpieces W, and the picking operation is performed according to the planned path. Therefore, the picking operation can be performed appropriately. That is, the planned path is generated based on the first captured image captured before the picking operation m3 of the first picking cycle. Therefore, if the position or posture of the workpiece W changes during the gripping or transporting operation in the picking operation m3 of the first picking cycle, there is a risk that the picking operation will not be performed appropriately even if the picking operation is performed according to the planned path. However, this example solves this problem.

[0086] The determination process is then performed based on the difference between the first captured image and the second captured image. Because the determination process is performed using a simple method, it can be completed in a shorter time than, for example, the time required for the generation process m2 of the first picking cycle. Therefore, the cycle time CT2 of the second picking cycle is shortened. In other words, the cycle time CT2 of the second picking cycle can be shorter than the cycle time CT1 of the first picking cycle, which generates a path before the picking operation.

[0087] On the other hand, if it is determined in step S7 that there is no planned route extracted as a candidate for the determined route in step S6, the process proceeds to step S2. Also, if it is determined in step S9 that there is no planned route extracted as a candidate for the determined route in step S8, the process also proceeds to step S2. In step S2, the generation unit 132 generates a determined route. That is, as shown in (B) of FIG. 5, if it is determined in determination process m10 by the determination unit 432 that there is no planned route that can be a candidate for the determined route, the generation unit 431 performs generation process m11.

[0088] In this generation process m11, similar to the generation process m2 in the first picking cycle, a determined path is generated based on the first captured image. In this case, the first captured image is the image captured by the photographing operation m5 in the second picking cycle, i.e., the captured image acquired in step S5. In this case, the determined path is the path for the picking operation m7 performed in the second picking cycle.

[0089] Once the determined path is generated, the process proceeds to step S3, where the control unit 434 causes the hand 13 to perform a picking operation according to the determined path. That is, as shown in FIG. 5B, when the generation process m11 by the generation unit 431 is completed, a picking operation m7 similar to the second picking cycle of (A) is performed. In the following step S4, the generation unit 431 generates multiple planned paths. That is, as shown in FIG. 5B, while the picking operation m7 is being performed, a generation process m8 is performed similar to the second picking cycle of (A). When the picking operation m7 is completed, the second picking cycle is completed, and the process proceeds to step S5. When the process proceeds to step S5, a third picking cycle is started.

[0090] 5B, if all planned routes are determined to be inappropriate as a result of the determination process m10, a generation process m11 is performed to generate a determined route. Therefore, although the cycle time CT3 of the second picking cycle is slightly longer than the cycle time CT1 of the first picking cycle, picking work can be continued.

[0091] On the other hand, if the generation unit 431 determines in step S12 that a planned path cannot be generated, the generation process m8 by the generation unit 431 is not performed. In other words, while the picking operation m7 is being performed, a planned path is not generated. Then, when the picking operation m7 is completed, the second picking cycle is completed and the process proceeds to step S14. When the process proceeds to step S14, the third picking cycle starts. In step S14, the processing unit 43 acquires a first captured image. That is, in the third picking cycle (not shown), the first captured image is captured by the shooting operation of the camera 2. The camera 2 outputs the captured first captured image to the processing unit 43.

[0092] In the following step S15, the generation unit 431 determines whether or not the determined path can be generated. This determination is made based on whether or not there is a pickable workpiece W in the container C, similar to the determination process in step S12. Specifically, the generation unit 431 reconfirms the presence of a pickable workpiece W based on the first captured image acquired in step S14. This is because the picking operation m7 of the second picking cycle may cause the workpiece W to, for example, collapse, and may displace to an appropriate position where it can be picked, or may change to an appropriate posture where it can be picked. If there is a pickable workpiece W, the generation unit 431 determines that the determined path can be generated, and if there is no pickable workpiece W, the generation unit 431 determines that the determined path cannot be generated.

[0093] If it is determined in step S15 that the determined route can be generated, the process returns to step S2, and the generation unit 431 generates the determined route. If it is determined in step S15 that the determined route cannot be generated, the process by the processing unit 43, i.e., the picking operation, ends.

[0094] As described above, the robot system 100 of this example is provided with a control device 4 having a generation unit 431, a determination unit 432, and a control unit 434. While the hand 13 is performing a picking operation, the generation unit 431 generates a planned path for the hand 13 for the next picking operation to be performed on the next target workpiece Wb based on the first captured image captured by the camera 2 before the start of the picking operation. When the picking operation is completed, the determination unit 432 performs an interference determination to determine whether the hand 13 will interfere with the workpiece W on the planned path based on the difference between the second captured image captured by the camera 2 after the picking operation is completed and the first captured image, and sets the planned path determined not to interfere as the determined path. The control unit 434 causes the hand 13 to perform the next picking operation according to the determined path.

[0095] This configuration generates a planned path for the hand 13 to pick the next target workpiece Wb during a picking operation, thereby shortening the cycle time of the next picking operation. Then, interference is detected for the planned path, and the hand 13 is caused to perform the next picking operation using the planned path determined to be free of interference as the determined path. This allows the hand 13 to perform the picking operation appropriately without interfering with the workpiece W. That is, because the planned path is generated based on the first captured image captured before the picking operation, if the position of the workpiece W changes due to the gripping or transporting operation during the picking operation, there is a risk that the picking operation will not be performed appropriately even if the picking operation is performed according to the planned path. However, the robot system 100 of this example can solve this problem. Furthermore, interference detection is performed using a simple method based on the difference between the first and second captured images, allowing interference detection to be performed in a short time. Therefore, the cycle time of the picking operation can be shortened. This, combined with the effect of being able to perform the picking operation appropriately, effectively shortens the cycle time.

[0096] Furthermore, in the robot system 100 of this example, the camera 2 is positioned above the multiple workpieces W and captures three-dimensional information of the multiple workpieces W. The determination unit 432 then performs interference determination based on the difference between the depth information of the first captured image and the depth information of the second captured image. Specifically, the determination unit 432 determines that interference occurs if an area where the height of the surfaces Fb of the multiple workpieces W in the second captured image is higher than the height of the surfaces Fa of the multiple workpieces W in the first captured image exists on the planned path, and determines that interference does not occur if the area does not exist on the planned path.

[0097] Generally, the hand 13 moves in the space above multiple workpieces W to perform picking operations. The fact that there is no area where the surface height of the workpieces W is high on the planned path means that the planned path is not restricted by other workpieces W. Therefore, even if the hand 13 is made to perform picking operations along the planned path, it can be assumed that there is no risk of the hand 13 interfering with other workpieces W, etc. In this way, interference determination can be performed effectively by performing interference determination based on the depth information of the captured image, specifically, the surface height of the workpieces W.

[0098] In the robot system 100 of this example, the generation unit 431 generates multiple planned paths, and the determination unit 432 performs interference determination for each of the multiple planned paths. This makes it possible to prevent the planned paths extracted in step S7 and step S9 from being determined to be non-existent. As a result, the cycle time can be reliably shortened.

[0099] Furthermore, in the robot system 100 of this example, when there are multiple planned routes that are determined not to interfere, the determination unit 432 determines the planned route that requires the shortest time for picking operation among the multiple planned routes that are determined not to interfere as the determined route. This can further reduce the cycle time of the picking operation.

[0100] Furthermore, in the robot system 100 of this example, the determination unit 432 performs position determination to determine whether the position of the target workpiece Wb has changed based on the difference between the first captured image and the second captured image, and if it determines that the position of the target workpiece Wb has not changed, performs interference determination. This allows for more appropriate picking operations because the planned path is one in which the position of the target workpiece Wb has not changed and the hand 13 does not interfere with other workpieces W. In other words, if the position of the target workpiece Wb has changed, there is a risk of a picking error occurring, such as the hand 13 not reaching the target workpiece Wb, but this example makes it possible to prevent such a picking error from occurring.

[0101] Furthermore, in the robot system 100 of this example, the control device 4 further includes a correction unit 433 that corrects the planned path when the determination unit 432 determines that the position of the target workpiece Wb has changed and the amount of change in the position of the target workpiece Wb is equal to or less than a predetermined value. The determination unit 432 then performs an interference determination for the planned path corrected by the correction unit 433. Even if the position of the target workpiece Wb has changed, if the amount of change is small, it is possible to have the hand 13 pick up the workpiece by correcting the planned path. With this configuration, even such a planned path can be used as the determined path as much as possible.

[0102] Other Embodiments As described above, the above embodiment has been described as an example of the technology disclosed in this application. However, the technology of the present disclosure is not limited to this and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, the components described in the above embodiment can be combined to create new embodiments. Furthermore, the components described in the accompanying drawings and detailed description may include not only components essential for solving the problem, but also components that are not essential for solving the problem in order to exemplify the technology. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not be interpreted as immediately determining that these non-essential components are essential.

[0103] For example, as shown in FIGS. 17 to 19, in the first and second captured images, a space including the container C may be virtually divided into multiple regions R. FIG. 17 is a diagram showing an example of depth information for the first captured image. FIG. 18 is a diagram showing an example of depth information for the second captured image. FIG. 19 is a diagram showing the difference between the depth information for the first captured image in FIG. 17 and the depth information for the second captured image in FIG. 18. In this case, the first and second captured images include the height of the surfaces of not only the multiple workpieces W but also the height of the surface of the container C as depth information.

[0104] The second photographed image in Fig. 18 shows the state after the target workpiece Wa has been picked by, for example, picking operation m3 of the first picking cycle in the first photographed image in Fig. 17. Specifically, the second photographed image in Fig. 18 shows the state in which the container C has moved when the target workpiece Wa has been picked, resulting in the displacement of the target workpiece Wb. In this example, the target workpiece Wb is displaced in the same direction as the container C, as shown by the arrow in Fig. 18.

[0105] In this case, when the depth information of the first captured image is compared with the depth information of the second captured image, as shown by hatching in FIG. 19, there are many no-passage areas RS where the height of the surface Fb of the workpiece W or the like in the second captured image is higher than the height of the surface Fa of the workpiece W or the like in the first captured image. Here, for example, if the amount of change in the position of the target workpiece Wb is less than a predetermined value, the planned route is corrected. Then, as shown in FIG. 19, if no no-passage areas RS exist on the corrected planned route Qa, the planned route Qa is extracted as a candidate for the determined route. In other words, a planned route Qa is extracted in which the hand 13 does not interfere with not only the workpiece W but also the container C.

[0106] In this way, by dividing the space including the container C into a plurality of regions R, if the container C moves, it is possible to determine whether the hand 13 will interfere with the container C. Therefore, more appropriate interference determination can be performed.

[0107] Furthermore, in step S5, the processing unit 43 may acquire the second captured image not after the end of the picking operation m3 of the first picking cycle, but during the picking operation m3. FIG. 20 is a time chart showing an example of a picking cycle. As shown in FIG. 20, while the picking operation m3 is being performed, the second captured image is captured by the photographing operation m5 of the camera 2. The camera 2 outputs the captured second captured image to the processing unit 43.

[0108] Specifically, the photographing operation m5 of the camera 2 is performed after the gripping operation in the picking operation m3 is completed, for example, during the transport operation. Of the series of picking operations, the operation that is most likely to change the position, etc., of the next target workpiece Wb is the gripping operation. Therefore, by acquiring the second photographed image taken at least after the gripping operation is completed, it is possible to accurately grasp that the position, etc., of the target workpiece Wb has changed from the difference between the first photographed image and the second photographed image.

[0109] In the following step S6, similar to the above embodiment, the determination unit 432 performs a determination process based on the difference between the first captured image and the second captured image. Specifically, as shown in FIG. 20, when the picking operation m3 is being performed, the determination unit 432 starts a determination process m6. More specifically, the determination process m6 is performed immediately after the photographing operation m5 of the camera 2. Then, when the picking operation m3 is completed, the first picking cycle is completed and the second picking cycle is started. At this time, for example, if the determination process m6 is completed, a picking operation m7 and a generation process m8 are performed, similar to the above embodiment. Then, in this second picking cycle, after the grasping operation in the picking operation m7 is completed, the photographing operation m15 of the camera 2 and the determination unit 432 performs a determination process m16. In this manner, the picking cycle is repeated.

[0110] In this way, after the gripping operation in the picking operation is completed, the cycle time CT4 of the second picking cycle can be further reduced by taking the second photographed image and performing the determination process of the determination unit 432. In other words, while the picking operation is being performed, not only is a planned path generated, but also taking the second photographed image and determining the planned path can be performed, thereby further reducing the cycle time of the next picking cycle.

[0111] The hand 13 may also be configured to be switchable between a single state in which only one target workpiece Wb is gripped in a gripping operation, and a multi-state in which multiple target works Wb are gripped together in a gripping operation.

[0112] In this case, the process of setting the determined path in step 10 in Fig. 4 may be performed as follows. If there is only one planned path extracted in step S8, i.e., one planned path determined not to interfere, the determination unit 432 sets that planned path as the determined path. Furthermore, if there are multiple planned paths extracted in step S8, the determination unit 432 selects a planned path for grasping by the hand 13 in a multi-state as the determined path from among the multiple planned paths. In other words, if the planned paths extracted in step S8 include both a planned path for grasping by the hand 13 in a single state and a planned path for grasping by the hand 13 in a multi-state, the latter planned path is selected as the determined path.

[0113] In this way, by using the planned path for the hand 13 to grip in the multi-state as the determined path, rather than the planned path for the hand 13 to grip in the single state, the cycle time for each target workpiece Wb can be shortened. Therefore, the cycle time for the picking operation can be substantially shortened.

[0114] Furthermore, the determination unit 432 may omit the position determination process. In other words, the determination unit 432 may perform the interference determination process and the determined path setting process as a series of determination processes.

[0115] Furthermore, the processing unit 43 may omit the correction unit 433. In other words, the determination unit 432 may exclude a planned route in which the position of the target workpiece Wb has changed from candidates for the determined route, regardless of the amount of change in the position.

[0116] Furthermore, the hand 13 is not limited to a suction type, and may be, for example, a type having a pair of fingers that grip the target workpiece Wb by opening and closing.

[0117] The above-described flowcharts are merely examples. Steps in the flowcharts may be changed, replaced, added, omitted, etc. as appropriate. The order of steps in the flowcharts may also be changed, and serial processing may be performed in parallel.

[0118] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered a circuit or processing circuit. A processor may also be a programmable processor that executes a program stored in a memory.

[0119] In this specification, a circuit, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0120] If the hardware is a processor considered to be a type of circuitry, the circuit, means, or unit is a combination of hardware and software used to configure the hardware and / or processor.

[0121] As described above, the robot system 100 according to the first aspect of the technology of the present disclosure includes a robot 1 having a hand 13 that performs a picking operation including a gripping operation to grip target workpieces Wa, Wb from multiple workpieces W and a transport operation to transport the target workpieces Wa, Wb to a predetermined position after the gripping operation, a camera 2 that photographs the multiple workpieces W, and a control device 4 that controls the robot 1. The control device 4 includes a generation unit 431 that, while the hand 13 is performing a picking operation, generates a planned path for the hand 13 for a next picking operation to be performed on the next target workpiece Wb based on a first captured image captured by the camera 2 before the start of the picking operation, a determination unit 432 that, when the gripping operation in the picking operation is completed, performs an interference determination to determine whether the hand 13 will interfere with the workpiece W on the planned path based on the difference between the first captured image and a second captured image captured by the camera 2 after the gripping operation is completed, and sets the planned path that is determined not to interfere as the determined path, and a control unit 434 that causes the hand 13 to perform the next picking operation according to the determined path.

[0122] According to this configuration, a planned path for the hand 13 to pick the next target workpiece Wb is generated during a picking operation, thereby shortening the cycle time of the next picking operation. Then, interference is detected for the planned path, and the hand 13 is caused to perform the next picking operation using the planned path determined to be free of interference as the determined path. This allows the hand 13 to perform the picking operation appropriately without interfering with the workpiece W. Furthermore, interference detection is performed using a simple method based on the difference between a first captured image taken before the picking operation and a second captured image taken after the completion of the gripping operation in the picking operation. This allows interference detection to be processed in a short time. In other words, interference detection can be performed in a shorter time than the time required to generate a path before the picking operation as in the conventional method. Therefore, the cycle time of the picking operation can be shortened. This, combined with the effect of being able to perform the picking operation appropriately, effectively shortens the cycle time.

[0123] Furthermore, in the robot system 100 according to a second aspect of the technology of the present disclosure, in the robot system 100 according to the first aspect, the camera 2 captures three-dimensional information of a plurality of workpieces W. Then, the determination unit 432 performs interference determination based on the difference between the depth information of the first captured image and the depth information of the second captured image.

[0124] In addition, in the robot system 100 relating to the third aspect of the technology of the present disclosure, in the robot system 100 relating to the first or second aspect, the judgment unit 432 judges that interference will occur if an area R where the height of the surface Fb of the multiple workpieces W in the second captured image is higher than the height of the surface Fa of the multiple workpieces W in the first captured image exists on the planned path, and judges that no interference will occur if the area R does not exist on the planned path.

[0125] According to these configurations, the hand 13 generally moves in the space above the multiple workpieces W to perform the picking operation. Therefore, interference determination can be performed effectively by determining the difference in depth information of the captured image, i.e., the change in the height direction of the multiple workpieces W, more specifically, the change in the height of the surfaces of the multiple workpieces W. In other words, since there is no area on the planned path where the surface height of the workpieces W is elevated, it can be seen that the planned path is not restricted by other workpieces W. Therefore, even if the hand 13 is made to perform the picking operation according to the planned path, it can be estimated that there is no risk of the hand 13 interfering with other workpieces W, etc. In this way, interference determination can be performed effectively.

[0126] Furthermore, in the robot system 100 relating to a fourth aspect of the technology of the present disclosure, the generation unit 431 generates a plurality of planned paths in the robot system 100 relating to any one of the first to third aspects, and the determination unit 432 performs interference determination for each of the plurality of planned paths.

[0127] This configuration increases the number of planned routes that can become the determined route, thereby increasing the possibility that the determined route will be set, thereby ensuring a reduction in cycle time.

[0128] In addition, in the robot system 100 relating to a fifth aspect of the technology of the present disclosure, in the robot system 100 relating to any one of the first to fourth aspects, when there are multiple planned routes that are determined not to interfere, the determination unit 432 determines as the determined route the planned route that has the shortest estimated time required for the picking operation among the multiple planned routes that are determined not to interfere.

[0129] This configuration further reduces the cycle time of the picking operation. Moreover, since the time required for the picking operation accounts for a relatively high proportion of the cycle time, shortening the time required for the picking operation can effectively shorten the cycle time.

[0130] Furthermore, in the robot system 100 according to a sixth aspect of the technology of the present disclosure, in the robot system 100 according to any one of the first to fifth aspects, the hand 13 is configured to be switchable between a single state in which only one target workpiece Wb is gripped in a gripping operation and a multi state in which multiple target workpieces Wb are gripped collectively in a gripping operation. When there are multiple planned paths determined to not cause interference, the determination unit 432 determines as the determined path the planned path that the hand 13 grips in the multi state from among the multiple planned paths determined to not cause interference.

[0131] According to this configuration, the cycle time per target workpiece Wb is shortened by selecting the planned path for the hand 13 to grip in the multi-state rather than the planned path for the hand 13 to grip in the single state as the determined path, thereby substantially shortening the cycle time for the picking operation.

[0132] In addition, in the robot system 100 relating to the seventh aspect of the technology of the present disclosure, in the robot system 100 relating to any one of the first to sixth aspects, the judgment unit 432 performs position judgment to determine whether the position of the target workpiece Wb has changed based on the difference between the first captured image and the second captured image, and if it is determined that the position of the target workpiece Wb has not changed, performs interference judgment.

[0133] According to this configuration, the picking operation can be performed more appropriately because the determined path is a planned path in which the position of the target workpiece Wb has not changed and the hand 13 does not interfere with other workpieces W. In other words, if the position of the target workpiece Wb has changed, there is a risk of a picking error occurring, such as the hand 13 not reaching the target workpiece Wb, but this technology can prevent such a picking error from occurring.

[0134] Furthermore, in the robot system 100 according to an eighth aspect of the technology of the present disclosure, in the robot system 100 according to any one of the first to seventh aspects, the control device 4 further includes a correction unit 433 that corrects the planned path when the determination unit 432 determines that the position of the target workpiece Wb has changed and the amount of change in the position of the target workpiece Wb is equal to or less than a predetermined value. Then, the determination unit 432 performs an interference determination for the planned path corrected by the correction unit 433.

[0135] Even if the position of the target workpiece Wb has changed, if the amount of change is small, it is possible to correct the planned path and have the hand 13 pick it up. With this configuration, even such a planned path can be adopted as the determined path.

[0136] Furthermore, a picking method according to a ninth aspect of the technique of the present disclosure is a method in which the hand 13 of the robot 1 performs a picking operation including a gripping operation in which target workpieces Wa, Wb are gripped from a plurality of workpieces W, and a transport operation in which the target workpieces Wa, Wb are transported to a predetermined position after the gripping operation. This picking method includes: while the hand 13 is performing the picking operation, generating a planned path for the hand 13 for a next picking operation to be performed on the next target workpiece Wb based on a first photographed image of the plurality of workpieces W taken before the start of the picking operation; when the gripping operation in the picking operation is completed, performing an interference determination to determine whether the hand 13 will interfere with the workpiece W on the planned path based on a difference between a second photographed image of the plurality of workpieces W taken after the completion of the gripping operation and the first photographed image; determining the planned path for which it is determined that there will be no interference as a determined path; and causing the hand 13 to perform the next picking operation according to the determined path.

[0137] According to this configuration, similar to the robot system 100 according to the first aspect, the cycle time of the picking operation can be shortened.

[0138] Furthermore, a computer program 421 according to a tenth aspect of the technology of the present disclosure is a program that causes the computer 43 to realize a function of the hand 13 of the robot 1 performing a picking operation including a gripping operation in which target workpieces Wa, Wb are gripped from multiple workpieces W and a transport operation in which the target workpieces Wa, Wb are transported to a predetermined position after the gripping operation. This computer program 421 causes the computer to realize the following functions: a function of generating a planned path for the hand 13 for a next picking operation to be performed on the next target workpiece Wb based on a first photographed image of the multiple workpieces W taken before the start of the picking operation when the hand 13 is performing the picking operation; a function of performing an interference determination, upon completion of the gripping operation in the picking operation, based on a difference between a second photographed image of the multiple workpieces W taken after the completion of the gripping operation and the first photographed image, to determine whether the hand 13 will interfere with the workpiece W on the planned path and set the planned path determined not to interfere as the determined path; and a function of causing the hand 13 to perform the next picking operation according to the determined path.

[0139] According to this configuration, similar to the robot system 100 according to the first aspect, the cycle time of the picking operation can be shortened. [Explanation of symbols]

[0140] 100 Robot Systems 1. Robot 2. Camera (photography device) 4. Control device 13 hands 43 Processing section 421 Computer Programs 431 Generation part 432 Judgment section 433 Correction Unit 434 Control Unit double work Wa,Wb Target work Fa,Fb surface R region

Claims

1. a robot having a hand that performs a picking operation including a gripping operation of gripping a target workpiece from a plurality of workpieces and a transport operation of transporting the target workpiece to a predetermined position after the gripping operation; An imaging device that images the plurality of works; a control device for controlling the robot, The control device a generation unit that generates a planned path of the hand for a next picking operation to be performed on the next target workpiece based on a first captured image captured by the imaging device before the start of the picking operation while the hand is performing the picking operation; a determination unit that, when the gripping operation in the picking operation is completed, performs an interference determination to determine whether or not the hand will interfere with the workpiece on the planned path based on a difference between a second captured image taken by the photographing device after the gripping operation is completed and the first captured image, and sets the planned path that is determined not to interfere as a determined path; a control unit that causes the hand to perform the next picking operation according to the determined path, The generation unit generates a plurality of the planned paths.

2. 2. The robot system according to claim 1, The imaging device images three-dimensional information of the plurality of workpieces, The determination unit performs the interference determination based on a difference between depth information of the first captured image and depth information of the second captured image.

3. 3. The robot system according to claim 2, The determination unit determines that an area where the surface height of the multiple workpieces in the second captured image is higher than the surface height of the multiple workpieces in the first captured image exists on the planned path, and determines that there will be no interference if the area does not exist on the planned path.

4. The robot system according to any one of claims 1 to 3, The determination unit is a robot system that performs the interference determination for each of the plurality of planned paths.

5. The robot system according to claim 4, When there are multiple planned routes that are determined not to interfere, the determination unit selects the planned route that has the shortest estimated time required for the picking operation as the determined route.

6. The robot system according to claim 4, The hand is formed to be switchable between a single state in which only one target workpiece is gripped in the gripping operation and a multi-state in which a plurality of target workpieces are gripped collectively in the gripping operation, When there are multiple planned paths that are determined not to interfere, the determination unit determines the planned path that is grasped by the hand in the multi-state as the determined path.

7. 2. The robot system according to claim 1, The judgment unit performs position judgment to determine whether the position of the target workpiece has changed based on the difference between the first captured image and the second captured image, and if it determines that the position of the target workpiece has not changed, performs the interference judgment.

8. The robot system according to claim 7, The control device further includes a correction unit that corrects the planned path when the determination unit determines that the position of the target workpiece has changed and the amount of change in the position of the target workpiece is equal to or less than a predetermined value. The determination unit performs the interference determination on the planned path corrected by the correction unit.

9. A picking method in which a robot hand performs a picking operation including a gripping operation in which a target workpiece is gripped from a plurality of workpieces, and a transport operation in which the target workpiece is transported to a predetermined position after the gripping operation, While the hand is performing the picking operation, generating a plurality of planned paths for the hand for a next picking operation to be performed on the next target workpiece based on first photographed images of the plurality of workpieces photographed before the start of the picking operation; When the gripping operation in the picking operation is completed, an interference determination is performed to determine whether or not the hand will interfere with the workpiece on at least one of the plurality of planned paths based on a difference between a second photographed image of the plurality of workpieces photographed after the gripping operation is completed and the first photographed image, and the planned path determined not to interfere is set as a determined path; and causing the hand to perform the next picking operation according to the determined path.

10. A computer program that causes a computer to realize a function of performing a picking operation including a gripping operation in which a robot hand grips a target workpiece from a plurality of workpieces, and a transport operation in which the target workpiece is transported to a predetermined position after the gripping operation, a function of generating, while the hand is performing the picking operation, a plurality of planned paths for the hand for the next picking operation to be performed on the next target workpiece based on first photographed images of the plurality of workpieces photographed before the start of the picking operation; When the gripping operation in the picking operation is completed, an interference determination is performed to determine whether or not the hand will interfere with the workpiece on at least one of the plurality of planned paths based on a difference between a second photographed image of the plurality of workpieces photographed after the gripping operation is completed and the first photographed image, and the planned path determined not to interfere is set as the determined path; and a function of causing the hand to perform the next picking operation in accordance with the determined path.

Images