Device and method for setting conveyance device coordinate system to robot coordinate system

Abstract

The present invention derives a technology for making it possible to more easily set a conveyance device coordinate system to a robot coordinate system (that is, calibration between a robot coordinate system and a conveyance device coordinate system).　A device 60 comprises: a first indicator 18; a camera 14 that acquires first image data in which the first indicator 18 is imaged and second image data in which the first indicator 18 conveyed after the imaging by a conveyance device 16 is imaged; a position data acquiring unit 52 that acquires first position data that indicates a three-dimensional position with respect to the camera in a first indicator coordinate system expressed by the first indictor 18 captured in the first image data, and second position data that indicates a three-dimensional position with respect to the camera in a first indicator coordinate system expressed by the first indictor captured in the second image data; a conveyance direction acquiring unit 54 that derives the conveyance direction; and a coordinate system setting unit 56 that sets the conveyance device coordinate system to the robot coordinate system on the basis of the conveyance direction.

Description

[Technical field]

[0001] The present disclosure relates to an apparatus and method for setting a transport device coordinate system to a robot coordinate system. [Background technology]

[0002] There is known an apparatus that sets the transport device coordinate system in the robot coordinate system by touching up the tip of the robot to the transport device (for example, Patent Document 1). There is also known an apparatus that calibrates the robot coordinate system with respect to the peripheral device by performing visual touch-up in which a camera captures images of multiple marks provided on the peripheral device of the robot (for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-174171 A [Patent Document 2] JP 2005-149299 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for a technique that enables easier setting of the transport device coordinate system in the robot coordinate system (that is, easier calibration of the robot coordinate system and the transport device coordinate system). [Means for solving the problem]

[0005] In one aspect of the present disclosure, an apparatus for setting a transport apparatus coordinate system that specifies the transport direction of a transport apparatus that transports a workpiece in a robot coordinate system that is set on a robot that performs work on a workpiece includes a first index that represents a first index coordinate system, the first index being installed on the transport apparatus so as to be transported by the transport apparatus, a camera that acquires first image data that captures the first index, and second image data that captures the first index transported by the transport apparatus after capturing the first image data.

[0006] The device also includes a position data acquisition unit that acquires first position data indicating a three-dimensional position relative to the camera of a first index coordinate system represented by a first index shown in the first image data and second position data indicating a three-dimensional position relative to the camera of the first index coordinate system represented by the first index shown in the second image data, a transport direction acquisition unit that determines a transport direction based on the first position data and the second position data, and a coordinate system setting unit that sets a transport device coordinate system in the robot coordinate system based on the transport direction determined by the transport direction acquisition unit.

[0007] In another aspect of the present disclosure, a method of setting a transport device coordinate system that specifies the transport direction of a transport device that transports a workpiece in a robot coordinate system that is set on a robot that performs work on a workpiece includes installing a first index that represents a first index coordinate system on a transport device so that it is transported by the transport device, acquiring first image data by capturing an image of the first index with a camera, and acquiring second image data by capturing an image of the first index transported by the transport device after capturing the first image data.

[0008] This method also acquires first position data indicating a three-dimensional position relative to the camera of a first index coordinate system represented by a first index shown in the first image data, and second position data indicating a three-dimensional position relative to the camera of the first index coordinate system represented by a first index shown in the second image data, determines a conveying direction based on the first position data and the second position data, and sets a conveying device coordinate system in the robot coordinate system based on the conveying direction determined by the conveying direction acquisition unit. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic perspective view of a robot system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram of the robot system shown in FIG. [Diagram 3] 2 is a flowchart showing an example of a method for setting a transport device coordinate system in a robot coordinate system in the robot system shown in FIG. [Figure 4] An example of image data captured in step S1 in FIG. 3 is shown. [Diagram 5] This shows a state in which the first indicator is transported in step S3 in FIG. [Figure 6] An example of image data captured in step S4 in FIG. 3 is shown. [Figure 7] In the robot system shown in FIG. 1, a transport device coordinate system is set in the robot coordinate system. [Figure 8] FIG. 11 is a schematic perspective view of a robot system according to another embodiment. [Figure 9] FIG. 9 is a block diagram of the robot system shown in FIG. 8. [Figure 10] 9 shows the rear side of the terminal device shown in FIG. 8. [Figure 11] 9 is a flowchart showing an example of a method for setting a transport device coordinate system in a robot coordinate system in the robot system shown in FIG. 8. [Figure 12] An example of image data captured in step S11 in FIG. 11 is shown. [Figure 13] This shows a state in which the first indicator is transported in step S13 in FIG. [Figure 14] An example of image data captured in step S14 in FIG. 11 is shown. [Figure 15] In the robot system shown in FIG. 8, the transport device coordinate system is set in the robot coordinate system. [Figure 16] FIG. 13 is a schematic perspective view of a robot system according to yet another embodiment. [Figure 17]FIG. 17 is a block diagram of the robot system shown in FIG. 16. [Figure 18] 17 is a flowchart showing an example of a method for setting a transport device coordinate system in a robot coordinate system in the robot system shown in FIG. 16. [Figure 19] An example of image data captured in step S22 in FIG. 18 is shown. [Figure 20] This shows a state in which the first indicator is transported in step S24 in FIG. [Figure 21] An example of image data captured in step S25 in FIG. 18 is shown. [Figure 22] An example of image data captured in step S27 in FIG. 18 is shown. [Figure 23] In the robot system shown in FIG. 16, the transport device coordinate system is set in the robot coordinate system. [Figure 24] 13 is a flowchart showing another example of a method for setting the transport device coordinate system in the robot coordinate system in the robot system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In various embodiments described below, similar elements are given the same reference numerals, and duplicated descriptions will be omitted. First, a robot system 10 according to one embodiment will be described with reference to Figs. 1 and 2. The robot system 10 includes a robot 12, a camera 14, a transport device 16, a first index 18, and a control device 20.

[0011] In this embodiment, the robot 12 is a vertical articulated robot that performs a predetermined task (e.g., work handling, welding, cutting, laser processing, etc.) on a workpiece (not shown). Specifically, the robot 12 has a robot base 22, a rotating body 24, a lower arm 26, an upper arm 28, and a wrist 30.

[0012] The robot base 22 is fixed to the floor of a work cell or on an automated guided vehicle (AGV). The rotating body 24 is provided on the robot base 22 so as to be rotatable about a vertical axis. The lower arm 26 is provided on the rotating body 24 so as to be rotatable about a horizontal axis, and the upper arm 28 is provided on the tip of the lower arm 26 so as to be rotatable about a horizontal axis.

[0013] The wrist 30 has a wrist base 30a provided at the tip of the upper arm 28 so as to be rotatable about two mutually perpendicular axes, and a wrist flange 30b provided on the wrist base 30a so as to be rotatable about a wrist axis A1. An end effector (robot hand, welding torch, cutting tool, laser processing head, etc.) that performs an operation on a workpiece is removably attached to the wrist flange 30b.

[0014] A plurality of servo motors 32 (FIG. 2) are provided on each of the robot base 22, the rotating body 24, the lower arm 26, the upper arm 28, and the wrist 30. These servo motors 32 rotate each movable component of the robot 12 (i.e., the rotating body 24, the lower arm 26, the upper arm 28, the wrist 30, and the wrist flange 30b) in response to a command from the control device 20, thereby moving an end effector (not shown) attached to the wrist flange 30b to an arbitrary position.

[0015] In this embodiment, the camera 14 is detachably attached to the wrist flange 30b instead of the end effector. The camera 14 is a two-dimensional camera having an imaging sensor (CMOS, CCD, etc.) and an optical lens (collimator lens, focus lens, etc.) that guides the subject image to the imaging sensor, and captures an image of the subject in response to a command from the control device 20. The camera 14 captures the subject image along the optical axis A2, and forms the subject image on the imaging sensor by the optical lens, thereby acquiring image data ID of the subject image. In this embodiment, the camera 14 is moved to an arbitrary position by the robot 12.

[0016] The transport device 16 is disposed around the robot 12 and transports the workpiece in a transport direction D1. For example, the transport device 16 is a belt conveyor or the like and has a transport surface 34 on which the workpiece is placed and a drive mechanism 36 that moves the transport surface 34 in the transport direction D1. The transport surface 34 may be defined by the outer surface of a timing belt.

[0017] Additionally, the drive mechanism 36 may include a servo motor (not shown) that drives the timing belt (i.e., the conveying surface 34) in response to a command from the control device 20. The conveying device 16 is disposed such that at least a portion of the conveying surface 34 is within the operating range of the robot 12 (i.e., the range within which the end effector can be positioned).

[0018] The control device 20 controls the robot 12, the camera 14, and the transport device 16. Specifically, as shown in Fig. 2, the control device 20 is a computer having a processor 40, a memory 42, and an I / O interface 44. The processor 40 has a CPU or a GPU, etc., is communicatively connected to the memory 42 and the I / O interface 44 via a bus 46, and performs calculation processing to realize a function of setting a transport device coordinate system, which will be described later, while communicating with these components.

[0019] The memory 42 has a RAM or a ROM, etc., and temporarily or permanently stores various data. The memory 42 can be composed of a computer-readable storage medium, such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. The I / O interface 44 has, for example, an Ethernet (registered trademark) port, a USB port, an optical fiber connector, or an HDMI (registered trademark) terminal, and communicates data with an external device by wire or wirelessly under a command from the processor 40. In this embodiment, the servo motors 32, the camera 14, and the transport device 16 (specifically, the drive mechanism 36) of the robot 12 are communicatively connected to the I / O interface 44.

[0020] The control device 20 is also provided with a display device 48 and an input device 50. The display device 48 and the input device 50 are communicatively connected to the I / O interface 44. The display device 48 has a liquid crystal display, an organic EL display, or the like, and visibly displays various data under instructions from the processor 40.

[0021] The input device 50 has a push button, a switch, a keyboard, a mouse, a touch panel, or the like, and receives data input from an operator. The display device 48 and the input device 50 may be integrated into the housing of the control device 20, or may be externally attached to the housing of the control device 20 as a separate computer (such as a PC).

[0022] 1, a robot coordinate system C1 is set for the robot 12. The robot coordinate system C1 is a coordinate system for automatically controlling the operation of each movable element of the robot 12. In this embodiment, the robot coordinate system C1 is fixed to the robot base 22 so that its origin is located at the center of the robot base 22 and its z axis is parallel to (specifically, coincides with) the rotation axis of the rotating body 24.

[0023] On the other hand, a camera coordinate system C2 is set for the camera 14. In this embodiment, the camera coordinate system C2 is set for the camera 14 so that its origin is located at the center of the image sensor of the camera 14 and its z axis is parallel to (specifically, coincides with) the optical axis A2. The camera 14 may be attached to the wrist flange 30b so that its optical axis A2 is parallel to (or coincides with) the wrist axis A1.

[0024] The camera coordinate system C2 defines the coordinates of each pixel of the image data ID (or the image sensor) captured by the camera 14. In this embodiment, the position of the camera 14 in the robot coordinate system C1 (in other words, the position of the wrist flange 30b) is known. Therefore, the positional relationship R between the robot coordinate system C1 and the camera coordinate system C2 of the camera 14 placed at an arbitrary position PSn in the robot coordinate system C1 is 1_2_nis known.

[0025] Therefore, the coordinates Q (X, Y, Z, W, P, R) of the camera coordinate system C2 in the robot coordinate system C1 are known, and the coordinates of the robot coordinate system C1 and the coordinates of the camera coordinate system C2 are transformed by a known transformation matrix M 1_2_n (simultaneous transformation matrix, Jacobian matrix, etc.). 1_2_n As data showing the transformation matrix M 1_2_n The parameters of the above, the coordinates of the camera coordinate system C2 in the robot coordinate system C1, etc. are stored in the memory 42.

[0026] Here, when the robot 12 and the transport device 16 cooperate to perform an operation on a workpiece, it is necessary to set the transport device coordinate system in the robot coordinate system C1. The transport device coordinate system is set with respect to the transport device 16, and defines the position of the transport surface 34 and the transport direction D1. The first index 18 is used to set the transport device coordinate system in the robot coordinate system C1 (in other words, to make the positional relationship between the robot coordinate system C1 and the transport device coordinate system known).

[0027] Specifically, the first indicator 18 is made of a flat paper material or plate material, and a pattern 18a is visibly displayed on the surface of the first indicator 18. The pattern 18a includes, for example, a figure consisting of a plurality of lines or curves, or a dot pattern. The first indicator 18 is configured to represent the first indicator coordinate system C3 by the pattern 18a, and is placed on the conveying surface 34 of the conveying device 16 so as to be conveyed by the conveying device 16. The first indicator 18 may be fixed to the conveying surface 34 using a fastener (bolt, etc.) or a jig so as to be immovable relative to the conveying surface 34. Alternatively, the pattern 18a of the first indicator 18 may be directly formed on the conveying surface 34 by printing, engraving, or the like.

[0028] Next, a method for setting the transport device coordinate system in the robot coordinate system C1 in the robot system 10 will be described with reference to Fig. 3. Before starting the flow of Fig. 3, the operator places the first index 18 on the transport surface 34 of the transport device 16 as shown in Fig. 1. At this time, the operator may place the first index 18 within the operating range of the robot 12. After placing the first index 18, the operator operates the input device 50 to give a coordinate system setting command to the processor 40. When the processor 40 receives the coordinate system setting command, it starts the flow shown in Fig. 3.

[0029] In step S1, the processor 40 captures an image of the first index 18 with the camera 14. Specifically, the processor 40 operates the robot 12 to move the camera 14 to an imaging position PS1 where the first index 18 can be accommodated within the field of view of the camera 14. Then, the processor 40 operates the camera 14 to capture an image of the first index 18.

[0030] An example of image data ID1 captured by the camera 14 in step S1 is shown in Fig. 4. Each pixel of the image data ID1 is expressed as a coordinate in the camera coordinate system C2, and the image data ID1 includes a first index 18. The camera 14 acquires image data ID1 (first image data) capturing an image of the first index 18 in this manner, and supplies the image data to the processor 40.

[0031] In step S2, the processor 40 acquires position data PD1 (first position data) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 captured in the image data ID1 relative to the camera 14. Here, the pattern 18a of the first index 18 captured in the image data ID1 is configured to represent the three-dimensional position of the first index coordinate system C3 in the camera coordinate system C2 set in the camera 14 at the time the image data ID1 is captured.

[0032] The processor 40 executes an image analysis program PG1 (or an image analysis application) for reading the first index coordinate system C3 from the pattern 18a, analyzes the pattern 18a of the first index 18 shown in the image data ID1, and identifies the first index coordinate system C3 represented by the first index 18 within the image data ID1. Then, the processor 40 acquires coordinates Q1 (X1, Y1, Z1, W1, P1, R1) in the camera coordinate system C2 of the identified first index coordinate system C3.

[0033] Of this coordinate Q1, the coordinate (X1, Y1, Z1) indicates the origin position of the first index coordinate system C3 in the camera coordinate system C2 of Figure 4, and the coordinate (W1, P1, R1) indicates the attitude (i.e., the directions of each axis) of the first index coordinate system C3 relative to the camera coordinate system C2 of Figure 4 (so-called yaw, pitch, roll).

[0034] In this way, the coordinate Q1 indicates the three-dimensional position of the first index coordinate system C3 in the camera coordinate system C2 in FIG. 4, in other words, the three-dimensional position of the first index coordinate system C3 relative to the camera 14 when the image data ID1 is captured. In other words, in this document, "position" may represent position and attitude. In this step S2, the processor 40 acquires the coordinate Q1 as the position data PD1. Therefore, the processor 40 functions as a position data acquisition unit 52 (FIG. 2) that acquires the position data PD1 (specifically, the coordinate Q1).

[0035] In step S3, the processor 40 transports the first index 18 by the transport device 16. Specifically, the processor 40 operates the drive mechanism 36 of the transport device 16 to transport the transport surface 34 a predetermined distance δ in the transport direction D1. As a result, as shown in Fig. 5, the first index 18 is transported by the distance δ in the transport direction D1 from the position at the time of capturing the image data ID1. Note that in Fig. 5, for ease of understanding, the position of the first index 18 at the time of capturing the image data ID1 is indicated by a dotted line B, and a first index coordinate system C3 represented by the first index 18 is also indicated.

[0036] In step S4, the processor 40 images the first index 18 with the camera 14. In this embodiment, the processor 40 images the first index 18 with the camera 14 in step S4, with the camera 14 positioned at the same imaging position PS1 as in step S1. That is, the imaging position PS1 is determined in the robot coordinate system C1 so that the first index 18 before and after the transfer in step S3 can be accommodated within the field of view of the camera 14.

[0037] An example of the image data ID2 captured by the camera 14 in step S4 is shown in Fig. 6. The image data ID2 captures the first index 18 after the transport in step S3. The camera 14 acquires the image data ID2 (second image data) capturing the image of the first index 18 after the transport, and supplies the image data to the processor 40.

[0038] In step S5, the processor 40 functions as the position data acquisition unit 52 to acquire position data PD2 (second position data) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 captured in the image data ID2 relative to the camera 14. Specifically, the processor 40 executes the image analysis program PG1 in the same manner as in step S2 described above to acquire coordinates Q2 (X2, Y2, Z2, W2, P2, R2) in the camera coordinate system C2 of the first index coordinate system C3 represented by the first index 18 captured in the image data ID2. The coordinates Q2 indicate the three-dimensional position of the first index coordinate system C3 relative to the camera 14 at the time of capturing the image data ID2. Thus, in step S5, the processor 40 acquires the coordinates Q2 as the position data PD2.

[0039] In step S6, the processor 40 obtains the conveying direction D1 based on the position data PD1 and PD2. Specifically, the processor 40 obtains the conveying direction D1 based on the known positional relationship R between the robot coordinate system C1 and the camera coordinate system C2 arranged at the image capturing position PS1. 1_2_1 (Specifically, the transformation matrix M 1_2_1), coordinate Q1 (X1, Y1, Z1, W1, P1, R1) in the camera coordinate system C2 acquired as position data PD1 in step S2 described above is converted to coordinate Q3 (X3, Y3, Z3, W3, P3, R3) in the robot coordinate system C1. This coordinate Q3 indicates the three-dimensional position (specifically, the three-dimensional position and orientation) in the robot coordinate system C1 of the first index coordinate system C3 represented by the first index C1 when the image data ID1 is captured (i.e., when step S1 is executed).

[0040] Similarly, the processor 40 calculates the positional relationship R 1_2_1 Using this, the coordinate Q2 (X2, Y2, Z2, W2, P2, R2) in the camera coordinate system C2 acquired as the position data PD2 in the above-mentioned step S5 is converted into a coordinate Q4 (X4, Y4, Z4, W4, P4, R4) in the robot coordinate system C1. This coordinate Q4 indicates the three-dimensional position in the robot coordinate system C1 of the first index coordinate system C3 represented by the first index C1 when the image data ID2 is captured (i.e., when step S4 is executed).

[0041] Then, the processor 40 calculates a vector VT from the coordinate (X3, Y3, Z3) of the coordinate Q3 to the coordinate (X4, Y4, Z4) of the coordinate Q4. 3_4 This vector VT 3_4 coincides with the axis A3 connecting the coordinates (X3, Y3, Z3) and the coordinates (X4, Y4, Z4), and indicates the position of the vector in the conveying direction D1 in the robot coordinate system C1.

[0042] This vector VT 3_4 (or axis A3) indicates the trajectory in the robot coordinate system C1 of the origin of the first index coordinate system C3 that has been moved in the conveying direction D1 in step S3, and can be regarded as being located on the conveying surface 34 (or at a position a known distance above the conveying surface 34). In other words, the vector VT 3_4 (or axis A3) indicates the conveying direction D1 in the robot coordinate system C1, and also serves as data indicating the position of the conveying surface 34 in the robot coordinate system C1.

[0043] In this way, the processor 40 determines the vector VT 3_4 (or axis A3). Therefore, in this embodiment, the processor 40 functions as a conveying direction acquisition unit 54 (FIG. 2) that determines the conveying direction D1 based on the position data PD1 and PD2.

[0044] In step S7, the processor 40 sets the transport device coordinate system C4 (FIG. 7) in the robot coordinate system C1 based on the transport direction D1 obtained in step S6. Specifically, the processor 40 sets the transport device coordinate system C4 (FIG. 7) in the robot coordinate system C1 based on the transport direction D1 obtained in step S6 (specifically, the vector VT 3_4 ) is defined as the positive y-axis direction of the transport device coordinate system C4.

[0045] The processor 40 also defines the z-axis direction of the transport device coordinate system C4 as a direction parallel to the yz plane of the robot coordinate system C1 and perpendicular to the y-axis of the transport device coordinate system C4 defined as above, and defines the x-axis direction of the transport device coordinate system C4 as a direction perpendicular to the y-axis and z-axis of the transport device coordinate system C4.

[0046] At this time, the processor 40 may randomly determine the z-axis positive direction and the x-axis positive direction of the transport apparatus coordinate system C4. Alternatively, the operator may operate the input device 50 to input setting information of the z-axis positive direction or the x-axis positive direction in advance, and the processor 40 may determine the z-axis positive direction or the x-axis positive direction of the transport apparatus coordinate system C4 according to the setting information.

[0047] This setting information may include information that defines the positive z-axis direction of the transport device coordinate system C4 as a direction closer to the positive z-axis direction of the robot coordinate system C1 (i.e., vertically upward) (i.e., a direction in which the inner product of the two becomes larger). Alternatively, the setting information may include information that defines the positive x-axis direction of the transport device coordinate system C4 as a direction opposite to that of the robot coordinate system C1.

[0048] The processor 40 also determines the origin of the transport device coordinate system C4 in the robot coordinate system C1. As an example, the processor 40 determines the above-mentioned coordinate Q3 (X1, Y1, Z1) or coordinate Q4 (X4, Y4, Z4) as the origin of the transport device coordinate system C4. As another example, when the processor 40 acquires the axis A3 in the above-mentioned step S6, the processor 40 may determine the origin of the transport device coordinate system C4 as a predetermined position on the axis A3 (for example, the upstream end or downstream end of the transport surface 34, or the center, upstream end or downstream end of the operating range of the robot 12).

[0049] The above-mentioned coordinate Q3 indicates the position and orientation (directions of each axis) of the first index coordinate system C3 in the robot coordinate system C1 before the transfer in step S3. In step S7, the processor 40 calculates the coordinates (W3, P3, R3) indicating the orientation among the coordinates Q3 by calculating the y-axis plus direction of the first index coordinate system C3 before the transfer in the transfer direction D1 (vector VT 3_4 ) to obtain the coordinates (W3c, P3c, R3c) of the new attitude. Then, the processor 40 may determine the coordinates (W3c, P3c, R3c) of the new attitude as the directions of the axes of the transport device coordinate system C4.

[0050] Similarly, the processor 40 may obtain coordinates (W4c, P4c, R4c) of a new posture by correcting coordinates (W4, P4, R4) indicating the posture among coordinates Q4 indicating the position and posture of the first index coordinate system C3 in the robot coordinate system C1 after the transfer in step S3. Then, the processor 40 may determine the coordinates (W4c, P4c, R4c) of the new posture as the directions of the axes of the transfer device coordinate system C4.

[0051] In this way, as shown in Fig. 7, the transport device coordinate system C4 is set in the robot coordinate system C1. The transport device coordinate system C4 is fixed with respect to the robot coordinate system C1, and its y-axis positive direction indicates the transport direction D1 in the robot coordinate system C1 with high accuracy. The xy plane (or the origin) of the transport device coordinate system C4 indicates the position of the transport surface 34 in the robot coordinate system C1. In this way, in this embodiment, the processor 40 functions as a coordinate system setting unit 56 (Fig. 2) that sets the transport device coordinate system C4 in the robot coordinate system C1 based on the transport direction D1.

[0052] As described above, the processor 40 functions as the position data acquisition unit 52, the transport direction acquisition unit 54, and the coordinate system setting unit 56, and sets the transport device coordinate system C4 in the robot coordinate system C1 based on the image data ID1 and ID2 of the first index 18 captured by the camera 14. Therefore, the camera 14, the first index 18, the position data acquisition unit 52, the transport direction acquisition unit 54, and the coordinate system setting unit 56 constitute a device 60 (FIG. 2) that sets the transport device coordinate system C4 in the robot coordinate system C1.

[0053] In this device 60, the first index 18 representing the first index coordinate system C3 is placed on the transport device 16 (specifically, on the transport surface 34) so ​​as to be transported by the transport device 16. In addition, the camera acquires first image data ID1 (FIG. 4) capturing an image of the first index 18, and second image data ID2 (FIG. 6) capturing an image of the first index 18 transported by the transport device 16 in step S3 after capturing the first image data ID1 (steps S1, S4).

[0054] The position data acquisition unit 52 also acquires first position data PD1 (specifically, coordinate Q1) indicating the three-dimensional position of the first index coordinate system C4 represented by the first index 18 in the first image data ID1 relative to the camera 14 (specifically, camera coordinate system C2) (step S2). Next, the position data acquisition unit 52 acquires second position data PD2 (specifically, coordinate Q2) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 in the second image data ID2 relative to the camera 14 (camera coordinate system C2) (step S5).

[0055] Then, the conveying direction acquisition unit 54 determines a conveying direction D1 (specifically, a vector VT 3_4 , or axis A3) is obtained (step S6). The coordinate system setting unit 56 sets the transfer device coordinate system C4 in the robot coordinate system C1 based on the transfer direction D1 obtained by the transfer direction acquisition unit 54 (step S7).

[0056] According to this configuration, the transport device coordinate system C4 can be set in the robot coordinate system C1 by capturing an image of the first index 18 transported by the transport device 16 with the camera 14, without touching up the hand (wrist flange 30b or end effector) of the robot 12 to the transport device 16 or performing visual touch-up by capturing an image of multiple marks provided on the transport device 16 with the camera 14. This makes it easier to set the transport device coordinate system C4 relative to the robot coordinate system C1 (i.e., calibrate it).

[0057] In the device 60, the first index 18 includes a pattern 18a that represents the three-dimensional position of the first index coordinate system C3 in the camera coordinate system C2 set in the camera 14 that captured the image data ID. With this configuration, the first index coordinate system C3 can be more effectively represented in the camera coordinate system C2 by the pattern 18a of the first index 18 that appears in the image data ID.

[0058] In the present embodiment, in step S4, the first index 18 is imaged with the camera 14 positioned at the same imaging position PS1 as in step S1. However, this is not limiting, and in step S4, the processor 40 may move the camera 14 from the imaging position PS1 to the imaging position PS2 and acquire image data ID2' of the first index 18 captured at the imaging position PS2.

[0059] In this case, the processor 40 functions as the position data acquisition unit 52 in step S5 and acquires coordinates Q2' (X2', Y2', Z2', W2', P2', R2') in the camera coordinate system C2 of the first index coordinate system C3 specified in the image data ID2'. Then, in step S7, the processor 40 acquires the known positional relationship R between the robot coordinate system C1 and the camera coordinate system C2 of the camera 14 arranged at the imaging position PS2. 1_2_2 (Specifically, the transformation matrix M 1_2_2 ) is used to transform the coordinate Q2' into the coordinate Q4 (X4, Y4, Z4, W4, P4, R4) in the robot coordinate system C1.

[0060] Next, a robot system 70 according to another embodiment will be described with reference to Fig. 8 and Fig. 9. The robot system 70 differs from the above-described robot system 10 in that it further includes a terminal device 72 and a second indicator 74. The terminal device 72 is, for example, a smartphone, a tablet terminal device, or a portable computer such as a notebook PC, and can be carried by the operator.

[0061] Specifically, as shown in Fig. 9, the terminal device 72 has a processor 80, a memory 82, an I / O interface 84, a display device 86, an input device 88, and a camera 14. The display device 86 and the input device 88 are integrally provided on the front surface of the terminal device 72 as shown in Fig. 8, while the camera 14 is integrally provided on the back surface of the terminal device 72 as shown in Fig. 10. A camera coordinate system C2 is set for the camera 14, as in the above-mentioned embodiment.

[0062] The processor 80 is communicatively connected to the memory 82, the I / O interface 84, the display device 86, the input device 88, and the camera 14 via a bus 90, and performs calculations to realize a function of setting a transport device coordinate system, which will be described later, while communicating with these components. The I / O interface 84 is communicatively connected to the I / O interface 44 of the control device 20 via wire or wirelessly.

[0063] The configurations of the processor 80, memory 82, I / O interface 84, display device 86, and input device 88 are similar to those of the processor 40, memory 42, I / O interface 44, display device 48, and input device 50 described above, respectively, so duplicated explanations will be omitted.

[0064] The second index 74 is used to set the transport device coordinate system C4 in the robot coordinate system C1 together with the first index 18. Specifically, as shown in Fig. 8, the second index 74 is made of a flat paper material, a plate material, or the like, similar to the above-mentioned first index 18, and has a pattern 74a visibly displayed on its surface.

[0065] The pattern 74a includes, for example, a figure consisting of a plurality of lines or curves, or a dot pattern. The second indicator 74 is fixed to the robot 12 (for example, the robot base 22) and is configured to represent a second indicator coordinate system C5 by the pattern 74a. Here, the second indicator 74 is configured to represent a positional relationship R between the second indicator coordinate system C5 represented by the pattern 74a and the robot coordinate system C1. 1_5 is known.

[0066] Therefore, the coordinates of the robot coordinate system C1 and the coordinates of the second index coordinate system C5 are transformed by a known transformation matrix M 1_5 (simultaneous transformation matrix, Jacobian matrix, etc.). In this embodiment, the control device 20 determines the positional relationship R 1_5 As data showing the transformation matrix M 1_5The parameters, the coordinates of the second index coordinate system C5 in the robot coordinate system C1, and the like are held and stored in the memory 42. In addition, in Fig. 8, the robot coordinate system C1 is shown by a dotted line for ease of understanding.

[0067] Next, a method for setting the transport device coordinate system C4 in the robot coordinate system C1 in the robot system 70 will be described with reference to Fig. 11. As in the above-described embodiment, the operator places the first index 18 on the transport surface 34 of the transport device 16 before starting the flow of Fig. 11.

[0068] In step S11, the processor 80 of the terminal device 72 captures images of the first index 18 and the second index 74 with the camera 14. Specifically, the operator places the terminal device 72 at an imaging position PS3 where the first index and the second index 74 can be placed within the field of view of the camera 14 of the terminal device 72. At this time, the processor 80 may display image data ID continuously captured by the camera 14 on the display device 86 of the terminal device 72 in real time (so-called live view display).

[0069] Then, the operator operates the input device 88 of the terminal device 72 placed at the imaging position PS3 to give an imaging command to the processor 80. In response to the imaging command, the processor 80 operates the camera 14 to capture images of the first index and the second index 74. An example of image data ID3 captured by the camera 14 in step S11 is shown in FIG.

[0070] The camera 14 acquires image data ID3 (first image data) capturing the first index 18 and the second index in this manner, and supplies the image data ID3 to the processor 80. When capturing the image data ID3, the operator may hold the terminal device 72 in his / her hand, or may fix the terminal device 72 to a fixed point using a fixture (a tripod, a stand, etc.).

[0071] In step S12, the processor 80 functions as the position data acquisition unit 52 (FIG. 9) and acquires position data PD3 (first position data) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 captured in the image data ID3 relative to the camera 14. Specifically, the processor 80 executes the above-mentioned image analysis program PG1 to analyze the pattern 18a of the first index 18 captured in the image data ID3 and identify the first index coordinate system C3 represented by the first index 18 within the image data ID3. The image analysis program PG1 may be stored in advance in the memory 82.

[0072] Then, the processor 80 acquires a coordinate Q5 (X5, Y5, Z5, W5, P5, R5) in the camera coordinate system C2 of the identified first index coordinate system C3. This coordinate Q5 indicates the three-dimensional position of the first index coordinate system C3 relative to the camera 14 when capturing the image data ID3. In this step S12, the processor 80 acquires the coordinate Q5 as position data PD3.

[0073] The processor 80 also functions as the position data acquisition unit 52, and acquires position data PD4 (third position data) indicating the three-dimensional position of the second index coordinate system C5 represented by the second index 74 in the image data ID3 relative to the camera 14. Specifically, the processor 80 executes the image analysis program PG1 to analyze the pattern 74a of the second index 74 in the image data ID3, and identifies the second index coordinate system C5 represented by the second index 74 within the image data ID3.

[0074] Then, the processor 80 acquires coordinates Q6 (X6, Y6, Z6, W6, P6, R6) in the camera coordinate system C2 of the identified second index coordinate system C5. This coordinate Q6 indicates the three-dimensional position of the second index coordinate system C5 relative to the camera 14 at the time of capturing the image data ID3. In this step S12, the processor 80 acquires the coordinate Q6 as position data PD4. Thus, in this step S12, the processor 80 acquires position data PD4 of the second index coordinate system C5 together with position data PD3 of the first index coordinate system C3.

[0075] When step S12 is completed, the processor 80 transmits a completion signal SG1 to the control device 20. The processor 40 of the control device 20 executes step S13 in response to the completion signal SG1. Specifically, in step S13, the processor 40 operates the conveying device 16 to convey the first index 18 by a distance δ in the conveying direction D1, similar to the above-described step S3.

[0076] When the processor 40 receives the completion signal SG1, it may generate a notification signal NS1 to notify the transport device 16 to transport the first index 18, and output the notification signal NS1 to the display device 48 (or a speaker provided in the control device 20). Then, when the operator recognizes the notification signal NS1, the operator may execute step S13 and manually operate the transport device 16 to transport the first index 18. The state in which this step S13 is completed is shown in FIG. 13. For ease of understanding, in FIG. 13, the position of the first index 18 at the time of capturing the image data ID3 is indicated by a dotted line B, and the first index coordinate system C3 represented by the first index 18 is also indicated.

[0077] After completing step S13, the processor 40 transmits a completion signal SG2 to the terminal device 72. Upon receiving the completion signal SG2, the processor 80 of the terminal device 72 executes step S14. At this time, the processor 80 may generate a notification signal NS2 to notify the camera 14 to re-image the first index 18 and the second index 74, and output the notification signal NS2 to the display device 86 (or a speaker provided in the terminal device 72).

[0078] In step S14, the processor 80 of the terminal device 72 captures images of the first index 18 and the second index 74 with the camera 14. Specifically, when the operator recognizes the notification signal NS2 output to the display device 86 (or the speaker of the terminal device 72), the operator places the terminal device 72 at an imaging position PS4 where the first index and the second index 74 can be placed within the field of view of the camera 14. This imaging position PS4 may be different from or the same as the imaging position PS3 in the above-mentioned step S11. In addition, the processor 80 may execute the above-mentioned live view display at this time.

[0079] Then, the operator operates the input device 88 of the terminal device 72 to operate the camera 14 to capture images of the first index 18 and the second index 74. An example of image data ID4 captured by the camera 14 in step S14 is shown in FIG. 14. This image data ID4 captures the second index 74 together with the first index 18 after transport in step S13. The camera 14 acquires image data ID4 (second image data) capturing the images of the transported first index 18 and second index 74, and supplies the image data to the processor 80.

[0080] In addition, if the terminal device 72 is fixed at a fixed point at the same imaging position PS3 = PS4 using a fixing device in steps S11 and S14, the processor 80 may automatically execute step S14 when it receives the above-mentioned completion signal SG2, and automatically capture image data ID4 using the camera 14.

[0081] In step S15, the processor 80 functions as the position data acquisition unit 52 (Figure 9) and acquires position data PD5 (second position data) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 appearing in the image data ID4 relative to the camera 14, similar to step S12 described above.

[0082] Specifically, the processor 80 executes the image analysis program PG1 to acquire coordinates Q7 (X7, Y7, Z7, W7, P7, R7) in the camera coordinate system C2 of the first index coordinate system C3 represented by the first index 18 captured in the image data ID4. The coordinates Q7 indicate the three-dimensional position of the first index coordinate system C3 relative to the camera 14 at the time the image data ID4 was captured. In this way, the processor 80 acquires the coordinates Q7 as position data PD5.

[0083] The processor 80 also functions as the position data acquisition unit 52, and acquires position data PD6 (fourth position data) indicating the three-dimensional position of the second index coordinate system C5 represented by the second index 74 captured in the image data ID4 relative to the camera 14. Specifically, the processor 80 executes the image analysis program PG1 to acquire coordinates Q8 (X8, Y8, Z8, W8, P8, R8) in the camera coordinate system C2 of the second index coordinate system C5 represented by the second index 74 captured in the image data ID4.

[0084] This coordinate Q8 indicates the three-dimensional position of the second index coordinate system C5 relative to the camera 14 when the image data ID4 was captured. The processor 80 acquires the coordinate Q8 as the position data PD6. Thus, in this step S15, the processor 80 acquires the position data PD6 of the second index coordinate system C5 together with the position data PD5 of the first index coordinate system C3.

[0085] In step S16, the processor 80 functions as the conveying direction acquisition unit 54 (FIG. 9) and obtains the conveying direction D1 based on the position data PD3, PD4, PD5 and PD6. Specifically, the processor 80 obtains the positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 represented in the camera coordinate system C2 of the image data ID3 (FIG. 12) based on the position data PD3 of the first index coordinate system C3 (i.e., the coordinate Q5 of the camera coordinate system C2) and the position data PD4 of the second index coordinate system C5 (i.e., the coordinate Q6 of the camera coordinate system C2) obtained in the above-mentioned step S12. 3_5_1 (First positional relationship) is calculated.

[0086] Here, the positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 in the three-dimensional space defined by the camera coordinate system C2 of the image data ID3 is 3_5_1 is known from the position data PD3 and PD4. The processor 80 calculates the positional relationship R 3_5_1 As the data, coordinates Q9 (X9, Y9, Z9, W9, P9, R9) of the first index coordinate system C3 in the second index coordinate system C5 shown in FIG. 12 are acquired.

[0087] Similarly, the processor 80 calculates the positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 represented in the camera coordinate system C2 of the image data ID4 (FIG. 14) based on the position data PD5 of the first index coordinate system C3 (i.e., the coordinate Q7 in the camera coordinate system C2) and the position data PD6 of the second index coordinate system C5 (i.e., the coordinate Q8 in the camera coordinate system C2) acquired in the above-mentioned step S15. 3_5_2 Specifically, the processor 80 calculates the positional relationship R from the position data PD5 and PD6. 3_5_2 As the data, coordinates Q10 (X10, Y10, Z10, W10, P10, R10) of the first index coordinate system C3 in the second index coordinate system C5 shown in FIG. 14 are acquired.

[0088] Then, the processor 80 calculates a vector VT from the coordinate (X9, Y9, Z9) of the coordinate Q9 to the coordinate (X10, Y10, Z10) of the coordinate Q10. 9_10 This vector VT 9_10 coincides with the axis A3 connecting the coordinates (X9, Y9, Z9) and the coordinates (X10, Y10, Z10), and indicates the position of the vector (in other words, the axis A3) of the conveying direction D1 in the second index coordinate system C5.

[0089] This vector VT 9_10(or axis A3) is expressed as a coordinate (or a function) of the second index coordinate system C5, indicates the trajectory in the second index coordinate system C5 of the origin of the first index coordinate system C3 that has been moved in the conveying direction D1 in step S13, and can be considered to be located on the conveying surface 34. Thus, the processor 80 calculates the vector VT 9_10 (or axis A3).

[0090] After completing step S16, the processor 80 calculates the data of the conveying direction D1 acquired in step S15 (specifically, the vector VT 9_10 Or the coordinates or function data of the axis A3) is supplied to the control device 20. When the processor 40 of the control device 20 receives the data of the conveying direction D1, it executes step S17.

[0091] In step S17, the processor 40 functions as the coordinate system setting unit 56 (FIG. 9) and sets the transport device coordinate system C4 (FIG. 15) in the robot coordinate system C1 based on the transport direction D1 obtained in step S16. Specifically, the processor 40 sets the known positional relationship R 1_5 (Specifically, the transformation matrix M 1_5 ) to obtain the conveying direction D1 (for example, vector VT 9_10 ) into the robot coordinate system C1.

[0092] Then, in the same manner as in step S7 described above, the processor 40 calculates the conveying direction D1 (vector VT 9_10 ) as the positive y-axis direction of the transport apparatus coordinate system C4. Then, the processor 40 determines the positive x-axis direction and the positive z-axis direction of the transport apparatus coordinate system C4 in the same manner as in step S7 described above.

[0093] The processor 40 also determines the origin of the transport device coordinate system C4 in the robot coordinate system C1. As an example, the processor 40 converts the above-mentioned coordinate Q9 (X9, Y9, Z9) into coordinate C9' (X9', Y9', Z9') in the robot coordinate system C1, and determines the coordinate C9' as the origin of the transport device coordinate system C4. As another example, the processor 40 may convert the above-mentioned coordinate Q10 (X10, Y10, Z10) into coordinate C10' (X10', Y10', Z10') in the robot coordinate system C1, and determine the coordinate C10' as the origin of the transport device coordinate system C4.

[0094] As yet another example, the processor 40 may convert the axis A3 obtained as the coordinate (or function) of the second index coordinate system C5 in the above-mentioned step S16 into the robot coordinate system C1, and set the origin of the transport device coordinate system C4 as a predetermined position on the axis A3 in the robot coordinate system C1. In this way, the transport device coordinate system C4 is set in the robot coordinate system C1, as shown in FIG.

[0095] As described above, in this embodiment, the processor 80 of the terminal device 72 functions as the position data acquisition unit 52 and the transport direction acquisition unit 54, while the processor 40 of the control device 20 functions as the coordinate system setting unit 56. The processors 40 and 80 cooperate with each other to set the transport device coordinate system C4 in the robot coordinate system C1 based on the image data ID3 and ID4 captured by the camera 14 of the first index 18 and the second index 74. Therefore, the camera 14, the first index 18, the position data acquisition unit 52, the transport direction acquisition unit 54, the coordinate system setting unit 56, and the second index 74 constitute an apparatus 100 (FIG. 9) that sets the transport device coordinate system C4 in the robot coordinate system C1.

[0096] In this device 100, a second index 74 representing a second index coordinate system C5 is placed at a known position in the robot coordinate system C1, and the camera 14 acquires first image data ID3 capturing an image of the first index 18 and the second index 74, and second image data ID4 capturing an image of the first index 18 and the second index 74 transported in step S13 (steps S11, S14).

[0097] The position data acquisition unit 52 (processor 80) also acquires third position data PD4 (specifically, coordinate Q6) indicating the three-dimensional position of the second index coordinate system C5 represented by the second index 74 in the first image data ID3 relative to the camera 14 (camera coordinate system C2) (step S12). Next, the position data acquisition unit 52 acquires fourth position data PD6 (specifically, coordinate Q8) indicating the three-dimensional position of the second index coordinate system C5 represented by the second index 74 in the second image data ID4 relative to the camera 14.

[0098] Then, the conveying direction acquisition unit (processor 80) determines the conveying direction D1 in the robot coordinate system C1 based on the first position data PD3 (coordinate Q5), the second position data PD5 (coordinate Q7), the third position data PD4 (coordinate Q6), and the fourth position data PD6 (coordinate Q8) (step S16).

[0099] According to this configuration, the transport device coordinate system C4 can be set in the robot coordinate system C1 simply by capturing images of the first index 18 and the second index 74 with the camera 14. Therefore, the transport device coordinate system C4 can be easily set (i.e., calibrated) relative to the robot coordinate system C1.

[0100] In the device 100, the conveying direction acquisition unit 54 calculates a first positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 in the first image data ID3 (FIG. 12) based on the first positional data PD3 and the third positional data PD4. 3_5_1 (Specifically, the coordinate Q9) is obtained (step S16).

[0101] Further, the conveying direction acquisition unit 54 calculates a second positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 in the second image data ID4 (FIG. 14) based on the second position data PD5 and the fourth position data PD6. 3_5_2 (Specifically, the coordinate Q10) is obtained (step S16).

[0102] Then, the conveying direction acquisition unit 54 calculates the first positional relationship R3_5_1 and the second positional relationship R 3_5_2 (Step S16) According to this configuration, the conveying direction D1 can be quickly and accurately determined using an existing algorithm (specifically, the image analysis program PG1, etc.).

[0103] In the device 100, the second index 74 includes a pattern 74a that represents the three-dimensional position of the second index coordinate system C5 in the camera coordinate system C2 set in the camera 14 that captured the image data ID3 and ID4. With this configuration, the second index coordinate system C5 can be more effectively represented in the camera coordinate system C2 by the pattern 74a of the second index 74 that appears in the image data ID3 and ID4.

[0104] 16 and 17, a robot system 110 according to still another embodiment will be described. The robot system 110 differs from the above-described robot system 70 in that the robot system 110 further includes a sensor 112. The sensor 112 detects a displacement Δ of the camera 14.

[0105] More specifically, the sensor 112 has at least one of a gyro sensor and an acceleration sensor, and is provided in the terminal device 72 so as to be fixed with respect to the camera 14. In this embodiment, the sensor 112 is communicatively connected to the I / O interface 84 of the terminal device 72, detects a displacement Δ (more specifically, a displacement amount and a displacement direction of the position and attitude) of the camera 14 (in other words, the terminal device 72), and supplies the same to the processor 80.

[0106] Next, a method for setting the transport device coordinate system C4 in the robot coordinate system C1 in the robot system 110 will be described with reference to Fig. 18. As in the above-described embodiment, the operator places the first index 18 on the transport surface 34 of the transport device 16 before starting the flow of Fig. 18.

[0107] In step S21, the processor 80 of the terminal device 72 starts detection of the displacement Δ of the camera 14 by the sensor 112. Specifically, the processor 80 operates the sensor 112 to continuously (for example, periodically) detect the displacement Δ of the position of the camera 14 (or the terminal device 72). The processor 80 sequentially acquires detection data of the displacement Δ from the sensor 112 and stores it in the memory 42.

[0108] In step S22, the processor 80 captures an image of the first index 18 with the camera 14. Specifically, the operator places the terminal device 72 at an imaging position PS5 where the first index 18 can be placed within the field of view of the camera 14 of the terminal device 72. At this time, the processor 80 may execute the above-mentioned live view display.

[0109] Then, the operator operates the input device 88 of the terminal device 72 to operate the camera 14 and capture an image of the first index 18. An example of image data ID5 captured by the camera 14 in step S22 is shown in Fig. 19. The camera 14 acquires the image data ID5 (first image data) thus capturing the image of the first index 18, and supplies it to the processor 80.

[0110] In step S23, the processor 80 functions as the position data acquisition unit 52 (FIG. 17) and acquires position data PD7 (first position data) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 captured in the image data ID5 relative to the camera 14. Specifically, the processor 80 executes the image analysis program PG1 as in the above-described embodiment to acquire the coordinates Q11 (X11, Y11, Z11, W11, P11, R11) of the first index coordinate system C3 in the camera coordinate system C2 in FIG. 19 as the position data PD7. When step S23 is completed, the processor 80 transmits a completion signal SG1 to the control device 20.

[0111] The processor 40 of the control device 20 executes step S24 in response to the completion signal SG1. Specifically, in step S24, the processor 40 operates the conveying device 16 to convey the first index 18 by a distance δ in the conveying direction D1, similar to the above-mentioned step S13. The state after completion of this step S24 is shown in FIG. 20. Note that in FIG. 20, for ease of understanding, the position of the first index 18 at the time of capturing the image data ID5 is indicated by a dotted line B, and the first index coordinate system C3 represented by the first index 18 is also shown.

[0112] After completing step S24, the processor 40 transmits a completion signal SG2 to the terminal device 72. Upon receiving the completion signal SG2, the processor 80 of the terminal device 72 executes step S25. At this time, the processor 80 may output a notification signal NS3 to notify the user that the first index 18 should be imaged again by the camera 14, as in the above-described embodiment.

[0113] In step S25, the processor 80 of the terminal device 72 captures an image of the first index 18 with the camera 14. Specifically, when the operator recognizes the notification signal NS3 output to the display device 86 (or the speaker of the terminal device 72), the operator places the terminal device 72 at an imaging position PS6 where the first index 18 can be placed within the field of view of the camera 14. This imaging position PS6 may be different from or the same as the imaging position PS5 in the above-mentioned step S22. In addition, the processor 80 may execute the above-mentioned live view display at this time.

[0114] Then, the operator operates the input device 88 of the terminal device 72 to operate the camera 14 and capture an image of the first index 18. An example of image data ID6 captured by the camera 14 in step S25 is shown in Fig. 21. The camera 14 acquires image data ID6 (second image data) capturing the image of the transported first index 18, and supplies it to the processor 80.

[0115] In step S26, the processor 80 functions as the position data acquisition unit 52 (FIG. 17) and acquires position data PD8 (second position data) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 captured in the image data ID6 relative to the camera 14. Specifically, the processor 80 executes the image analysis program PG1 in the same manner as in step S23 described above, thereby acquiring coordinates Q12 (X12, Y12, Z12, W12, P12, R12) of the first index coordinate system C3 in the camera coordinate system C2 in FIG. 21 as the position data PD8.

[0116] In step S27, the processor 80 captures an image of the second index 74 with the camera 14. Specifically, the operator places the terminal device 72 at an imaging position PS7 where the second index 74 can be placed within the field of view of the camera 14 of the terminal device 72. This imaging position PS7 is different from the imaging positions PS5 and PS6 described above. At this time, the processor 80 may execute the above-mentioned live view display.

[0117] Then, the operator operates the input device 88 of the terminal device 72 to operate the camera 14 and capture an image of the second index 74. An example of image data ID7 captured by the camera 14 in step S27 is shown in FIG. 22. The camera 14 acquires image data ID7 (third image data) capturing the image of the second index 74, and supplies the image data to the processor 80. Thus, in this embodiment, the operator captures images of the first index 18 and the second index 74 separately in steps S22 and S25, and in step S27.

[0118] In step S28, the processor 80 functions as the position data acquisition unit 52 (FIG. 17) and acquires position data PD9 (third position data) indicating the three-dimensional position of the second index coordinate system C5 represented by the second index 74 captured in the image data ID7 relative to the camera 14. Specifically, the processor 80 executes the image analysis program PG1 in the same manner as in the above-described embodiment to acquire, as the position data PD9, coordinates Q13 (X13, Y13, Z13, W13, P13, R13) of the second index coordinate system C5 in the camera coordinate system C2 in FIG.

[0119] In step S29, the processor 80 obtains the conveying direction D1 based on the position data PD7, PD8, and PD9 and the displacement Δ detected by the sensor 112. Here, after the start of step S21, the sensor 112 continues to detect the displacement Δ of the camera 14 (terminal device 72) while capturing the image data ID5 in step S22, the image data ID6 in step S25, and the image data ID7 in step S27.

[0120] Therefore, the processor 80 calculates, from the detection data of the sensor 112, a displacement Δ 5_6 and the displacement Δ from the imaging position PS6 to the imaging position PS7 at which the image data ID7 was captured in step S27. 6_7 It is possible to obtain:

[0121] The processor 80 calculates the displacement Δ 5_6 and Δ 6_7 From the position data PD7 (coordinate Q11) acquired in step S23 and the position data PD9 (coordinate Q13) acquired in step S27, the positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 before the transfer in step S24 is calculated. 3_5_3 Specifically, the processor 80 determines the positional relationship R 3_5_3 As the data, coordinates Q14 (X14, Y14, Z14, W14, P14, R14) in the second index coordinate system C5 of the first index coordinate system C3 before the transfer are acquired.

[0122] The processor 80 also calculates the displacement Δ 6_7 From the position data PD8 (coordinate Q12) acquired in step S26 and the position data PD9 (coordinate Q13) acquired in step S27, the positional relationship R between the first index coordinate system C3 and the second index coordinate system C5 after the transfer in step S24 is calculated. 3_5_4 Specifically, the processor 80 determines the positional relationship R 3_5_4As the data, the coordinates Q15 (X15, Y15, Z15, W15, P15, R15) in the second index coordinate system C5 of the first index coordinate system C3 after the transfer are acquired.

[0123] Then, the processor 80 calculates a vector VT from the coordinate (X14, Y14, Z14) of the coordinate Q14 to the coordinate (X15, Y15, Z15) of the coordinate Q15. 14_15 This vector VT 14_15 coincides with the axis A3 connecting the coordinates (X14, Y14, Z14) and the coordinates (X15, Y15, Z15), and indicates the position of the vector (in other words, the axis A3) of the conveying direction D1 in the second index coordinate system C5.

[0124] This vector VT 14_15 (or axis A3) is expressed as a coordinate (or a function) in the second index coordinate system C5, indicates the trajectory in the second index coordinate system C5 of the origin of the first index coordinate system C3 that has been moved in the conveying direction D1 in step S24, and can be considered to be located on the conveying surface 34. Thus, the processor 80 calculates the vector VT 14_15 (or axis A3).

[0125] After completion of step S29, the processor 80, as in the above-described embodiment, converts the data of the conveying direction D1 acquired in step S29 (specifically, the vector VT 14_15 Or the coordinates or function data of the axis A3) is supplied to the control device 20. When the processor 40 of the control device 20 receives the data of the conveying direction D1, it executes step S30.

[0126] In step S30, the processor 40 functions as the coordinate system setting unit 56 (FIG. 17) and sets the transport device coordinate system C4 (FIG. 23) in the robot coordinate system C1 based on the transport direction D1 obtained in step S29. Specifically, the processor 40 sets the known positional relationship R between the second index coordinate system C5 and the robot coordinate system C1, similar to the above-mentioned step S17. 1_5Using the vector V 14_15 ) into the robot coordinate system C1.

[0127] Then, the processor 40 converts the conveying direction D1 (vector VT 14_15 ) is defined as the y-axis positive direction of the transport device coordinate system C4, and also as the x-axis positive direction and z-axis positive direction of the transport device coordinate system C4.

[0128] The processor 40 also determines the origin of the transport device coordinate system C4 in the robot coordinate system C1. As an example, the processor 40 converts the above-mentioned coordinate Q14 (X14, Y14, Z14) into coordinate C14' (X14', Y14', Z14') in the robot coordinate system C1, and determines the coordinate C14' as the origin of the transport device coordinate system C4. As another example, the processor 40 may convert the above-mentioned coordinate Q15 (X15, Y15, Z15) into coordinate C15' (X15', Y15', Z15') in the robot coordinate system C1, and determine the coordinate C15' as the origin of the transport device coordinate system C4.

[0129] As yet another example, the processor 40 may convert the axis A3 obtained as the coordinate (or function) of the second index coordinate system C5 in the above-mentioned step S29 into the robot coordinate system C1, and set the origin of the transport device coordinate system C4 as a predetermined position on the axis A3 in the robot coordinate system C1. In this way, the transport device coordinate system C4 is set in the robot coordinate system C1, as shown in FIG.

[0130] As described above, in this embodiment, the processors 40 and 80 cooperate with each other to function as the position data acquisition unit 52, the transport direction acquisition unit 54, and the coordinate system setting unit 56, and set the transport device coordinate system C4 in the robot coordinate system C1 based on the image data ID5, ID6, and ID7 of the first index 18 and the second index 74 captured by the camera 14, and the displacement Δ detected by the sensor 112. Therefore, the camera 14, the first index 18, the position data acquisition unit 52, the transport direction acquisition unit 54, the coordinate system setting unit 56, the second index 74, and the sensor 112 constitute a device 120 (FIG. 17) that sets the transport device coordinate system C4 in the robot coordinate system C1.

[0131] In this device 120, the camera 14 acquires the third image data ID3 capturing an image of the second index 74 (step S27) along with the first image data ID5 capturing an image of the first index 18 (step S22) and the second image data ID6 (step S25). In addition, the position data acquisition unit 52 (processor 80) acquires third position data PD9 indicating the three-dimensional position of the second index coordinate system C5 represented by the second index 74 captured in the third image data ID3 relative to the camera 14 (camera coordinate system C2) (step S28).

[0132] The sensor 112 also detects the displacement Δ of the camera 14 while capturing the first image data ID5, the second image data ID6, and the third image data ID7. 5_6 and Δ 6_7 Then, the conveying direction acquisition unit 54 detects the first position data PD7, the second position data PD8, the third position data PD9, and the displacement Δ 5_6 and Δ 6_7 Based on this, the conveying direction D1 is obtained (step S29).

[0133] Here, due to space constraints in the work cell, there are cases where the first index and the second index 74 cannot be captured simultaneously by the camera 14, as shown in Figures 12 and 14. According to this embodiment, as shown in Figures 19, 21 and 22, even if the first index and the second index 74 are captured separately, the transport direction D1 can be obtained. Therefore, even if there is not enough space in the work cell, the transport device coordinate system C4 can be effectively set in the robot coordinate system C1.

[0134] 3, 11 or 18, the processor 40 or 80 captures an image of the first index 18 by the camera 14 before and after the first index 18 is transported only once by the transport device 16, and determines the transport direction D1. However, the present invention is not limited to this, and the processor 40 or 80 may transport the first index 18 multiple times by the transport device 16, capture an image of the first index 18 by the camera 14 before and after the transport, and determine the transport direction D1 multiple times.

[0135] Such an embodiment will be described below with reference to Fig. 24. The flow in Fig. 24 shows the flow of another method for setting the transport device coordinate system C4 to the robot coordinate system C1 in the above-mentioned robot system 10. In step S31, the processor 40 sets a number "n" indicating the number of times to image the first index 18 in step S32 described later to "1" (n=1).

[0136] In step S32, the processor 40 images the first index 18 with the camera 14. Specifically, the processor 40 operates the robot 12 to place the camera 14 at an imaging position PSn where the first index 18 can be placed within the field of view of the camera 14. The processor 40 then operates the camera 14 to image the first index 18, thereby acquiring image data IDn as shown in FIG. 4 or 6, for example. If the number "n" is set to n=1 at the start of step S32, the processor 40 images the image data ID1 with the camera 14 placed at the imaging position PS1.

[0137] In step S33, similar to step S2 or S5 described above, the processor 40 functions as a position data acquisition unit 52 (Figure 2) to acquire position data PDn (specifically, coordinate Qn in the camera coordinate system C2) indicating the three-dimensional position of the first index coordinate system C3 represented by the first index 18 appearing in the image data IDn relative to the camera 14.

[0138] If the number "n" is set to n=1 at the start of step S33, the processor 40 acquires position data PD1 (specifically, coordinate Q1 in the camera coordinate system C2) indicating the three-dimensional position of the first index coordinate system C3 represented in the image data ID1 relative to the camera 14. In step S34, the processor 40 increments the number "n" by "1" (n=n+1).

[0139] In step S35, the processor 40 determines whether the number "n" is n>2. If the processor 40 determines YES, the process proceeds to step S36, whereas if the processor 40 determines NO, the process proceeds to step S38. If the processor 40 proceeds to step S35 after executing step S32 for the first time (i.e., in a state where n=1 is set), the number "n" at this point in time has been incremented to n=2 in step S34. Therefore, in this case (i.e., when the first index 18 is imaged only once), the processor 40 determines NO in step S35 and proceeds to step S38.

[0140] In step S36, the processor 40 functions as the conveying direction acquisition unit 54 (FIG. 2) in the same manner as in step S6 described above, and determines the conveying direction D1 based on the position data PDn (specifically, the coordinate Qn) acquired in the most recently executed step S33 and the position data PDn-1 (specifically, the coordinate Qn-1) acquired in the step S33 executed before the acquisition of the position data PDn. _n Request.

[0141] 24, the processor 40 repeatedly executes a loop of steps S32 to S38 until it determines YES in step S37 described later. Therefore, the processor 40 repeatedly conveys the first index 18 by the conveying device 16 in step S38 described later, and obtains position data PDn-1 (first position data) from image data IDn-1 (first image data) captured in step S32 before the conveying in step S33 executed before the conveying. Meanwhile, the processor 40 obtains position data PDn (second position data) from image data IDn (second image data) captured in step S32 after the conveying in step S33 executed after the conveying.

[0142] In step S36, the processor 40 calculates the conveying direction D1 based on the position data PDn (second position data) acquired in the most recent step S33 and the position data PDn-1 (first position data) acquired in the step S33 executed before the acquisition of the position data PDn, in the same manner as in step S6 described above. _n (For example, the above vector VT 3_4 Similar vectors to VT _n ) is required.

[0143] In step S37, the processor 40 determines whether the number "n" is n=n MAX Determine whether this number n MAX is the maximum number of times (for example, n MAX = 10) and is preset by an operator. If the processor 40 determines that the answer is YES, the process proceeds to step S39, whereas if the processor 40 determines that the answer is NO, the process proceeds to step S38.

[0144] In step S38, the processor 40 transports the first index 18 by the transport device 16, similar to step S3 described above. Note that while the processor 40 transports the first index 18 in the transport direction D1 in the n-th step S38, the processor 40 may transport the first index 18 by the transport device 16 in the direction D2 opposite to the transport direction D1 in the (n+1)-th step S38 executed thereafter.

[0145] In step S36, which is executed after the conveyance in the opposite direction D2, the processor 40 calculates a vector VT _n In this case, the processor 40 calculates the vector VT _n ' and the vector VT _n In the conveying direction D1 _n In this manner, the processor 40 repeatedly executes the loop of steps S32 to S38 until it determines YES in step S37, and each time it executes step S36, it calculates the conveying direction D1 _n Request.

[0146] In step S39, the processor 40 functions as the conveying direction acquisition unit 54 to obtain the final conveying direction D1. _F Specifically, the processor 40 calculates the conveying direction D1 _n Multiple vectors VT obtained as _1 , V.T. _2 , VT _n The composite vector VT _F (or the average vector) is calculated, and the vector VT _F The final conveying direction D1 _F Obtain as.

[0147] In step S40, the processor 40 functions as the coordinate system setting unit 56 (FIG. 2) and, similarly to step S6 described above, calculates the conveying direction D1 obtained in step S39. _FSpecifically, the processor 40 sets the transport device coordinate system C4 in the robot coordinate system C1 based on the vector VT _F is defined as the positive y-axis direction of the transport device coordinate system C4.

[0148] In this manner, in the present embodiment, the processor 40 calculates the conveying direction D1 each time the conveying device 16 conveys the first index 18 multiple times. _n The multiple conveying directions D1 _n Based on this, the final conveying direction D1 _F According to this configuration, the conveying direction D1 of the conveying device 16 can be obtained with higher accuracy.

[0149] In addition, the concept of the flow in FIG. 24 (that is, each time the conveying device 16 conveys the first indicator 18 multiple times, the conveying direction D1 _n Calculate multiple conveying directions D1 _n Based on the final conveying direction D1 _F It should be understood that the above-mentioned process (obtaining the conveying direction D1) can be applied to the flow of FIG. 11 or FIG. 18. For example, in the case of the flow of FIG. 11, the processors 40 and 80 cooperate with each other to repeatedly execute the loop of steps S11 to S16, and obtain the conveying direction D1 obtained each time step S16 is executed. _n Based on this, the final conveying direction D1 _F You may ask for:

[0150] 18, the processor 80 of the terminal device 72 executes steps S27 and S28 after step S21. After that, the processors 40 and 80 cooperate with each other to repeatedly execute a loop of steps S22 to S26 and S29, and calculate the conveying directions D1 obtained each time step S29 is executed. _n Based on this, the final conveying direction D1 _F You may ask for:

[0151] In the above embodiment, the second index 74 includes the pattern 74a that represents the second index coordinate system C5 in the camera coordinate system C2. However, the present invention is not limited to this, and the second index 74 may be configured with the shape 74b of the robot 12. As an example, the shape 74b may be configured with a plurality of faces, edges, concaves, and convexities formed on a component of the robot 12 (e.g., the robot base 22). As another example, the shape 74b may be configured with the shape of the entire robot 12 (i.e., the robot base 22, the rotating body 24, the lower arm 26, the upper arm 28, the wrist 30, and the wrist flange 30b) that is stationary at a predetermined position.

[0152] In this embodiment, when the camera 14 captures a shape 74b as the second index 74 in the above-mentioned steps S11, S14 or S27, the shape 74b captured in the captured image data ID represents the three-dimensional position (i.e., coordinate Q) of the second index coordinate system C5 in the camera coordinate system C2 of the camera 14.

[0153] Then, in the above-mentioned steps S12, S15 or 28, the processor 40 or 80 functions as a position data acquisition unit 52, executes an image analysis program PG2 (image analysis application) for reading the second index coordinate system C4 from the shape 74b, and identifies the second index coordinate system C4 represented by the shape 74b within the image data ID.

[0154] Then, the processor 40 or 80 acquires the coordinate Q in the camera coordinate system C2 of the specified second index coordinate system C4 as the position data PD. Note that the first index 18 is not limited to the pattern 18a and may include, for example, a shape 18b (surface, edge, concave portion, convex portion) formed on the conveying surface 34.

[0155] In the above-mentioned robot systems 70 (FIG. 9) and 110 (FIG. 17), the processor 80 of the terminal device 72 functions as the position data acquisition unit 52 and the conveying direction acquisition unit 54. However, in the robot system 70 or 110, the processor 40 of the control device 20 may function as the position data acquisition unit 52, the conveying direction acquisition unit 54, and the coordinate system setting unit 56. In this case, the processor 80 of the terminal device 72 supplies the image data ID3, ID4, ID5, ID6, and ID7 captured by the camera 14 to the control device 20.

[0156] Alternatively, the terminal device 72 may be omitted, and the camera 14 may be, for example, a digital camera. Then, the operator may transmit image data ID3, ID4, ID5, ID6, and ID7 captured by the camera 14 to the I / O interface 44 of the control device 20 via wired or wireless communication.

[0157] In addition, in the above-mentioned step S6, the processor 40 calculates the conveying direction D1 by a vector VT from the coordinate Q3 (X3, Y3, Z3) to the coordinate Q4 (X4, Y4, Z4) which indicates the locus of the origin of the first index coordinate system C3. 3_4 However, the present invention is not limited to this, and the processor 40 can also calculate the conveying direction D1 based on the amount of change between the coordinates (W3, P3, Q3) indicating the attitude among the coordinates Q3 and the coordinates (W4, P4, R4) indicating the attitude among the coordinates Q4.

[0158] Similarly, in the above-mentioned steps S16, S29 or S36, the processor 80 may determine the conveying direction D1 based on the amount of change in the coordinates Q9 (W9, P9, R9) and coordinate Q10 (W10, P10, R10) indicating the posture, the amount of change in the coordinates Q14 (W14, P14, R14) and coordinate Q15 (W15, P15, R15) indicating the posture, or the amount of change in the coordinates Qn-1 and coordinate Qn indicating the posture.

[0159] 3, 11, 18, or 24 in accordance with a computer program PG3 prestored in the memory 42 or 82. The functions of the position data acquiring unit 52, the conveying direction acquiring unit 54, and the coordinate system setting unit 56 executed by the processor 40 or 80 may be functional modules realized by the computer program PG3.

[0160] Furthermore, the control device 20 may have a first control device 20A that controls the robot 12 and a second control device 20B that controls the transport device 16. Furthermore, the robot 12 is not limited to a vertical articulated robot, but may be any type of robot, such as a horizontal articulated robot or a parallel link robot. Furthermore, the transport device 16 is not limited to a belt conveyor, but may be any device capable of transporting a workpiece. Although the present disclosure has been described above through the embodiments, the above-mentioned embodiments do not limit the invention according to the claims.

[0161] As described above, the present disclosure describes the following aspects. (Aspect 1) An apparatus for setting a transport device coordinate system that specifies a transport direction of a transport device that transports a workpiece in a robot coordinate system that is set in a robot that performs work on a workpiece, the apparatus comprising: a camera that acquires a first index, the first index representing a first index coordinate system, the first index being installed on the transport device so as to be transported by the transport device; first image data that captures an image of the first index; and second image data that captures the first index transported by the transport device after the first image data is captured; a position data acquisition unit that acquires first position data indicating a three-dimensional position relative to the camera of the first index coordinate system represented by the first index captured in the first image data; and second position data indicating the three-dimensional position relative to the camera of the first index coordinate system represented by the first index captured in the second image data; a transport direction acquisition unit that determines the transport direction based on the first position data and the second position data; and a coordinate system setting unit that sets the transport device coordinate system in the robot coordinate system based on the transport direction determined by the transport direction acquisition unit. (Aspect 2) The device described in aspect 1, further comprising a second index installed at a known position in the robot coordinate system and representing a second index coordinate system, wherein the camera acquires first image data capturing the first index and the second index and second image data capturing the transported first index and second index, the position data acquisition unit further acquires third position data indicating a three-dimensional position relative to the camera of the second index coordinate system represented by the second index captured in the first image data and fourth position data indicating the three-dimensional position relative to the camera of the second index coordinate system represented by the second index captured in the second image data, and the transport direction acquisition unit determines the transport direction further based on the third position data and the fourth position data. (Aspect 3) The device described in aspect 2, wherein the transport direction acquisition unit determines a first positional relationship between the first index coordinate system and the second index coordinate system in the first image data based on the first position data and the third position data, determines a second positional relationship between the first index coordinate system and the second index coordinate system in the second image data based on the second position data and the fourth position data, and determines the transport direction based on the first positional relationship and the second positional relationship. (Aspect 4) An apparatus as described in aspect 1, comprising: a second index installed at a known position in the robot coordinate system and representing a second index coordinate system; and a sensor for detecting displacement of the camera, wherein the camera further acquires third image data capturing an image of the second index, the position data acquisition unit further acquires third position data indicating a three-dimensional position relative to the camera of the second index coordinate system represented by the second index captured in the third image data, the sensor detects displacement of the camera while capturing the first image data, the second image data, and the third image data, and the conveying direction acquisition unit determines the conveying direction further based on the third position data and the displacement detected by the sensor. (Aspect 5) An apparatus described in any of aspects 2 to 4, wherein the second index includes a pattern or a robot shape representing a three-dimensional position of the second index coordinate system in a camera coordinate system set in the camera that captured the image data. (Aspect 6) In the device according to any one of aspects 1 to 5, the first index includes a pattern representing a three-dimensional position of the first index coordinate system in a camera coordinate system set in the camera that captured the image data. (Aspect 7) A method for setting a transport device coordinate system that specifies the transport direction of a transport device that transports a workpiece in a robot coordinate system that is set on a robot that performs work on a workpiece, the method comprising the steps of: installing a first index representing a first index coordinate system on the transport device so that it is transported by the transport device; acquiring first image data by capturing an image of the first index with a camera; acquiring second image data by capturing an image of the first index transported by the transport device after capturing the first image data with a camera; acquiring first position data that indicates a three-dimensional position relative to the camera of the first index coordinate system represented by the first index captured in the first image data; and acquiring second position data that indicates the three-dimensional position relative to the camera of the first index coordinate system represented by the first index captured in the second image data; determining the transport direction based on the first position data and the second position data; and setting the transport device coordinate system in the robot coordinate system based on the determined transport direction. [Explanation of symbols]

[0162] 10,70,110 Robot Systems 12. Robot 14 Camera 16 Transport equipment 18 First Index 20 Control device 40,80 Processor 52 Location data acquisition unit 54 Conveying direction acquisition unit 56 Coordinate system setting section 60,100,120 equipment 74 Second Index 112 Sensors

Claims

1. A device for setting a conveying device coordinate system that defines a conveying direction of a conveying device that conveys a workpiece in a robot coordinate system that is set in a robot that performs an operation on the workpiece, comprising: a first index representing a first index coordinate system, the first index being mounted on the transport device so as to be transported by the transport device; A camera that acquires first image data obtained by capturing an image of the first index and second image data obtained by capturing an image of the first index transported by the transport device after capturing the first image data; a position data acquisition unit that acquires first position data indicating a three-dimensional position of the first index in the first index coordinate system relative to the camera represented by the first index captured in the first image data, and second position data indicating a three-dimensional position of the first index in the first index coordinate system relative to the camera represented by the first index captured in the second image data; a conveying direction acquisition unit that determines the conveying direction based on the first position data and the second position data; a coordinate system setting unit that sets the transport device coordinate system in the robot coordinate system based on the transport direction obtained by the transport direction acquisition unit.

2. a second index located at a known position in the robot coordinate system and representing a second index coordinate system; The camera acquires the first image data obtained by capturing an image of the first index and the second index, and the second image data obtained by capturing an image of the transported first index and the second index, The position data acquisition unit further acquires third position data indicating a three-dimensional position of the second index coordinate system represented by the second index captured in the first image data relative to the camera, and fourth position data indicating a three-dimensional position of the second index coordinate system represented by the second index captured in the second image data relative to the camera, The device according to claim 1 , wherein the conveying direction acquisition unit determines the conveying direction further based on the third position data and the fourth position data.

3. The conveying direction acquisition unit determining a first positional relationship between the first index coordinate system and the second index coordinate system in the first image data based on the first position data and the third position data; determining a second positional relationship between the first index coordinate system and the second index coordinate system in the second image data based on the second position data and the fourth position data; The apparatus according to claim 2 , further comprising: a conveying direction determining unit configured to determine the conveying direction based on the first positional relationship and the second positional relationship.

4. a second index located at a known position in the robot coordinate system and representing a second index coordinate system; a sensor for detecting a displacement of the camera; The camera further acquires third image data capturing the second index; the position data acquisition unit further acquires third position data indicating a three-dimensional position of the second index in the second index coordinate system relative to the camera, the third position data indicating a three-dimensional position of the second index in the third image data; the sensor detects the displacement of the camera while capturing the first image data, the second image data, and the third image data; The device according to claim 1 , wherein the conveying direction acquisition unit determines the conveying direction further based on the third position data and the displacement detected by the sensor.

5. The device according to claim 2 or 4, wherein the second index includes a pattern or a shape of the robot that represents a three-dimensional position of the second index coordinate system in a camera coordinate system set in the camera that captured the image data.

6. The device according to claim 1 , wherein the first index includes a pattern that represents a three-dimensional position of the first index coordinate system in a camera coordinate system set in the camera that captured the image data.

7. A method for setting a transport device coordinate system that defines a transport direction of a transport device that transports a workpiece in a robot coordinate system that is set in a robot that performs an operation on the workpiece, comprising the steps of: placing a first index representing a first index coordinate system on the transport device so as to be transported by the transport device; Acquiring first image data by capturing an image of the first index with a camera, and acquiring second image data by capturing an image of the first index transported by the transport device with the camera after capturing the first image data; obtain first position data indicating a three-dimensional position of the first index in the first index coordinate system relative to the camera represented by the first index captured in the first image data, and second position data indicating a three-dimensional position of the first index in the first index coordinate system relative to the camera represented by the first index captured in the second image data; determining the conveying direction based on the first position data and the second position data; and setting the transport device coordinate system in the robot coordinate system based on the determined transport direction.