Simulation device using 3D position information obtained from the output of a visual sensor

Abstract

This simulation device comprises a simulation implementation unit that simulates a movement of a robot device, and estimates a movement path of a robot. The simulation device comprises a position information generation unit that generates three-dimensional position information for a surface of a workpiece on the basis of the output of a vision sensor, which has imaged an actual workpiece. A display unit displays the three-dimensional position information for the surface of the workpiece, superimposed on an image of a robot device model, an image of a workpiece model, and the movement path of the robot.

Description

[Technical field]

[0001] The present invention relates to a simulation device that uses three-dimensional position information obtained from the output of a visual sensor. [Background technology]

[0002] In a robot device equipped with a robot and a work tool, the position and posture of the work tool are changed by changing the position and posture of the robot. The robot device can perform various tasks that change the position and posture of the work tool. The position and posture of the robot are changed based on an operation program generated before performing the task. The operation program includes teaching points that determine the position and posture of the robot.

[0003] When performing a new task with a robot device, it is necessary to create an operation program. The position and posture of the robot at the teaching point can be taught by driving an actual robot. Also, a simulation device that simulates the operation of the robot device is known (for example, JP 2015-93345 A). With the simulation device, the operation of the robot device can be confirmed by an image. An operator can teach the teaching point by simulating the operation of the robot device with the simulation device.

[0004] In addition, in conventional technology, it is known to attach a three-dimensional visual sensor to a robot and process position information obtained by the three-dimensional visual sensor to detect the position of an object relative to the robot and perform various tasks (for example, JP 2006-35384 A).

[0005] Furthermore, there is known a control method in which a two-dimensional or three-dimensional visual sensor is attached to a robot, and the robot device is actually driven to correct the teaching point (for example, Japanese Patent Application Laid-Open No. 7-84631 and Japanese Patent Application Laid-Open No. 2004-255547). In such a device, for example, a marker is placed on the actual workpiece. The position of the marker on the actual workpiece is obtained based on the output of the visual sensor. The position of the teaching point in the operation program can be corrected based on the actual position of the marker. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2015-93345 A [Patent Document 2] JP 2006-35384 A [Patent Document 3] Japanese Patent Application Publication No. 7-84631 [Patent Document 4] JP 2004-255547 A Summary of the Invention [Problem to be solved by the invention]

[0007] In a simulation device, it is preferable to determine an accurate position and posture of a robot with respect to a workpiece to be operated. However, the installation position of the robot device when placed on an installation surface and the installation positions of peripheral devices such as a stand that supports the workpiece rarely match the desired design values. In particular, a deviation in the installation positions of the peripheral devices corresponds to a deviation in the position where the workpiece is placed. The actual installation position of the robot and the position where the workpiece is placed often deviate from the positions in the simulation. For this reason, it is preferable to also shift the positions of the teaching points and the posture of the robot at the teaching points.

[0008] In the conventional technology, an operating program created by a simulation device is input to a control device of an actual robot device, and an operator actually drives the robot to correct the positions of teaching points in the operating program and the posture of the robot at the teaching points.

[0009] On the other hand, it is difficult to adjust the positions of the teaching points and the posture of the robot at the teaching points in the simulation device, taking into account the actual installation position of the robot and the position where the workpiece is placed. In other words, it is difficult to predict how the installation position of the robot or the installation position of the peripheral device that fixes the workpiece will be shifted, and there is a problem that it is difficult to correct the operation program in the simulation device.

[0010] In a control where a marker is placed on the surface of an actual workpiece and the position of the marker is detected by a visual sensor, there is a problem that the marker position set on the workpiece in the simulation must match the marker position placed on the actual workpiece. Also, if the detection of the marker fails, the position of the workpiece cannot be determined. For this reason, it is necessary to create a marker that makes it easy to detect the workpiece using the output of the visual sensor. [Means for solving the problem]

[0011] A simulation device according to a first aspect of the present disclosure performs a simulation of the operation of a robot device having a robot. The simulation device includes a model generation unit that generates a robot device model and a workpiece model in the simulation based on three-dimensional shape data of the robot device and three-dimensional shape data of a workpiece. The simulation device includes a display unit that displays an image of the robot device model and an image of the workpiece model. The simulation device includes an operation information setting unit that generates an operation program including teaching points. The simulation device includes a simulation execution unit that performs a simulation of the operation of the robot device based on the teaching points to estimate a motion path of the robot. The simulation device includes a position information generation unit that generates three-dimensional position information including information on the positions of three-dimensional points set on the surface of the workpiece based on an output of a visual sensor that captures an image of an actual workpiece. The display unit displays a plurality of three-dimensional points corresponding to the three-dimensional position information, superimposed on an image of the robot device model, an image of the workpiece model, and a motion path of the robot. The workpiece has a shape larger than the imaging range when the visual sensor is placed at the imaging position. The position information generating unit generates multiple pieces of three-dimensional position information acquired by imaging multiple times with the imaging position changed, and generates three-dimensional position information for the entire surface of the workpiece facing the visual sensor by connecting the multiple pieces of three-dimensional position information. The display unit displays three-dimensional points for the entire surface of the workpiece facing the visual sensor.

[0012] A simulation device according to a second aspect of the present disclosure performs a simulation of the operation of a robot device having a robot. The simulation device includes a model generation unit that generates a robot device model and a workpiece model in the simulation based on three-dimensional shape data of the robot device and three-dimensional shape data of a workpiece. The simulation device includes a simulation execution unit that performs a simulation of the operation of the robot device. The simulation device includes an operation information setting unit that sets operation information defined in an operation program based on the simulation of the operation of the robot device. The simulation device includes a position information generation unit that generates three-dimensional position information of the surface of the workpiece based on the output of a visual sensor that captures an image of an actual workpiece, and a position detection unit that detects the position of the actual workpiece by matching reference data of the workpiece with the three-dimensional position information of the surface of the workpiece. The operation information setting unit corrects the operation information included in the operation program so as to correspond to the position of the actual workpiece based on the position of the workpiece model in the simulation and the position of the actual workpiece detected by the position detection unit. A workpiece coordinate system in which an origin is set for the workpiece is set in advance. The workpiece coordinate system is a coordinate system that is fixed to the workpiece and whose position and posture change together with the workpiece. The operation information defined in the operation program includes the position and orientation of the work coordinate system, and the positions of teaching points defined in the work coordinate system and the orientation of the robot at the teaching points. The operation information setting unit corrects the position and orientation of the work coordinate system set in the work model by the simulation so that they correspond to the position and orientation of the work coordinate system of the actual work calculated from the three-dimensional position information. The operation information setting unit maintains the position of the teaching point defined in the work coordinate system and the attitude of the robot at the teaching point without change, and corrects the position and attitude of the work coordinate system defined in the operation program. Effect of the Invention

[0013] According to one aspect of the present disclosure, it is possible to provide a simulation device that can easily determine and correct deviations in the position and posture of a robot. [Brief description of the drawings]

[0014] [Figure 1]FIG. 2 is a perspective view of a first robot device in the embodiment. [Diagram 2] FIG. 1 is a block diagram of a first robot system including a first robot device and a first simulation device according to an embodiment. [Diagram 3] This is an image taken when a simulation was performed using an operation program. [Figure 4] 1 is a perspective view of a robot device and a workpiece when an image of the workpiece is captured by a visual sensor. FIG. [Diagram 5] The image shows a robot device model and a cloud of 3D points detected by a visual sensor. [Figure 6] The images show a robot device model, a workpiece model, and a cloud of 3D points detected by a visual sensor. [Figure 7] FIG. 4 is a block diagram of a second robot system including a first robot device and a second simulation device in the embodiment. [Figure 8] 1 shows an image of a robot device model for explaining a workpiece coordinate system set in a workpiece model, an image of the workpiece model, and an image displaying a cloud of three-dimensional points. [Figure 9] The images show a motion path after correcting the motion information in the motion program, an image of a robot device model, an image of a work model, and a cloud of three-dimensional points. [Figure 10] FIG. 2 is a perspective view of a second robot device in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] A simulation device according to an embodiment will be described with reference to Figs. 1 to 10. The simulation device according to the embodiment simulates the operation of a robot device equipped with a robot. The simulation device according to the embodiment processes the output of a three-dimensional visual sensor. The three-dimensional visual sensor acquires information about the surface of a workpiece as a work target. The simulation device according to the embodiment displays position information about the surface of the workpiece together with a simulation model, and corrects teaching points determined in the simulation based on the position information about the surface of the workpiece.

[0016] FIG. 1 is a perspective view of a first robot device in this embodiment. FIG. 2 is a block diagram of a first robot system in this embodiment. With reference to FIGS. 1 and 2, the first robot system includes a first robot device 3 and a first simulation device 4. The first robot device 3 performs spot welding. The first robot device 3 includes a welding gun 5 as a work tool, and a robot 1 that changes the position and attitude of the welding gun 5. The robot device 3 includes a control device 2 that controls the robot 1 and the welding gun 5. The first robot system includes a visual sensor 30 that acquires information about the surface of a workpiece 65 as an object of work.

[0017] The first workpiece 65 in this embodiment is a plate-like member. The first workpiece 65 has protrusions 65a formed corresponding to the welding points 68a to 68c. The first workpiece 65 has a surface 65b. The second workpiece 66 is a plate-like member having the same planar shape as the first workpiece 65. The robot device 3 performs spot welding at the three welding points 68a to 68c. The first workpiece 65 and the second workpiece 66 are fixed to each other. The workpieces 65 and 66 are supported by a stand 69.

[0018] The robot 1 of this embodiment is a multi-joint robot including a plurality of joints. The robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a swivel base 13. The swivel base 13 is supported by a base 14. The robot 1 includes a wrist 15 connected to an end of the upper arm 11. The wrist 15 includes a flange 16 to which a welding gun 5 is fixed. The robot 1 of this embodiment has six drive shafts, but is not limited to this form. The robot may be any robot capable of moving a work tool.

[0019] Robot 1 in this embodiment includes a robot driving device 21 that drives components such as upper arm 11. Robot driving device 21 includes a plurality of driving motors for driving upper arm 11, lower arm 12, rotating base 13, and wrist 15. Welding gun 5 includes a tool driving device 22 that drives welding gun 5. Tool driving device 22 in this embodiment includes a motor that drives a movable electrode relative to a fixed electrode of welding gun 5.

[0020] The control device 2 includes an arithmetic processing device 24 (computer) including a CPU (Central Processing Unit) as a processor. The arithmetic processing device 24 has a RAM (Random Access Memory) and a ROM (Read Only Memory) that are connected to the CPU via a bus. The robot device 3 drives the robot 1 and the welding gun 5 based on a pre-created operation program. The robot device 3 of this embodiment automatically performs spot welding on the workpieces 65 and 66 at the welding points 68a to 68c.

[0021] The arithmetic processing device 24 of the control device 2 includes a storage unit 42 that stores information related to the control of the robot device 3. The storage unit 42 can be configured with a non-transitory storage medium capable of storing information. For example, the storage unit 42 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. An operating program for the robot device 3 to perform spot welding work is stored in the storage unit 42.

[0022] The arithmetic processing device 24 includes an operation control unit 43 that outputs operation commands. The operation control unit 43 outputs operation commands to the robot driving unit 44 for driving the robot 1 based on an operation program. The robot driving unit 44 includes an electric circuit for driving a drive motor. The robot driving unit 44 supplies electricity to the robot driving device 21 based on the operation command. The operation control unit 43 also outputs operation commands to the work tool driving unit 45 for driving the tool driving device 22. The work tool driving unit 45 includes an electric circuit for passing electricity through the electrode and an electric circuit for driving the motor of the movable electrode. The work tool driving unit 45 supplies electricity to the tool driving device 22 based on the operation command.

[0023] The operation control unit 43 corresponds to a processor that operates according to an operation program of the robot device. The processor reads the operation program and performs the control defined in the operation program, thereby functioning as the operation control unit 43.

[0024] The robot 1 includes a state detector for detecting the position and posture of the robot 1. The state detector in this embodiment includes a rotation angle detector 23 attached to the drive motor of each drive shaft of the robot drive device 21. The rotation angle detector 23 is configured by, for example, an encoder. The position and posture of the robot 1 are detected by the output of the rotation angle detector 23.

[0025] The control device 2 includes a teaching operation panel 49 serving as an operation panel with which an operator manually operates the robot device 3. The teaching operation panel 49 includes an input unit 49a for inputting information relating to the robot 1, the welding gun 5, and the visual sensor 30. The input unit 49a is composed of operating members such as a keyboard and a dial. The teaching operation panel 49 includes a display unit 49b for displaying information relating to the control of the robot device 3. The display unit 49b is composed of a display panel such as a liquid crystal display panel.

[0026] In the robot device 3 of this embodiment, a robot coordinate system 71 is set that does not move when the position and posture of the robot 1 change. In the example shown in FIG. 1, the origin of the robot coordinate system 71 is located on the base 14 of the robot 1. The robot coordinate system 71 is also called a world coordinate system or a reference coordinate system. The position of the origin of the robot coordinate system 71 is fixed, and the orientation of the coordinate axes is fixed. Even if the position and posture of the robot 1 change, the position and posture of the robot coordinate system 71 do not change.

[0027] A tool coordinate system 72 having an origin set at an arbitrary position of the work tool is set in the robot device 3. The position and posture of the tool coordinate system 72 changes together with the welding gun 5. In this embodiment, the origin of the tool coordinate system 72 is set at the tool tip point (tip point of the fixed electrode). The position of the robot 1 corresponds to the position of the tool tip point (position of the origin of the tool coordinate system 72). The posture of the robot 1 corresponds to the posture of the tool coordinate system 72 with respect to the robot coordinate system 71.

[0028] The visual sensor 30 of this embodiment is a three-dimensional camera capable of acquiring three-dimensional position information of the surface of an object. The visual sensor 30 of this embodiment is a stereo camera including a first camera 31 and a second camera 32. Each of the cameras 31 and 32 is a two-dimensional camera capable of capturing a two-dimensional image. The relative positions of the two cameras 31 and 32 are determined in advance. The visual sensor 30 of this embodiment includes a projector 33 that projects a pattern light such as a striped pattern toward the surface 65b of the workpiece 65.

[0029] The first simulation device 4 simulates the operation of the robot device 3. The simulation device 4 arranges a three-dimensional model of the robot 1, a three-dimensional model of the welding gun 5, and three-dimensional models of the workpieces 65 and 66 in the same virtual space, and performs a simulation of the operation of the robot device 3.

[0030] The simulation device 4 in this embodiment is configured with an arithmetic processing device including a CPU as a processor. The simulation device 4 includes a storage unit 53 that stores any information related to the simulation of the robot device 3. The storage unit 53 can be configured with a non-transitory storage medium capable of storing information. For example, the storage unit 53 can be configured with a storage medium such as a volatile memory, a non-volatile memory, a magnetic storage medium, or an optical storage medium. A program for implementing the simulation of the robot device is stored in the storage unit 53.

[0031] Three-dimensional shape data 50 of the robot 1, the welding gun 5, and the workpieces 65, 66 are input to the simulation device 4. The three-dimensional shape data 50 includes three-dimensional shape data of the robot, the work tool, the peripheral equipment, and the workpieces for simulating the robot device. As the three-dimensional shape data 50, for example, data output from a CAD (Computer Aided Design) device can be used. The three-dimensional shape data 50 is stored in a storage unit 53.

[0032] The simulation device 4 includes an input unit 51 for inputting information related to the simulation of the robot device 3. The input unit 51 is composed of operation members such as a keyboard, a mouse, and a dial. The simulation device 4 includes a display unit 52 for displaying information related to the simulation of the robot device 3. As will be described later, the display unit 52 displays an image of a model of the robot device 3 and images of models of the workpieces 65, 66. The display unit 52 is composed of a display panel such as a liquid crystal display panel. Note that when the simulation device includes a touch panel type display panel, the display panel functions as both the input unit and the display unit.

[0033] The simulation device 4 includes a processing unit 54 that performs calculation processing for simulating the robot device 3. The processing unit 54 also has a function of processing information acquired by the visual sensor 30. The processing unit 54 includes a model generation unit 55 that generates a robot device model that is a model of the robot device and a workpiece model that is a model of the workpiece, based on three-dimensional shape data 50 including three-dimensional shape data of the robot device 3 and three-dimensional shape data of the workpieces 65, 66.

[0034] The processing unit 54 includes a simulation execution unit 56 that performs a simulation of the operation of the robot device 3. The simulation execution unit 56 has a function of moving a robot device model on a screen in response to an operation of the input unit 51 by an operator. Alternatively, the simulation execution unit 56 performs a simulation of the operation of the robot device 3 based on teaching points generated in advance. For example, the simulation execution unit 56 performs a simulation of the operation of the robot device 3 based on an operation program 41 including teaching points.

[0035] The processing unit 54 includes a position information generating unit 59 that generates three-dimensional position information of the surface of the workpiece 65 based on the output of the visual sensor 30 that captures images of the actual workpieces 65, 66. The processing unit 54 includes an operation information setting unit 57 that sets operation information defined in the operation program 41 of the robot device 3 based on a simulation of the operation of the robot device 3. The operation information setting unit 57 generates an operation program including teaching points. The operation information of the robot device 3 includes the positions of the teaching points and the posture of the robot 1 at the teaching points.

[0036] The processing unit 54 includes a display control unit 58 that controls the image to be displayed on the display unit 52. The processing unit 54 includes a distance calculation unit 60 that calculates the distance between any two points by designating two points in the image displayed on the display unit 52. The distance calculation unit 60 can calculate the distance between a point in the image of the workpiece model and one three-dimensional point included in a point cloud of three-dimensional points arranged on the surface of the workpiece.

[0037] Processing unit 54 corresponds to a processor that operates according to a simulation program. The simulation program is created in advance and stored in storage unit 53. The processor reads the simulation program and performs control defined in the program, thereby functioning as processing unit 54. Moreover, model generation unit 55, simulation execution unit 56, motion information setting unit 57, display control unit 58, position information generation unit 59, and distance calculation unit 60 included in processing unit 54 correspond to a processor that operates according to a simulation program. The processor performs control defined in the program, thereby functioning as each unit.

[0038] FIG. 3 shows an example of an image displayed on the display unit of the simulation device. Image 81 shows the state after a simulation of the robot device 3 has been performed. The model generation unit 55 generates a three-dimensional model. The model generation unit 55 generates a robot device model 3M. The model generation unit 55 generates a robot model 1M based on the three-dimensional shape data of the robot 1. The model generation unit 55 generates a welding gun model 5M based on the three-dimensional shape data of the welding gun 5. The model generation unit 55 generates workpiece models 65M, 66M based on the three-dimensional shape data of the workpieces 65, 66.

[0039] The display control unit 58 displays images of the robot model 1M, the welding gun model 5M, and the workpiece models 65M, 66M. The processing unit 54 can set a robot coordinate system 71 set in the actual robot device 3 in a virtual space in which the robot device model 3M and the workpiece models 65M, 66M are arranged. As with the actual robot device 3, the robot coordinate system 71 can be used to specify the position and posture of the robot and the position and posture of the workpiece in the simulation.

[0040] The installation position where the robot 1 is installed and the positions where the workpieces 65, 66 are arranged can be input as design values ​​by an operator using the input unit 51. Alternatively, the installation position of the robot 1 and the positions of the workpieces 65, 66 may be included in the three-dimensional shape data 50. Here, the installation position of the stand and the position of the workpiece relative to the stand may be input as the position of the workpiece.

[0041] The simulation execution unit 56 changes the position and posture of the robot model 1M in the image 81 in response to the operation of the input unit 51. The worker operates the input unit 51 to bring the welding gun model 5M to a desired position and posture. The worker specifies teaching points 89a to 89h for performing welding work. The position and posture of the robot model 1M are adjusted at each of the teaching points 89a to 89h. Here, spot welding is performed at three locations. The teaching points 89b, 89e, and 89g correspond to the welding points 68a, 68b, and 68c where spot welding is performed.

[0042] The worker sets teaching points 89a and 89b so that the welding gun model 5M faces the point where the first spot welding is performed, as shown by an arrow 101. The worker sets teaching points 89c, 89d, and 89e so that the welding gun model 5M faces the point where the second spot welding is performed, as shown by an arrow 102. The worker sets teaching points 89f and 89g so that the welding gun model 5M faces the point where the third spot welding is performed, as shown by an arrow 103. The worker sets teaching point 89h so that the welding gun model 5M retreats from the point where the third spot welding is performed, as shown by an arrow 104. The worker specifies a method of moving the position of the robot that moves between the teaching points 89a to 89h. Here, it is specified that the position of the robot moves linearly between the teaching points.

[0043] The simulation execution unit 56 executes a simulation according to the positions of the teaching points 89a-89h designated by the operator, the posture of the robot at each of the teaching points 89a-89h, and the method of moving the position of the robot. The simulation execution unit 56 can change the position and posture of the robot model 1M in the image 81 based on the teaching points 89a-89h. Then, the simulation execution unit 56 can estimate a movement path 86a based on the result of the simulation. The movement path 86a is a path along which the tool tip point of the robot device 3 moves. The display control unit 58 displays the movement path 86a superimposed on the images of the robot device model 3M and the workpiece models 65M, 66M.

[0044] In this way, the worker can operate the input unit 51 to change the position and posture of the robot model 1M in the image 81 to set the teaching points 89a to 89h. Alternatively, the worker can create an operation program 41 in which the teaching points are determined in advance and input it to the simulation device 4. The simulation execution unit 56 performs a simulation of the operation of the robot model 1M based on the operation program 41 to calculate the operation path 86a. Then, the display control unit 58 can display the operation path 86a in addition to the images of the robot device model 3M and the workpiece models 65M, 66M.

[0045] The worker moves the robot device model 3M in the image 81 to check the state of the robot device. Then, the worker can correct the position of the teaching point and the posture of the robot at the teaching point. When it is confirmed that the robot device model 3M operates in a desired state, the operation information setting unit 57 sets the position of the teaching point and the posture of the robot at the teaching point as operation information in the operation program. In other words, the operation information setting unit 57 can generate the operation program.

[0046] Incidentally, the position of the robot model 1M and the positions of the workpiece models 65M, 66M in the virtual space are specified by design values, which are desired values. For example, an operator can use the robot coordinate system 71 to specify the position of the robot model 1M and the positions of the workpiece models 65M, 66M as desired values.

[0047] Here, the actual installation position of the robot 1 and the positions of the workpieces 65, 66 fixed to the stand 69 often deviate from the desired design values. As a result, the positions of the workpieces 65, 66 relative to the installation position of the robot 1 often deviate from the desired positions. In the simulation, it is difficult to estimate the exact positions of the workpiece models 65M, 66M relative to the robot device model 3M. Thus, in the simulation, errors occur in the positions where the respective devices and members are arranged.

[0048] The first simulation device 4 performs control to capture images of the actual workpieces 65, 66 with the visual sensor 30 and display, on the simulation image, three-dimensional position information of the surface of the workpiece 65 so as to correspond to the position of the actual workpiece 65. The three-dimensional position information of the surface of the workpiece 65 is displayed so that the deviation of the position where the workpiece 65 is placed relative to the installation position of the robot 1 can be seen.

[0049] FIG. 4 shows a perspective view of the robot and the workpiece when an image of the actual workpiece is captured using the visual sensor. With reference to FIG. 2 and FIG. 4, in this embodiment, the visual sensor 30 is attached to the robot 1 when an operation program is generated. When the robot device 3 actually performs spot welding, the visual sensor 30 is removed. The visual sensor 30 can be fixed to any position of the robot 1. In this embodiment, the visual sensor 30 is fixed to the flange 16 via a support member. The visual sensor 30 is supported by the robot 1 so that the position and posture of the visual sensor 30 can be changed.

[0050] The worker fixes the reference workpieces 65, 66 to the stand 69. It is preferable to select the reference workpieces 65, 66 that have a small manufacturing error. The worker also fixes the workpieces 65, 66 to the reference positions on the stand 69. It is preferable to fix the workpieces 65, 66 so that the positional error of the workpieces 65, 66 on the stand 69 is small.

[0051] In the robot device 3, a sensor coordinate system 73 is set for the visual sensor 30. The sensor coordinate system 73 is a coordinate system whose origin is fixed at an arbitrary position of the visual sensor 30. The position and orientation of the sensor coordinate system 73 change together with the visual sensor 30. The sensor coordinate system 73 in this embodiment is set so that the Z axis is parallel to the optical axis of the camera included in the visual sensor 30.

[0052] The visual sensor 30 can capture images in the imaging range 35. In this embodiment, the visual sensor 30 captures images of the workpieces 65, 66 from above the workpieces 65, 66. The visual sensor 30 captures an image of a surface 65b of the workpiece 65 facing the visual sensor 30. Based on the output of the visual sensor 30, the position information generating unit 59 of the processing unit 54 can set a plurality of three-dimensional points on the surface 65b of the workpiece 65 that is placed within the imaging range 35 and within the range in which the visual sensor 30 focuses.

[0053] The position information generating unit 59 calculates the distance from the visual sensor 30 to a point on the surface of the object based on the parallax between the image captured by the first camera 31 and the image captured by the second camera 32. This calculation is performed for the entire pixel area of ​​the image captured by the first camera 31 to obtain a distance image. The position information generating unit 59 obtains a three-dimensional point as a coordinate value in an arbitrary coordinate system from the position information of the visual sensor for each point in the distance image. In this example, the position information generating unit 59 sets the three-dimensional point in the sensor coordinate system 73.

[0054] When the position and posture of the robot 1 change, the position and posture of the sensor coordinate system 73 also change. The simulation device 4 of this embodiment is calibrated so that the coordinate values ​​of the positions of three-dimensional points in the sensor coordinate system 73 can be converted into coordinate values ​​in the robot coordinate system 71 based on the position and posture of the robot 1. In other words, the simulation device 4 is calibrated so that the coordinate values ​​of the robot coordinate system 71 of three-dimensional points set on the surface 65b of the workpiece 65 can be calculated based on the output of the visual sensor 30.

[0055] The position information generating unit 59 can generate the three-dimensional position information of the surface of the object in the form of a three-dimensional map. A three-dimensional map is a representation of the position information of the surface of the object as a set of coordinate values ​​(x, y, z) of three-dimensional points on the surface of the object. The coordinate values ​​at this time can be represented in any coordinate system such as a sensor coordinate system or a robot coordinate system. Alternatively, the position information generating unit 59 can generate the three-dimensional position information as a distance image. A distance image represents the position information of the surface of the object using a two-dimensional image. In the distance image, the distance from the visual sensor 30 to the three-dimensional point can be represented by the darkness or color of each pixel.

[0056] In this embodiment, the position information generating unit 59 is disposed in the processing unit 54 of the arithmetic processing device 24, but is not limited to this. The position information generating unit may be disposed inside the visual sensor. That is, the visual sensor may include a arithmetic processing device having a processor such as a CPU, and the processor of the visual sensor may function as the position information generating unit. In this case, three-dimensional position information is output from the visual sensor.

[0057] In this embodiment, the workpiece 65 has a shape larger than the imaging range of the visual sensor 30 when the visual sensor 30 is placed at the imaging position. Since the workpiece 65 is larger than the imaging range 35 of the visual sensor 30, it is not possible to image the entire surface 65b of the workpiece 65 in one imaging. For this reason, in this embodiment, the robot 1 changes its position and posture to change the position of the visual sensor 30, and images the entire surface 65b of the workpiece 65 in multiple times.

[0058] As indicated by arrow 105, imaging is performed multiple times by changing the position of imaging range 35 on workpiece 65. At this time, imaging is performed so that a part of imaging range 35 in the previous imaging overlaps with a part of imaging range 35 in the current imaging. That is, imaging is performed while shifting the position of imaging range 35 little by little so that the same part of workpiece 65 is imaged in the previous imaging and the current imaging. The three-dimensional position information acquired by the previous imaging and the three-dimensional position information acquired by the current imaging also include a common part.

[0059] The position information generating unit 59 joins together multiple pieces of three-dimensional position information acquired by the output of the visual sensor 30 so that common parts of the three-dimensional position information overlap. The position information generating unit 59 generates three-dimensional position information of the entire surface 65b of the workpiece 65. In particular, the position information generating unit 59 generates three-dimensional position information for the surface 65b visible to the visual sensor 30. By performing this control, it is possible to acquire the three-dimensional position information of the entire workpiece 65 using the visual sensor 30 whose imaging range is small compared to the size of the workpiece.

[0060] Regarding the movement of the visual sensor, an operation program for the robot device for imaging can be created in advance, and the robot can be automatically driven to perform imaging multiple times. Alternatively, an operator can manually change the position and posture of the robot to perform imaging multiple times. For example, an operator can change the position and posture of the robot by operating a teaching operation panel. Alternatively, as in direct teaching, a force sensor can be attached to the robot to perform control to change the position and posture of the robot so as to follow an external force applied to the robot. An operator can change the position and posture of the robot via the force sensor by directly pushing or pulling the robot.

[0061] FIG. 5 shows an image when a group of three-dimensional points is displayed as three-dimensional position information acquired by the visual sensor. The display control unit 58 displays the three-dimensional position information of the surface 65b of the workpiece 65 as a three-dimensional point 87a. In addition to the multiple three-dimensional points 87a, the image 82 displays a group of three-dimensional points 87 in dashed lines so as to surround the three-dimensional points 87a. The display control unit 58 can display the three-dimensional points 87a based on the coordinate values ​​of the three-dimensional points 87a acquired by the position information generating unit 59. The three-dimensional points 87a correspond to the actual positions of the surface 65b of the workpiece 65. The three-dimensional points 87a are arranged at actual relative positions with respect to the robot.

[0062] In this example, the display control unit 58 displays a movement path 86a calculated by simulation. It can be seen that the movement path 86a is deviated from the position of the point cloud 87 corresponding to the surface 65b of the workpiece 65. The worker can correct the position of each of the teaching points 89a to 89h and the posture of the robot at each of the teaching points 89a to 89h so that they correspond to the position and posture of the point cloud 87. The worker can correct the teaching points by operating the input unit 51 while changing the position and posture of the robot model 1M.

[0063] For example, the operator can operate the input unit 51 to modify the teaching point 89b corresponding to the welding point 68a to the teaching point 90b. The teaching point 89e corresponding to the welding point 68b can be modified to the teaching point 90e. Furthermore, the teaching point 89g corresponding to the welding point 68c can be modified to the teaching point 90g. The other teaching points 89a, 89c, 89d, 89f, and 89h can also be modified to match the positions and orientations of the point cloud 87.

[0064] In this way, in the simulation device of this embodiment, the 3D position information of the actual workpiece can be superimposed on the images of the robot model and the workpiece model displayed in the simulation. Therefore, the deviation of the teaching point set on the actual workpiece can be easily confirmed. Then, the position of the teaching point and the posture of the robot at the teaching point can be corrected using the simulation device.

[0065] The operation information setting unit 57 can correct the operation information in the operation program 41. That is, the operation information setting unit 57 can correct the positions of the teaching points and the posture of the robot at the teaching points. In this way, the operation information setting unit 57 can correct the operation program based on the corrected teaching points.

[0066] In addition to correcting the teaching points in the operation program, the operator may correct the actual installation position of the robot, the installation position of the peripheral device that supports the workpiece, or the position of the workpiece on the peripheral device. For example, the operator may correct the installation position of the stand 69 that fixes the workpieces 65 and 66, or the reference position of the workpieces 65 and 66 on the stand 69.

[0067] 6 shows an image displaying a robot device model, a workpiece model, and a point cloud of three-dimensional points in this embodiment. The worker can select information to be displayed on the display unit 52 from among the model of the robot, the motion path, and the three-dimensional position information. In the image 83, the robot device model 3M, a point cloud 87 of three-dimensional points 87a, and workpiece models 65M and 66M are displayed by the operation of the worker.

[0068] The distance calculation unit 60 of the processing unit 54 can calculate the actual distance between any point on the workpiece models 65M, 66M included in the simulation image and one three-dimensional point 87a. In this example, the worker designates a point 65Ma at a corner of the surface of the workpiece model 65M and a three-dimensional point 87aa located at a corner of the point cloud 87. The point 65Ma and the three-dimensional point 87aa correspond to each other. The point 65Ma and the three-dimensional point 87aa are points corresponding to a corner of the surface 65b of the workpiece 65.

[0069] The distance calculation unit 60 acquires the coordinate value of the point 65Ma and the coordinate value of the three-dimensional point 87aa. For example, the coordinate value in the robot coordinate system 71 is acquired. Then, the distance calculation unit 60 calculates the distance between the point 65Ma and the three-dimensional point 87aa. The display control unit 58 can display the distance calculated by the distance calculation unit 60 on the image 83. In this way, the distance calculation unit 60 can calculate and display the distance between an arbitrary point in the model and an arbitrary point in the three-dimensional position information. The worker can acquire the distance between any two points in the simulation. For example, the worker can know how much the position of the workpiece model relative to the robot model is deviated from the position of the workpiece relative to the actual robot.

[0070] The position information generating unit in the above embodiment can obtain three-dimensional points on the surface of the stand and the installation surface that are visible to the visual sensor. In this embodiment, the position information generating unit excludes three-dimensional points set on the surface of the stand and the installation surface from the three-dimensional points on the surface of the object. For example, the position information generating unit can select three-dimensional points placed on the surface of the workpiece based on a judgment value of the distance from the visual sensor.

[0071] The three-dimensional points displayed on the display unit are not limited to this form. For example, three-dimensional points placed on the surface of the stand may be displayed on the simulation image. In other words, three-dimensional points set on the surface of the robot's peripheral device may be displayed on the simulation image.

[0072] Furthermore, in the above embodiment, the three-dimensional position information on the top surface of the workpiece 65 is acquired, but this is not limited to the embodiment. By changing the position and posture of the robot 1, the workpieces 65, 66 can be imaged from various directions to acquire three-dimensional position information on the surface. For example, the positions of three-dimensional points set on the side surfaces of the workpieces 65, 66 may be acquired. Then, the display control unit 58 may display the three-dimensional points set on the side surfaces of the workpieces 65, 66 on the display unit.

[0073] In the above embodiment, the position and posture of the robot are changed to capture images multiple times because the workpiece is large relative to the imaging range of the visual sensor, but this is not limited to the embodiment. If the workpiece is small relative to the imaging range, the entire workpiece can be captured in one imaging by the visual sensor. In this case, the visual sensor may be fixed to a stand or the like. For example, the visual sensor may be fixed above the workpiece, and three-dimensional position information of the entire top surface of the workpiece may be obtained in one imaging.

[0074] 7 shows a block diagram of a second robot system in this embodiment. The second robot system includes a first robot device 3 and a second simulation device 8. The second simulation device 8 detects the actual position of the workpiece based on three-dimensional position information acquired from the output of a visual sensor 30. Then, the second simulation device 8 automatically corrects the operation information included in the operation program so as to correspond to the actual position of the workpiece.

[0075] The second simulation device 8 has a processing unit 63 with a different configuration from the processing unit 54 of the first simulation device 4. The processing unit 63 of the second simulation device 8 includes a position detection unit 61 that detects the actual position of the workpiece by matching reference data of the workpiece with three-dimensional position information of the surface of the workpiece. The position detection unit 61 corresponds to a processor that operates according to a simulation program. The processor functions as the position detection unit 61 by carrying out control defined in the program.

[0076] Further, the motion information setting unit 57 corrects the motion information including the information of the teaching points in the motion program. The motion information setting unit 57 corrects the motion information so as to correspond to the actual position of the workpiece based on the position of the workpiece model in the simulation and the actual position of the workpiece detected by the position detection unit 61.

[0077] FIG. 8 shows an image displaying a point cloud of three-dimensional points acquired from the output of the visual sensor. With reference to FIG. 7 and FIG. 8, in the image 84, a point cloud 87 of three-dimensional points 87a is displayed as three-dimensional position information by capturing images of the workpieces 65, 66 with the visual sensor 30. In the image 84, the robot device model 3M, the workpiece models 65M, 66M, the point cloud 87 of three-dimensional points 87a, and the movement path 86a are displayed. The movement path 86a is a path generated by simulation based on the positions of the workpiece models 65M, 66M. The point cloud 87 corresponding to the position of the surface 65b of the actual workpiece 65 is shifted from the upper surface of the workpiece model 65M. As a result, the movement path 86a is also shifted from the position of the point cloud 87 of the three-dimensional points 87a.

[0078] The position detection unit 61 detects the actual position of the workpiece corresponding to the point cloud 87 by matching the point cloud 87 with the reference data of the workpiece 65. In the matching, three-dimensional shape data of the workpiece output from a CAD device or the like can be used as the reference data of the workpiece.

[0079] As the matching, for example, the position detection unit 61 generates a plurality of mesh information from a large number of three-dimensional points 87a. The actual position of the workpiece may be detected by matching this mesh information with mesh information generated from the three-dimensional shape data of the workpiece.

[0080] Alternatively, a characteristic part having a characteristic shape in the workpiece may be selected, and the three-dimensional shape data of the characteristic part may be matched with the point cloud of three-dimensional points. A characteristic part in the point cloud of three-dimensional points may be specified, and the position of the characteristic part may be detected. The actual position of the workpiece may be detected based on the positions of a plurality of characteristic parts. For example, the convex part 65a of the workpiece 65 may be selected as the characteristic part. By matching the three-dimensional shape data of the convex part 65a with the point cloud of three-dimensional points as reference data, the positions of a plurality of convex parts in the point cloud may be specified. Alternatively, the contour of the workpiece may be estimated from the point cloud of three-dimensional points, and matching may be performed with the contour in the three-dimensional shape data of the workpiece.

[0081] In this embodiment, in order to determine the position and orientation of the workpiece, a coordinate system (user coordinate system) having an origin on the surface of the workpiece and having a predetermined orientation with respect to the workpiece is set in advance. In this embodiment, this coordinate system is called the workpiece coordinate system. The workpiece coordinate system is a coordinate system fixed to the workpiece.

[0082] A workpiece coordinate system 74a is set on the upper surface of the workpiece model 65M. In this embodiment, the origin of the workpiece coordinate system is located at one corner of the surface 65b of the workpiece 65. The X-axis and Y-axis of the workpiece coordinate system are set along the direction in which the edge of the surface 65b extends. For example, the position of the workpiece corresponds to the position of the origin of the workpiece coordinate system, and can be expressed by the coordinate values ​​(x, y, z) of the robot coordinate system 71. The posture of the workpiece corresponds to the orientation of the workpiece coordinate system, and can be expressed by the coordinate values ​​(w, p, r) of the robot coordinate system 71.

[0083] The operation information defined in the operation program can be specified by the position and orientation of the workpiece coordinate system expressed in the robot coordinate system, and the position of a teaching point expressed in the workpiece coordinate system and the orientation of the robot at the teaching point.

[0084] The position detection unit 61 detects the position of the workpiece 65 corresponding to the point cloud 87 by matching the point cloud 87 including the three-dimensional points 87a with the reference data of the workpiece. The position detection unit 61 calculates a point corresponding to a corner of the upper surface of the workpiece, and sets the workpiece coordinate system 74b so that this point becomes the origin. In addition, the workpiece coordinate system 74b is set so that the X-axis and the Y-axis extend along the edge of the point cloud 87. The position and orientation of the workpiece coordinate system 74b corresponding to the actual position and orientation of the workpiece 65 can be calculated by the robot coordinate system 71. In addition, the display control unit 58 can display the workpiece coordinate system 74b set by the position detection unit 61.

[0085] The motion information setting unit 57 performs control to correct the position and orientation of the work coordinate system 74a set in the work model 65M by simulation in the motion program so that the position and orientation of the work coordinate system 74b are calculated from the point group 87 of the three-dimensional points 87a. In this way, the motion information setting unit 57 can correct the motion information in the motion program. Here, the position and orientation of the work coordinate system can be corrected from among the motion information.

[0086] 9 shows an image displaying the motion path, robot model, workpiece model, and three-dimensional point cloud after the motion information is corrected in the motion program. By correcting the position and posture of the workpiece coordinate system as the motion information, the positions of the teaching points and the posture of the robot at the teaching points are corrected. Each of the teaching points 89b, 89e, and 89g is moved as indicated by the arrow 106, and corrected teaching points 90b, 90e, and 90g are set. In addition, the simulation execution unit 56 executes a simulation, and a corrected motion path 86b is displayed.

[0087] In the second simulation device, it is possible to automatically correct deviations in the relative positions and relative postures of the workpieces 65, 66 with respect to the robot 1. More specifically, it is possible to automatically correct the operation information included in the operation program so as to correspond to the actual position of the workpiece based on the three-dimensional position information acquired from the output of the visual sensor.

[0088] In addition, by setting a workpiece coordinate system on the surface of the workpiece and determining the positions of the teaching points and the posture of the robot at the teaching points in the workpiece coordinate system, control for correcting the operation program can be simplified. There is no need to change the positions of the teaching points expressed in the workpiece coordinate system and the posture of the robot at the teaching points in the operation program, and the operation information can be corrected by changing the position and posture of the workpiece coordinate system.

[0089] In the above embodiment, the motion information setting unit executes control to correct the position and posture of the workpiece coordinate system in the motion program, but is not limited to this embodiment. The motion information setting unit can correct the motion information in the motion program by any control.

[0090] For example, the position of the teaching point and the posture of the robot at the teaching point as the operation information defined in the operation program can be specified by the coordinate values ​​of the robot coordinate system. In this case, the operation information setting unit calculates the relative position and relative posture of the work coordinate system set in the point cloud of three-dimensional points with respect to the work coordinate system set in the work model. The movement direction and movement amount of the work coordinate system correspond to the errors in the position and posture of the work. For this reason, the operation information setting unit can correct the position of the teaching point expressed in the robot coordinate system and the posture of the robot at the teaching point based on the relative position and relative posture of the work coordinate system.

[0091] Here, the operation information setting unit 57 can have a function of converting the coordinate values ​​of one coordinate system for one point in the simulation into coordinate values ​​of another coordinate system. For example, the coordinate values ​​of one point expressed in the work coordinate system set in the work model can be converted into coordinate values ​​expressed in the robot coordinate system based on the position and posture of the robot. Or, one coordinate value expressed in the robot coordinate system can be converted into coordinate values ​​expressed in the work coordinate system. Or, the coordinate values ​​of the sensor coordinate system may be converted into coordinate values ​​of the robot coordinate system based on the position and posture of the robot.

[0092] In the above embodiment, a welding gun is attached to the robot as a work tool, but this is not limited to this embodiment. Any work tool can be adopted depending on the work to be performed by the robot device. The simulation device of this embodiment can be applied to a robot device that performs any work. Next, an example of a robot device that performs work other than spot welding will be described.

[0093] FIG. 10 shows a perspective view of a second robot device of this embodiment. The second robot device 9 includes a hand 6 as a working tool attached to a robot 1. The hand 6 is formed so as to be able to grip and release a workpiece 67. The second robot device 9 includes a conveyor 7 for transporting the workpiece 67 as a peripheral device of the robot 1. The conveyor 7 transports the workpiece 67 in the direction indicated by an arrow 107. The second robot device 9 transports the workpiece 67 from a predetermined position and places it on the conveyor 7.

[0094] In the second robot device 9 as well, there may be an error in the installation position of the robot 1 and the installation position of the conveyor 7. In the simulation device of the second robot device 9 as well, a reference workpiece 67 is placed at a reference position of the conveyor 7. The workpiece is imaged by a visual sensor attached to the robot or the like to obtain three-dimensional position information of the surface of the workpiece. In the simulation device, the three-dimensional position information can be displayed in addition to images of the robot device model and the workpiece model. In the simulation device, the worker can correct the operation information of the operation program. In addition, the simulation device can correct the operation information included in the operation program based on the three-dimensional position information obtained from the output of the visual sensor. For example, the teaching point for the robot device to release the workpiece on the conveyor can be corrected.

[0095] The visual sensor of this embodiment is a stereo camera including multiple two-dimensional cameras, but is not limited to this form. Any sensor capable of acquiring three-dimensional position information of the surface of an object can be adopted as the visual sensor. For example, a TOF (Time of Flight) camera that acquires position information of a three-dimensional point based on the flight time of light can be adopted as the visual sensor. Alternatively, a device that scans a laser range finder in a predetermined area to detect the position of the surface of an object may be adopted as the visual sensor.

[0096] The processing unit of the simulation device in this embodiment is configured by a processing device separate from the control device of the robot, but is not limited to this form. The control device of the robot may have the function of the simulation device. That is, the processor of the processing device of the control device may function as the processing unit. Furthermore, in the case where the teaching pendant includes a processing device having a processor, the teaching pendant may have the function of the simulation device. That is, the processor of the teaching pendant may function as the processing unit.

[0097] The above-described embodiments can be combined as appropriate. In each of the above-described controls, the order of steps can be changed as appropriate within the scope of not changing the functions and actions. In each of the above-described figures, the same or equivalent parts are given the same reference numerals. Note that the above-described embodiments are examples and do not limit the invention. In addition, the embodiments include modifications of the embodiments shown in the claims. [Explanation of symbols]

[0098] 1. Robot 1M Robot Model 3.9 Robotic Devices 3M Robotic Equipment Model 4.8 Simulation device 30 Visual Sensor 35 Image range 50 3D shape data 52 Display section 54,63 Processing section 55 Model Generation Unit 56 Simulation Execution Department 57 Operation information setting section 59 Location information generation section 60 Distance calculation unit 61 Position detection unit 65,66,67 Work 65M,66M Work Model 65b surface 71 Robot Coordinate System 73 Sensor coordinate system 74a, 74b Work coordinate system 81~84 images 86a, 86b Motion path 87 point cloud 87a,87aa 3D point 89a~89h Teaching points 90b, 90e, 90g Teaching points

Claims

1. A simulation device that performs a simulation of an operation of a robot device including a robot, a model generating unit that generates a robot device model and a workpiece model for a simulation based on three-dimensional shape data of the robot device and three-dimensional shape data of the workpiece; a display unit that displays an image of the robot device model and an image of the workpiece model; a motion information setting unit that generates a motion program including teaching points; a simulation execution unit that executes a simulation of the operation of the robot device based on the teaching points to estimate a motion path of the robot; a position information generating unit that generates three-dimensional position information including information on the positions of three-dimensional points set on the surface of the workpiece based on an output of a visual sensor that captures an image of an actual workpiece; the display unit displays a plurality of three-dimensional points corresponding to the three-dimensional position information, superimposed on the image of the robot device model, the image of the workpiece model, and the motion path of the robot; The workpiece has a shape larger than an imaging range when the visual sensor is disposed at an imaging position, The position information generating unit generates a plurality of pieces of three-dimensional position information acquired by capturing images a plurality of times with the imaging position changed, and generates three-dimensional position information for the entire surface of the workpiece facing the visual sensor by connecting the plurality of pieces of three-dimensional position information together; The display unit displays three-dimensional points for the entire surface of the workpiece facing the visual sensor.

2. The simulation device according to claim 1 , further comprising a distance calculation unit that calculates a distance between a point on the workpiece model and one three-dimensional point.

3. A simulation device that performs a simulation of an operation of a robot device including a robot, a model generating unit that generates a robot device model and a workpiece model for a simulation based on three-dimensional shape data of the robot device and three-dimensional shape data of the workpiece; a simulation execution unit that executes a simulation of an operation of the robot device; an operation information setting unit that sets operation information defined in an operation program based on a simulation of an operation of the robot device; a position information generating unit that generates three-dimensional position information of a surface of a workpiece based on an output of a visual sensor that captures an image of the actual workpiece; a position detection unit that detects an actual position of the workpiece by matching the reference data of the workpiece with three-dimensional position information of the surface of the workpiece; the operation information setting unit corrects the operation information included in the operation program so as to correspond to the actual position of the workpiece based on the position of the workpiece model in the simulation and the position of the actual workpiece detected by the position detection unit; A work coordinate system with an origin set for the work is preset. The workpiece coordinate system is a coordinate system that is fixed to the workpiece and whose position and orientation change together with the workpiece. The operation information defined in the operation program includes a position and a posture of the workpiece coordinate system, and a position of a teaching point defined in the workpiece coordinate system and a posture of the robot at the teaching point, the operation information setting unit corrects the position and orientation of the work coordinate system set in the work model in the simulation so as to correspond to the position and orientation of the work coordinate system of the actual work calculated from the three-dimensional position information; A simulation device in which the operation information setting unit maintains the position of a teaching point defined in a work coordinate system and the posture of the robot at the teaching point without change, and corrects the position and posture of the work coordinate system defined in an operation program.

4. 4. The simulation device according to claim 3, wherein the motion information setting section has a function of converting, for one point in a simulation, a coordinate value in one coordinate system into a coordinate value in another coordinate system.

Images