A simulation device that uses 3D position information obtained from the output of a vision sensor.

Abstract

The simulation device of the present invention includes a simulation execution unit that simulates the actions of a robot device and estimates the robot's motion path. The simulation device also includes a position information generation unit that generates three-dimensional position information of the workpiece surface based on the output of a vision sensor that captures an image of the actual workpiece. A display unit overlays the three-dimensional position information of the workpiece surface onto an image of the robot device model, an image of the workpiece model, and the robot's motion path to display the simulation device.

Description

Technical Field

[0001] Invention Field

[0002] This invention relates to a simulation device that uses three-dimensional position information obtained from the output of a vision sensor. Prior Technology

[0003] Background of the Invention

[0004] A robotic device, equipped with a robot and its working tools, changes the position and posture of the robot to alter the position and posture of the working tools. The robotic device can perform various tasks that involve changing the position and posture of the working tools. The robot's position and posture are changed according to a motion program generated before the task is performed. The motion program contains teach points that determine the robot's position and posture.

[0005] When performing new tasks with a robotic device, motion programs must be created. These programs can drive an actual robot to teach it the position and posture of the teaching point. Furthermore, a simulation device for simulating the motion of a robotic device is known (e.g., Japanese Patent Application Publication No. 2015-93345). The simulation device allows for visual confirmation of the robotic device's motion. Operators can use the simulation device to teach the teaching point by demonstrating the robotic device's motion.

[0006] Furthermore, in the prior art, it is known to install three-dimensional vision sensors on robots, and to detect the position of objects relative to the robot by processing the position information obtained by the three-dimensional vision sensors, and to perform various tasks (e.g., Japanese Patent Application Publication No. 2006-35384).

[0007] Furthermore, a known control method involves installing two-dimensional or three-dimensional sensors on a robot to actually drive the robot device to correct the teach point (e.g., Japanese Patent Application Publication Nos. 7-84631 and 2004-255547). In such devices, for example, markers are disposed on the actual workpiece. The position of the markers on the actual workpiece is obtained based on the output of the vision sensor. The position of the teach point in the motion program can be corrected based on the position of the actual markers. Prior technology documents Patent documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2015-93345 Patent Document 2: Japanese Patent Application Publication No. 2006-35384 Patent Document 3: Japanese Patent Application Publication No. 7-84631 Patent Document 4: Japanese Patent Application Publication No. 2004-255547 Summary of the Invention

[0009] Invention Summary The problem the invention aims to solve

[0010] In simulation devices, it is advisable to determine the correct position and posture of the robot for the workpiece being worked on. However, the robot's placement on the mounting surface and the positions of surrounding machines such as the support platform for the workpiece rarely match the expected design values. In particular, deviations in the placement of surrounding machines correspond to deviations in the placement of the workpiece. The actual robot placement and workpiece placement positions often deviate from the simulated positions. Therefore, it is advisable to slightly shift the position of the teaching point and the robot's posture at the teaching point.

[0011] In previous technologies, motion programs created using simulation devices were input into the control system of the actual robot. Operators then drove the robot, correcting the positions of the teach points in the motion program and the robot's posture at those teach points.

[0012] However, simulation devices struggle to account for the actual robot placement and workpiece positioning to adjust the teaching point's location and the robot's posture at the teaching point. In other words, it's difficult to predict how the robot's placement or the surrounding machines that hold the workpiece will deviate from their intended positions, making it challenging to correct the motion program within the simulation device.

[0013] In the control of placing markings on the surface of actual workpieces and detecting the marking positions using a vision sensor, there is a problem that the marking positions set on the workpiece in the simulation must be consistent with the marking positions placed on the actual workpiece. Furthermore, when marking detection fails, the marking position cannot be identified. Therefore, it is necessary to create a system where markings on the workpiece can be easily detected by the output of a vision sensor. The means to solve the problem

[0014] The simulation device disclosed herein simulates the actions of a robot device equipped with a robot. The simulation device includes a model generation unit that generates a simulated robot device model and a workpiece model based on the three-dimensional shape data of the robot device and the workpiece. The simulation device includes a display unit that displays images of the robot device model and the workpiece model. The simulation device includes a motion information setting unit that generates a motion program including teaching points. The simulation device includes a simulation execution unit that simulates the actions of the robot device based on the teaching points to estimate the robot's motion path. The simulation device includes a position information generation unit that generates three-dimensional position information of the workpiece surface based on the output of a vision sensor that captures an image of the actual workpiece. The display unit overlays the three-dimensional position information of the workpiece surface onto the images of the robot device model, the workpiece model, and the robot's motion path.

[0015] The second-state simulation device disclosed herein simulates the actions of a robot device equipped with a robot. The simulation device includes a model generation unit that generates a simulated robot device model and a workpiece model based on the three-dimensional shape data of the robot device and the three-dimensional shape data of the workpiece. The simulation device also includes a simulation execution unit that simulates the actions of the robot device. Furthermore, the simulation device includes a motion information setting unit that sets motion information determined in the motion program based on the simulated actions of the robot device. The simulation device further includes a position information generation unit and a position detection unit. The position information generation unit generates three-dimensional position information of the workpiece surface based on the output of a vision sensor that captures an image of the actual workpiece. The position detection unit detects the position of the actual workpiece by matching the workpiece's reference data with the three-dimensional position information of the workpiece surface. Finally, the motion information setting unit corrects the motion information included in the motion program based on the position of the simulated workpiece model and the position of the actual workpiece detected by the position detection unit, so that it corresponds to the position of the actual workpiece. Invention Effects

[0016] Based on the present disclosure, a simulation device can be provided that can easily identify or correct deviations in the position and posture of a robot. Simple Explanation of the Diagram

[0017] Figure 1 is a perspective view of the first robotic device in the embodiment. Figure 2 is a block diagram of the first robot system with a first robot device and a first simulation device in an embodiment. Figure 3 shows an image of the simulation performed by the action program. Figure 4 is a 3D view of the robot device and the workpiece when the workpiece is photographed by a vision sensor. Figure 5 shows an image of a robot device model and an image of a group of three-dimensional points detected by a vision sensor. Figure 6 shows images of the robot device model, the workpiece model, and a group of points of three-dimensional points detected by a vision sensor. Figure 7 is a block diagram of a second robot system with a first robot device and a second simulation device in an embodiment. Figure 8 shows an image of the robot device model used to illustrate the workpiece coordinate system set on the workpiece model, an image of the workpiece model, and an image of the point group of three-dimensional points. Figure 9 shows the corrected motion path, robot device model, workpiece model, and point group of three-dimensional points in the motion program. Figure 10 is a perspective view of the second robotic device in the embodiment. Implementation

[0018] Forms used to implement inventions

[0019] The simulation apparatus of this embodiment will be described with reference to Figures 1 to 10. This simulation apparatus simulates the actions of a robotic device equipped with a robot. This simulation apparatus processes the output of a three-dimensional vision sensor. The three-dimensional vision sensor acquires information about the surface of a workpiece, which is the object of the operation. This simulation apparatus displays the position information of the workpiece surface along with the simulated model, or corrects the teaching points determined in the simulation based on the position information of the workpiece surface.

[0020] Figure 1 is a perspective view of the first robot device of this embodiment. Figure 2 is a block diagram of the first robot system of this embodiment. Referring to Figures 1 and 2, the first robot system includes a first robot device 3 and a first simulation device 4. The first robot device 3 performs point welding. The first robot device 3 includes a welding torch 5 as a working tool and a robot 1 for changing the position and posture of the welding torch 5. The robot device 3 includes a control device 2 for controlling the robot 1 and the welding torch 5. The first robot system includes a vision sensor 30, which acquires information about the surface of the workpiece 65, which is the object of the work.

[0021] The first workpiece 65 in this embodiment is a plate-shaped member. The first workpiece 65 has protrusions 65a formed corresponding to welding points 68a-68c. The first workpiece 65 has a surface 65b. The second workpiece 66 is a plate-shaped member having the same planar shape as the first workpiece 65. The robot device 3 performs point welding at three welding points 68a-68c. The first workpiece 65 and the second workpiece 66 are fixed to each other. Workpieces 65 and 66 are supported by a frame 69.

[0022] The robot 1 of this embodiment is a multi-joint robot comprising a plurality of joints. Robot 1 includes an upper arm 11 and a lower arm 12. The lower arm 12 is supported by a rotating base 13. The rotating base 13 is supported by a base 14. Robot 1 includes a wrist 15 connected to the end of the upper arm 11. The wrist 15 includes a flange 16 for fixing a welding gun 5. Robot 1 of this embodiment has six drive axes, but is not limited to this configuration. The robot can be any robot with a mobile working tool.

[0023] The robot 1 of this embodiment includes a robot drive device 21 that drives the upper arm 11 and other constituent components. The robot drive device 21 includes a plurality of drive motors for driving the upper arm 11, lower arm 12, rotary base 13, and wrist 15. The welding torch 5 includes a tool drive device 22 that drives the welding torch 5. The tool drive device 22 of this embodiment includes a motor that drives a movable electrode relative to a fixed electrode of the welding torch 5.

[0024] The control device 2 includes a computing processing unit 24 (computer), which includes a CPU (Central Processing Unit) as a processor. The computing processing unit 24 has RAM (Random Access Memory) and ROM (Read Only Memory) interconnected to the CPU via buses. The robot device 3 drives the robot 1 and the welding gun 5 according to a pre-made motion program. In this embodiment, the robot device 3 automatically performs point welding on workpieces 65 and 66 at welding points 68a to 68c.

[0025] The processing unit 24 of the control device 2 includes a memory unit 42, which stores information related to the control of the robot device 3. The memory unit 42 can be constructed using a non-temporary memory medium capable of storing information. For example, the memory unit 42 can be constructed using volatile memory, non-volatile memory, magnetic memory media, or optical memory media. The motion program used by the robot device 3 to perform the spot welding operation is stored in the memory unit 42.

[0026] The processing unit 24 includes a motion control unit 43 that sends motion commands. The motion control unit 43 sends motion commands to the robot drive unit 44 to drive the robot 1 according to the motion program. The robot drive unit 44 includes circuitry for driving the drive motor. The robot drive unit 44 supplies power to the robot drive device 21 according to the motion commands. Furthermore, the motion control unit 43 sends motion commands to drive the tool drive device 22 to the work tool drive unit 45. The work tool drive unit 45 includes circuitry for flowing power into the electrodes and circuitry for driving the motor of the movable electrodes. The work tool drive unit 45 supplies power to the tool drive device 22 according to the motion commands.

[0027] The motion control unit 43 is equivalent to a processor that drives the robot device according to its motion program. The processor reads the motion program and implements the controls determined in the motion program to perform the functions of the motion control unit 43.

[0028] Robot 1 includes a state detector for detecting the position and orientation of robot 1. In this embodiment, the state detector includes a rotation angle detector 23, which is mounted on the drive motors of each drive shaft of the robot drive unit 21. The rotation angle detector 23 is, for example, an encoder. The position and orientation of robot 1 are detected by the output of the rotation angle detector 23.

[0029] The control device 2 includes a teaching operation panel 49, which serves as a control panel for operators to manually operate the robot device 3. The teaching operation panel 49 includes an input section 49a, which can input information about the robot 1, the welding gun 5, and the vision sensor 30. The input section 49a is composed of operating components such as a keyboard and a dial pad. The teaching operation panel 49 includes a display section 49b, which displays information related to the control of the robot device 3. The display section 49b is composed of a display panel such as a liquid crystal display panel.

[0030] In this embodiment of the robot device 3, a robot coordinate system 71 is provided that remains stationary when the position and posture of the robot 1 change. In the example shown in Figure 1, the origin of the robot coordinate system 71 is located on the base 14 of the robot 1. The robot coordinate system 71 is also referred to as a world coordinate system or a reference coordinate system. The origin of the robot coordinate system 71 is fixed, and the directions of the coordinate axes are fixed. Even if the position and posture of the robot 1 change, the position and posture of the robot coordinate system 71 remain unchanged.

[0031] The robot device 3 is equipped with a tool coordinate system 72, which has an origin set at any position on the working tool. The position and orientation of the tool coordinate system 72 change with the welding torch 5. In this embodiment, the origin of the tool coordinate system 72 is set at the tool tip point (the tip of the fixed electrode). The position of the robot 1 corresponds to the position of the tool tip point (the position of the origin of the tool coordinate system 72). Furthermore, the orientation of the robot 1 corresponds to the orientation of the tool coordinate system 72 relative to the robot coordinate system 71.

[0032] The vision sensor 30 of this embodiment is a three-dimensional camera capable of acquiring three-dimensional position information of the surface of an object. The vision sensor 30 of this embodiment is a stereo camera including a first camera 31 and a second camera 32. Cameras 31 and 32 are two-dimensional cameras capable of capturing two-dimensional images. The relative positions of the two cameras 31 and 32 are predetermined. The vision sensor 30 of this embodiment includes a projector 33, which projects patterned light, such as striped patterns, onto the surface 65b of the workpiece 65.

[0033] The first simulation device 4 simulates the actions of the robot device 3. The simulation device 4 arranges the three-dimensional model of the robot 1, the three-dimensional model of the welding gun 5, and the three-dimensional models of the workpieces 65 and 66 in the same virtual space to simulate the actions of the robot device 3.

[0034] The simulation device 4 in this embodiment is configured as an arithmetic processing unit including a CPU as a processor. The simulation device 4 has a memory unit 53 that stores arbitrary information related to the simulation of the robot device 3. The memory unit 53 can be configured using a non-temporary memory medium capable of storing information. For example, the memory unit 53 can be configured using volatile memory, non-volatile memory, magnetic memory medium, or optical memory medium. The program used to implement the simulation of the robot device is stored in the memory unit 53.

[0035] In the simulation device 4, three-dimensional shape data 50 of robot 1, welding gun 5, and workpieces 65 and 66 has been input. The three-dimensional shape data 50 includes the three-dimensional shape data of the robot, working tools, peripheral machines, and workpieces used for simulating the robotic device. The three-dimensional shape data 50 can be, for example, data output from a CAD (Computer-Aided Design) device. The three-dimensional shape data 50 is stored in the memory unit 53.

[0036] The simulation device 4 includes an input unit 51, which can input simulation information related to the robot device 3. The input unit 51 is composed of operating components such as a keyboard, mouse, and dial pad. The simulation device 4 includes a display unit 52, which displays simulation information related to the robot device 3. As described later, the display unit 52 displays images of the robot device 3 model and images of the workpieces 65 and 66 models. The display unit 52 is composed of a display panel such as a liquid crystal display panel. Furthermore, when the simulation device has a touch panel type display panel, the display panel functions as both an input unit and a display unit.

[0037] The simulation device 4 includes a processing unit 54, which performs computational processing for the simulation of the robot device 3. Furthermore, the processing unit 54 has the function of processing information acquired by the vision sensor 30. The processing unit 54 includes a model generation unit 55, which generates a model of the robot device (i.e., a robot device model) and a model of the workpieces (i.e., a workpiece model) based on three-dimensional shape data 50 containing three-dimensional shape data of the robot device 3 and three-dimensional shape data of the workpieces 65 and 66.

[0038] The processing unit 54 includes a simulation execution unit 56, which simulates the actions of the robot device 3. The simulation execution unit 56 has the function of moving the robot device model on the screen in response to operations performed by the input unit 51 by the operator. Alternatively, the simulation execution unit 56 simulates the actions of the robot device 3 based on pre-generated teaching points. For example, the simulation execution unit 56 simulates the actions of the robot device 3 based on an action program 41 that includes teaching points.

[0039] The processing unit 54 includes a position information generation unit 59, which generates three-dimensional position information of the surface of the workpiece 65 based on the output of the vision sensor 30 that captures images of the actual workpieces 65 and 66. The processing unit 54 also includes a motion information setting unit 57, which sets motion information determined in the motion program 41 of the robot device 3 based on the simulation of the robot device 3's motion. The motion information setting unit 57 generates a motion program that includes teaching points. The motion information of the robot device 3 includes the position of the teaching points and the posture of the robot 1 at the teaching points.

[0040] The processing unit 54 includes a display control unit 58, which controls the image displayed on the display unit 52. The processing unit 54 also includes a distance calculation unit 60, which calculates the distance between any two points in the image displayed on the display unit 52 by specifying any two points. The distance calculation unit 60 can calculate the distance between a point in the image of the workpiece model and a single three-dimensional point included in a group of three-dimensional points disposed on the surface of the workpiece.

[0041] The processing unit 54 is equivalent to a processor that operates according to a simulated program. A simulated program is pre-created and stored in the memory unit 53. The processor reads the simulated program and implements controls determined within the program to function as the processing unit 54. Furthermore, the model generation unit 55, simulation execution unit 56, motion information setting unit 57, display control unit 58, position information generation unit 59, and distance calculation unit 60 included in the processing unit 54 are equivalent to processors that operate according to a simulated program. The processor implements controls determined within the program to function as each unit.

[0042] Figure 3 shows an example of an image displayed on the display unit of the simulation device. Figure 81 shows the state after the simulation of the robot device 3 has been performed. The model generation unit 55 generates a three-dimensional model. The model generation unit 55 generates a robot device model 3M. The model generation unit 55 generates the robot model 1M based on the three-dimensional shape data of the robot 1. The model generation unit 55 generates a welding gun model 5M based on the three-dimensional shape data of the welding gun 5. The model generation unit 55 generates workpiece models 65M and 66M based on the three-dimensional shape data of workpieces 65 and 66.

[0043] The display control unit 58 displays images of the robot model 1M, the welding gun model 5M, and the workpiece models 65M and 66M. The processing unit 54 can set the robot coordinate system 71 of the actual robot device 3 in the virtual space configured with the robot device model 3M and the workpiece models 65M and 66M. The robot coordinate system 71 can be used in the simulation to specify the position and posture of the robot and the position and posture of the workpiece, just like the actual robot device 3.

[0044] The robot 1's position and the positions of workpieces 65 and 66 can be set by the operator inputting design values ​​through the input unit 51. Alternatively, the robot 1's position and the positions of workpieces 65 and 66 can be included in the three-dimensional shape data 50. Furthermore, the position of the workpiece can be determined by inputting the platform's position and the workpiece's position relative to the platform.

[0045] The simulation execution unit 56, in response to the operation of the input unit 51, changes the position and posture of the robot model 1M in the image 81. The operator operates the input unit 51 to position and posture the welding gun model 5M as desired. The operator designates teaching points 89a-89h for the welding operation. At each teaching point 89a-89h, the position and posture of the robot model 1M are adjusted. Point welding is then performed at three locations. Teaching points 89b, 89e, and 89g correspond to welding points 68a, 68b, and 68c where point welding is performed.

[0046] The operator sets teaching points 89a and 89b, causing the welding gun model 5M to move to the point where the first point welding is performed, as shown by arrow 101. The operator sets teaching points 89c, 89d, and 89e, causing it to move to the point where the second point welding is performed, as shown by arrow 102. The operator sets teaching points 89f and 89g, causing it to move to the point where the third point welding is performed, as shown by arrow 103. The operator sets teaching point 89h, causing the welding gun model 5M to retreat from the point where the third point welding is performed, as shown by arrow 104. The operator specifies the method of moving the robot's position between teaching points 89a and 89h. Here, it is specified that the robot's position moves in a straight line between the teaching points.

[0047] The simulation execution unit 56 performs simulations according to the positions of the teaching points 89a-89h specified by the operator, the robot's posture at each teaching point 89a-89h, and the robot's movement method. The simulation execution unit 56 can change the position and posture of the robot model 1M in image 81 based on the teaching points 89a-89h. Then, the simulation execution unit 56 can estimate the motion path 86a based on the simulation results. The motion path 86a is the path of the tool tip of the robot device 3. The display control unit 58 displays the motion path 86a overlaid on the images of the robot device model 3M and the workpiece models 65M and 66M.

[0048] Thus, the operator can manipulate the input unit 51 to change the position and posture of the robot model 1M in the image 81 to set teaching points 89a~89h. Alternatively, the operator can pre-create an action program 41 that determines the teaching points and input it into the simulation device 4. The simulation execution unit 56 simulates the actions of the robot model 1M according to the action program 41 and calculates the action path 86a. Then, in addition to displaying the images of the robot device model 3M and the workpiece models 65M and 66M, the display control unit 58 can also display the action path 86a.

[0049] The operator checks the status of the mobile robot device model 3M in image 81. Then, the operator can adjust the position of the teaching point and the robot's posture at the teaching point. Once it is confirmed that the robot device model 3M is driven in the desired state, the motion information setting unit 57 sets the position of the teaching point and the robot's posture at the teaching point as motion information in the motion program. That is, the motion information setting unit 57 can generate the motion program.

[0050] Incidentally, the positions of robot model 1M and workpiece models 65M and 66M in the virtual space are specified using design values ​​as expected values. For example, operators can use robot coordinate system 71 to specify the positions of robot model 1M and workpiece models 65M and 66M as expected values.

[0051] Therefore, the actual placement of robot 1 and the positions of workpieces 65 and 66 fixed to the platform 69 will mostly deviate from the expected design values. As a result, the positions of workpieces 65 and 66 relative to the placement of robot 1 will mostly deviate from the expected positions. In the simulation, it is difficult to estimate the correct positions of workpiece models 65M and 66M relative to robot device model 3M. Thus, errors will occur in the placement of various devices and components in the simulation.

[0052] The first simulation device 4 implements control to capture images of the actual workpieces 65 and 66 using the vision sensor 30, and displays the three-dimensional position information of the surface of the workpiece 65 in the simulated image in a manner corresponding to the position of the actual workpiece 65. The three-dimensional position information of the surface of the workpiece 65 is displayed in a way that the deviation of the configuration position of the workpiece 65 from the setting position of the robot 1 can be known.

[0053] Figure 4 shows a perspective view of the robot and the workpiece when the vision sensor captures an image of the actual workpiece. Referring to Figures 2 and 4, in this embodiment, the vision sensor 30 is mounted on the robot 1 when generating the motion program. The robot device 3 removes the vision sensor 30 during actual welding at the implementation point. The vision sensor 30 can be fixed at any position on the robot 1. In this embodiment, the vision sensor 30 is fixed to the flange 16 via a support member. The vision sensor 30 is supported on the robot 1 in a manner that changes in position and posture.

[0054] The operator fixes workpieces 65 and 66, which serve as references, onto the stand 69. Workpieces 65 and 66, chosen as references, can be selected based on their small manufacturing tolerances. Furthermore, the operator fixes workpieces 65 and 66 at the reference positions on the stand 69. It is preferable to fix workpieces 65 and 66 in a manner that minimizes the positional error of workpieces 65 and 66 on the stand 69.

[0055] In the robot device 3, a sensor coordinate system 73 is set for the vision sensor 30. The sensor coordinate system 73 is a coordinate system whose origin is fixed at any position of the vision sensor 30. The position and orientation of the sensor coordinate system 73 change together with the vision sensor 30. In this embodiment, the sensor coordinate system 73 is set such that the Z-axis is parallel to the optical axis of the camera included in the vision sensor 30.

[0056] The vision sensor 30 can capture images within the camera range 35. In this embodiment, the vision sensor 30 captures images of the workpieces 65 and 66 from above. The vision sensor 30 also captures images of the surface 65b of the workpiece 65, which is opposite to the vision sensor 30. The position information generation unit 59 of the processing unit 54 can set a plurality of three-dimensional points on the surface 65b of the workpiece 65, which is located within the focus range of the vision sensor 30 within the camera range 35, based on the output of the vision sensor 30.

[0057] The position information generation unit 59 calculates the distance from the vision sensor 30 to a point on the surface of the object based on the parallax between the image captured by the first camera 31 and the image captured by the second camera 32. This calculation is performed on the entire pixel area of ​​the image obtained by the first camera 31 to obtain a distance image. For each point in the distance image, the position information generation unit 59 obtains three-dimensional points as coordinate values ​​on an arbitrary coordinate system from the position information of the vision sensor. In this example, the position information generation unit 59 sets the three-dimensional points in the sensor coordinate system 73.

[0058] If the position and posture of robot 1 change, the position and posture of sensor coordinate system 73 will also change. The simulation device 4 of this embodiment corrects the position and posture of robot 1 so that the coordinate values ​​of the three-dimensional points of sensor coordinate system 73 can be converted into the coordinate values ​​of robot coordinate system 71. That is, correction is performed based on the output of vision sensor 30 so that the coordinate values ​​of robot coordinate system 71 of the three-dimensional points set on the surface 65b of workpiece 65 can be calculated.

[0059] The position information generation unit 59 can generate three-dimensional position information of the object's surface in the form of a three-dimensional map. The three-dimensional map represents the object's surface position information as a set of coordinate values ​​(x, y, z) of three-dimensional points on the object's surface. These coordinate values ​​can be represented using any coordinate system, such as a sensor coordinate system or a robot coordinate system. Alternatively, the position information generation unit 59 can generate three-dimensional position information using a distance image. A distance image represents the object's surface position information using a two-dimensional image. In the distance image, the distance from the vision sensor 30 to the three-dimensional points can be represented by the density or color of each pixel.

[0060] In this embodiment, the position information generation unit 59 is disposed within the processing unit 54 of the computing processing device 24, but it is not limited to this embodiment. The position information generation unit can also be disposed inside the vision sensor. That is, if the vision sensor includes a computing processing device with a processor such as a CPU, the processor of the vision sensor can function as the position information generation unit. In this case, three-dimensional position information is output from the vision sensor.

[0061] In this embodiment, the workpiece 65 has a shape larger than the imaging range when the vision sensor 30 is positioned at the camera location. Since the workpiece 65 is relatively large relative to the imaging range 35 of the vision sensor 30, it is impossible to capture the entire surface 65b of the workpiece 65 in a single shot. Therefore, in this embodiment, the robot 1 changes its position and posture, thereby changing the position of the vision sensor 30, and captures the entire surface 65b of the workpiece 65 multiple times.

[0062] As indicated by arrow 105, the position of the camera range 35 on the workpiece 65 is changed, and multiple shots are taken. At this time, the shots are taken in a manner that overlaps a portion of the camera range 35 from the previous shot with a portion of the camera range 35 in the current shot. That is, the position of the camera range 35 is gradually shifted while shooting, so that the same portion of the workpiece 65 is captured in both the previous and current shots. The three-dimensional position information obtained from the previous shot and the three-dimensional position information obtained from the current shot also contain the common portion.

[0063] The position information generation unit 59 combines multiple three-dimensional position information obtained from the output of the vision sensor 30 in a manner where the common parts of the three-dimensional position information overlap. The position information generation unit 59 generates three-dimensional position information for the entire surface 65b of the workpiece 65. Specifically, the position information generation unit 59 generates three-dimensional position information for the surface 65b visible from the vision sensor 30. By performing this control, the vision sensor 30, whose imaging range is relatively small compared to the workpiece size, can be used to obtain the overall three-dimensional position information of the workpiece 65.

[0064] Regarding the movement of the vision sensor, motion programs for the robot device can be pre-programmed to automatically drive the robot to perform multiple camera shots. Alternatively, operators can manually change the robot's position and posture to perform multiple camera shots. For example, operators can change the robot's position and posture by operating a teaching control panel. Alternatively, force sensors can be installed on the robot for direct teaching, and control can be implemented to change the robot's position and posture by following the external forces applied to the robot. Operators can also change the robot's position and posture by directly pushing or pulling the robot, using force sensors.

[0065] Figure 5 shows an image displaying a group of three-dimensional points acquired by a vision sensor as three-dimensional position information. The display control unit 58 uses three-dimensional points 87a to represent the three-dimensional position information of the surface 65b of the workpiece 65. In image 82, in addition to the plurality of three-dimensional points 87a, a group 87 of three-dimensional points surrounding the three-dimensional points 87a is represented by a dashed line. The display control unit 58 can display the three-dimensional points 87a based on the coordinate values ​​of the three-dimensional points 87a acquired by the position information generation unit 59. The three-dimensional points 87a correspond to the actual position of the surface 65b of the workpiece 65. The three-dimensional points 87a are positioned relative to the actual position of the robot.

[0066] In this example, the display control unit 58 displays the motion path 86a calculated through simulation. It can be seen that the motion path 86a deviates from the position of the point group 87 corresponding to the surface 65b of the workpiece 65. The operator can correct the positions of each teaching point 89a~89h and the robot's posture at each teaching point 89a~89h to align them with the position and posture of the point group 87. The operator can operate the input unit 51 to change the position and posture of the robot model 1M while simultaneously correcting the teaching points.

[0067] For example, the operator can operate the input unit 51 to modify the teaching point 89b corresponding to the welding point 68a to the teaching point 90b. The teaching point 89e corresponding to the welding point 68b can be modified to the teaching point 90e. Similarly, the teaching point 89g corresponding to the welding point 68c can be modified to the teaching point 90g. Other teaching points 89a, 89c, 89d, 89f, and 89h can also be modified according to the position and posture of the point group 87.

[0068] Thus, in the simulation device of this embodiment, the three-dimensional position information of the actual workpiece can be overlaid on the images of the simulated robot model and the workpiece model for display. Therefore, deviations from the teaching point set on the actual workpiece can be easily identified. Then, the simulation device can be used to correct the position of the teaching point and the robot's posture at the teaching point.

[0069] The motion information setting unit 57 can modify the motion information in the motion program 41. That is, the motion information setting unit 57 can modify the position of the teaching point and the robot's posture at the teaching point. In this way, the motion information setting unit 57 can modify the motion program based on the modified teaching point.

[0070] Furthermore, in addition to modifying the teaching points in the motion program, operators can also modify the actual robot's setup position, the setup position of the peripheral machines supporting the workpiece, or the position of the workpiece within the peripheral machines. For example, operators can also modify the setup position of the platform 69 that fixes the workpieces 65 and 66, or the position on the platform 69 that serves as the reference for the workpieces 65 and 66.

[0071] Figure 6 shows an image displaying the robot device model, workpiece model, and a group of three-dimensional points according to this embodiment. The operator can select the information to be displayed on the display unit 52 from the robot model, motion path, and three-dimensional position information. In image 83, through the operator's operation, the robot device model 3M, the group of three-dimensional points 87a, and the workpiece models 65M and 66M are displayed.

[0072] The distance calculation unit 60 of the processing unit 54 can calculate the actual distance between any point in the workpiece models 65M and 66M contained in the simulated image and a three-dimensional point 87a. In this example, the operator specifies a point 65Ma at the corner of the surface of the workpiece model 65M and a three-dimensional point 87aa located at the corner of the point group 87. Point 65Ma and three-dimensional point 87aa correspond to each other. Point 65Ma and three-dimensional point 87aa are points corresponding to the corners of the surface 65b of the workpiece 65.

[0073] The distance calculation unit 60 obtains the coordinate values ​​of point 65Ma and the three-dimensional point 87aa. For example, it obtains the coordinate values ​​of the robot coordinate system 71. Then, the distance calculation unit 60 calculates the distance between point 65Ma and the three-dimensional point 87aa. The display control unit 58 can display the distance calculated by the distance calculation unit 60 on the image 83. In this way, the distance calculation unit 60 can calculate and display the distance between any point in the model and any point in the three-dimensional position information. The operator can obtain the distance between any two points in the simulation. For example, the operator can know how much the position of the working model relative to the robot model deviates from the actual position of the workpiece relative to the robot.

[0074] The position information generation unit in the above embodiment can obtain three-dimensional points of the surface of the platform and the mounting surface that can be seen from the vision sensor. In this embodiment, the position information generation unit excludes three-dimensional points set on the surface of the platform and the mounting surface from the three-dimensional points of the object's surface. For example, the position information generation unit can select three-dimensional points arranged on the surface of the workpiece based on a determination value of the distance calculated from the vision sensor.

[0075] The three-dimensional points displayed on the display are not limited to this form. For example, it is also acceptable to display three-dimensional points arranged on the surface of the platform in a simulated image. That is, it is also acceptable to display three-dimensional points set on the surface of the robot's peripheral machinery in a simulated image.

[0076] Furthermore, while the above embodiment obtains three-dimensional position information of the upper surface of workpiece 65, it is not limited to this embodiment. The three-dimensional position information of the surface can be obtained by changing the position and posture of robot 1 and photographing workpieces 65 and 66 from various directions. For example, obtaining the positions of three-dimensional points set on the sides of workpieces 65 and 66 is also acceptable. Then, the display control unit 58 can display the three-dimensional points set on the sides of workpieces 65 and 66 on the display unit.

[0077] In the above-described embodiment, since the workpiece is relatively large relative to the camera range of the vision sensor, multiple camera shots can be taken by changing the robot's position and posture, but this is not limited to this embodiment. When the workpiece is relatively small relative to the camera range, the vision sensor can take a single shot to capture the entire workpiece. In this case, it is also acceptable to fix the vision sensor to a stand or similar device. For example, it is also acceptable to fix the vision sensor above the workpiece and obtain the overall three-dimensional position information of the upper surface of the workpiece with a single shot.

[0078] Figure 7 shows a block diagram of the second robot system of this embodiment. The second robot system includes a first robot device 3 and a second simulation device 8. The second simulation device 8 detects the position of the actual workpiece based on the three-dimensional position information obtained from the output of the vision sensor 30. Then, the second simulation device 8 automatically corrects the motion information contained in the motion program to correspond to the position of the actual workpiece.

[0079] The second simulation device 8 has a processing unit 63 whose configuration differs from that of the first simulation device 4's processing unit 54. The processing unit 63 of the second simulation device 8 includes a position detection unit 61, which detects the actual position of the workpiece by matching the workpiece's reference data with the three-dimensional position information of the workpiece's surface. The position detection unit 61 is equivalent to a processor driven according to a simulation program. The processor functions as the position detection unit 61 by implementing controls determined in the program.

[0080] Furthermore, the motion information setting unit 57 modifies the motion information, which includes the teaching point information, in the motion program. The motion information setting unit 57 modifies the motion program according to the position of the simulated workpiece model and the actual position of the workpiece detected by the position detector 61, so that it corresponds to the position of the actual workpiece.

[0081] Figure 8 shows an image of a group of three-dimensional points obtained from the output of the vision sensor. Referring to Figures 7 and 8, image 84 displays a group of three-dimensional points 87a as three-dimensional position information, obtained by the vision sensor 30 capturing images of workpieces 65 and 66. Image 84 shows the robot device model 3M, workpiece models 65M and 66M, the group of three-dimensional points 87a, and the motion path 86a. The motion path 86a is a path generated by simulation based on the positions of workpiece models 65M and 66M. The group of points 87 corresponding to the position of the actual workpiece 65's surface 65b deviates from the upper surface of the workpiece model 65M. As a result, the motion path 86a also deviates from the position of the group of three-dimensional points 87a.

[0082] The position detection unit 61 detects the actual position of the workpiece corresponding to the point group 87 by matching the reference data of the workpiece 65. In the matching process, the reference data of the workpiece can be the three-dimensional shape data of the workpiece output from a CAD device or the like.

[0083] As a matching process, for example, the position detection unit 61 generates information of a plurality of meshes from a number of three-dimensional points 87a. Alternatively, the information of these meshes can be matched with information of meshes generated from the three-dimensional shape data of the workpiece, thereby detecting the actual position of the workpiece.

[0084] Alternatively, a characteristic part of the workpiece with a distinctive shape can be selected, and the 3D shape data of the characteristic part can be matched with a group of 3D points. The characteristic part within the group of 3D points can be identified, and its position detected. The actual position of the workpiece can be detected based on the positions of multiple characteristic parts. For example, a protrusion 65a of workpiece 65 can be selected as a characteristic part. By using the 3D shape data of protrusion 65a as reference data, the positions of multiple protrusions within the group of 3D points can be identified. Alternatively, the contour of the workpiece can be estimated from the group of 3D points and matched with the contour in the workpiece's 3D shape data.

[0085] In this embodiment, a coordinate system (user coordinate system) is pre-set to determine the position and orientation of the workpiece: the workpiece surface has an origin, and the workpiece has a pre-determined orientation. In this embodiment, this coordinate system is called the workpiece coordinate system. The workpiece coordinate system is a coordinate system that is fixed to the workpiece.

[0086] A workpiece coordinate system 74a is set on the upper surface of the workpiece model 65M. In this embodiment, the origin of the workpiece coordinate system is set at one corner of the surface 65b of the workpiece 65. Then, the X-axis and Y-axis of the workpiece coordinate system are set along the extension direction of the edge of the surface 65b. For example, the position of the workpiece, which corresponds to the position of the origin of the workpiece coordinate system, can be represented by the coordinate values ​​(x, y, z) of the robot coordinate system 71. Furthermore, the posture of the workpiece, which corresponds to the orientation of the workpiece coordinate system, can be represented by the coordinate values ​​(w, p, r) of the workpiece coordinate system 71.

[0087] The motion information determined in the motion program can be specified by the following: the position and orientation of the workpiece coordinate system represented by the robot coordinate system, and the position of the teaching point represented by the workpiece coordinate system and the robot's orientation at the teaching point.

[0088] The position detection unit 61 detects the position of the workpiece 65 corresponding to the point group 87 by matching the point group 87 containing three-dimensional points 87a with the reference data of the workpiece. The position detection unit 61 calculates the point corresponding to the corner of the upper surface of the workpiece and sets the workpiece coordinate system 74b with the point as the origin. Furthermore, the workpiece coordinate system 74b is set with the X-axis and Y-axis extending along the edge of the point group 87. The position and orientation of the workpiece coordinate system 74b corresponding to the actual position and orientation of the workpiece 65 can be calculated by the robot coordinate system 71. Furthermore, the display control unit 58 can display the workpiece coordinate system 74b set by the position detection unit 61.

[0089] The motion information setting unit 57 performs control and, within the motion program, corrects the position and orientation of the workpiece coordinate system 74a set in the workpiece model 65M during simulation, so that it corresponds to the position and orientation of the workpiece coordinate system 74b calculated using the point group 87 of the three-dimensional points 87a. In this way, the motion information setting unit 57 can correct the motion information within the motion program. Here, the position and orientation of the workpiece coordinate system in the motion information can be corrected.

[0090] Figure 9 is an image showing the corrected motion path, robot model, workpiece model, and point group of 3D points in the motion program. The positions and postures of each teach point and the robot's posture at the teach points are corrected by adjusting the position and posture of the workpiece coordinate system, which serves as motion information. Each teach point 89b, 89e, and 89g moves as indicated by arrow 106 and is set as the corrected teach points 90b, 90e, and 90g. Furthermore, the simulation execution unit 56 performs a simulation, thereby displaying the corrected motion path 86b.

[0091] The second simulation device can automatically correct deviations in the relative position and posture of workpieces 65 and 66 relative to robot 1. More specifically, it can automatically correct the motion information contained in the motion program based on the three-dimensional position information obtained from the output of the vision sensor, so as to correspond to the actual position of the workpiece.

[0092] Furthermore, by setting a workpiece coordinate system on the workpiece surface, and using the workpiece coordinate system to determine the position of the teaching point and the robot's posture at the teaching point, the control of correcting the motion program can be simplified. In the motion program, there is no need to change the position of the teaching point represented by the workpiece coordinate system and the robot's posture at the teaching point; the motion information can be corrected simply by changing the position and posture of the workpiece coordinate system.

[0093] In the above-described embodiment, the motion information setting unit implements control to correct the position and orientation of the workpiece coordinate system in the motion program, but is not limited to this embodiment. The motion information setting unit can arbitrarily control and correct the motion information in the motion program.

[0094] For example, the coordinate values ​​of the robot coordinate system can be used to specify the position of the teach point, which serves as motion information in the motion program, and the robot's posture at the teach point. In this case, the motion information setting unit calculates the relative position and relative posture of the workpiece coordinate system, which is a group of points set in three dimensions, relative to the workpiece coordinate system set on the workpiece model. The direction and amount of movement of the workpiece coordinate system correspond to the error in the position and posture of the workpiece. Therefore, the motion information setting unit can correct the position of the teach point represented by the robot coordinate system and the robot's posture at the teach point based on the relative position and relative posture of the workpiece coordinate system.

[0095] Therefore, the motion information setting unit 57 can have the function of converting the coordinate value of a point in the simulation from one coordinate system to the coordinate value of another coordinate system. For example, based on the robot's position and posture, the coordinate value of a point represented by the workpiece coordinate system set on the workpiece model can be converted into the coordinate value represented by the robot coordinate system. Alternatively, a coordinate value represented by the robot coordinate system can be converted into a coordinate value represented by the workpiece coordinate system. Alternatively, based on the robot's position and posture, the coordinate value of the sensor coordinate system can be converted into the coordinate value of the robot coordinate system.

[0096] In the above embodiment, the welding gun is mounted on the robot as a working tool, but this embodiment is not limited to this one. Any working tool suitable for the operation performed by the robot device can be used. The simulation device of this embodiment is applicable to robot devices performing any operation. Next, an example of a robot device performing operations other than spot welding will be described.

[0097] Figure 10 shows a perspective view of the second robot device of this embodiment. The second robot device 9 has a hand 6 mounted on the robot 1 as a working tool. The hand 6 is configured to grasp or release a workpiece 67. The second robot device 9 has a conveyor 7 as a peripheral machine of the robot 1 for transporting the workpiece 67. The conveyor 7 transports the workpiece 67 in the direction indicated by arrow 107. The second robot device 9 performs the operation of transporting the workpiece 67 from a predetermined position and placing it on the conveyor 7.

[0098] In the second robot device 9, errors sometimes occur in the positioning of robot 1 and conveyor 7. In the simulation device of the second robot device 9, the reference workpiece 67 is also placed at the reference position on conveyor 7. Three-dimensional position information of the workpiece surface is obtained by photographing the workpiece using a vision sensor installed on the robot, etc. In the simulation device, in addition to displaying images of the robot device model and the workpiece model, three-dimensional position information can also be displayed. Operators can correct the motion information of the motion program in the simulation device. Furthermore, the simulation device can correct the motion information contained in the motion program based on the three-dimensional position information obtained from the output of the vision sensor. For example, the teaching point used by the robot device to release the workpiece on the conveyor can be corrected.

[0099] The vision sensor in this embodiment is a stereo camera that includes a plurality of two-dimensional cameras, but is not limited to this embodiment. As a vision sensor, any sensor capable of acquiring three-dimensional position information of the surface of an object can be used. For example, a TOF (Time of Flight) camera can be used, which acquires position information of three-dimensional points based on the time of flight of light. Alternatively, a device that uses a laser rangefinder to scan a predetermined area to detect the position of the surface of an object can also be used as a vision sensor.

[0100] The processing unit of the simulation device in this embodiment is composed of a computational processing unit independent of the robot's control device, but it is not limited to this embodiment. It is permissible for the robot's control device to have the functions of a simulation device. That is, it is permissible for the processor of the control device's computational processing unit to function as a processing unit. Furthermore, when the teaching operation panel includes a computational processing unit with a processor, it is permissible for the teaching operation panel to have the functions of a simulation device. That is, it is permissible for the processor of the teaching operation panel to function as a processing unit.

[0101] The above embodiments can be appropriately combined. In each of the above controls, the order of steps can be appropriately changed without altering the function or effect. In the above figures, the same or equivalent parts are marked with the same symbol. Furthermore, the above embodiments are illustrative and are not intended to limit the invention. Also, the embodiments include modifications to the embodiments shown in the claims.

[0102] 1: Robot 1M: Robot Model 2: Control device 3,9: Robotic Device 3: First Robot Device 3M: Robotic Device Model 4,8: Simulation device 4: First simulation device 5: Welding gun 5M: Welding Gun Model 6: Hands 7: Conveyor 8: Second simulation device 9: Second Robot Device 11: Upper arm 12: Lower arm 13: Rotary base 14: Base 15: Wrist 16: Flange 21: Robot drive unit 22: Tool drive device 23: Rotation Angle Detector 24: Computational processing unit 30: Vision Sensor 31: The First Camera 32: Second Camera 33: Projector 35: Camera range 41: Action Programming 42, 53: Memory Department 43: Motion Control Department 44: Robot Drive Department 45: Operation Tool Drive Unit 49: Instruction of the operation panel 49a, 51: Input Section 49b, 52: Display section 50: Three-dimensional shape data 54, 63: Processing Department 55: Model Generation Department 56: Simulation Execution Department 57: Action Information Setting Department 58: Display Control Unit 59: Location Information Generation Department 60: Distance Calculation Unit 61: Position Detection Department 65, 66, 67: Workpiece 65: Workpiece No. 1 65a:convex part 65b: Surface 65M, 66M: Workpiece Model 65Ma: point 66: Second workpiece 68a~68c: Welding points 69: Erection 71: Robot Coordinate System 72: Tool Coordinate System 73: Sensor Coordinate System 74a, 74b: Workpiece coordinate system 81~84: Images 86a, 86b: Action paths 87: Group 87a, 87aa: 3D points 89a~89h,90b,90e,90g: teaching points 101~107: Arrows CAD: Computer-Aided Design CPU: Central Processing Unit RAM: Random Access Memory ROM: Read-only memory Time of Flight (TOF) X, Y, Z: Axes

Claims

1. A simulation device for simulating the actions of a robot device equipped with a robot, the simulation device comprising: a model generation unit that generates a simulated robot device model and a workpiece model based on three-dimensional shape data of the robot device and three-dimensional shape data of a workpiece; a display unit that displays an image of the robot device model and an image of the workpiece model; an action information setting unit that generates an action program including teaching points; a simulation execution unit that simulates the actions of the robot device based on the teaching points to estimate the action path of the robot; and a position information generation unit that generates three-dimensional position information of the surface of the workpiece based on the output of a vision sensor that captures an actual image of the workpiece; the display unit displays the three-dimensional position information of the surface of the workpiece overlaid on the image of the robot device model, the image of the workpiece model, and the action path of the robot.

2. The simulation apparatus of claim 1, wherein the aforementioned three-dimensional position information is constituted by a plurality of three-dimensional points set on the surface of the aforementioned workpiece, and the aforementioned simulation apparatus further comprises a distance calculation unit for calculating the distance between a point in the aforementioned workpiece model and a three-dimensional point.

3. The simulation apparatus of claim 1 or 2, wherein the aforementioned workpiece has a shape larger than the camera range when the vision sensor is positioned at the camera position, the aforementioned position information generation unit generates the aforementioned three-dimensional position information of the surface of the aforementioned workpiece by generating a plurality of the aforementioned three-dimensional position information obtained by a plurality of cameras with varying camera positions, and combining the plurality of the aforementioned three-dimensional position information.

4. A simulation device for simulating the actions of a robot device equipped with a robot, the simulation device comprising: a model generation unit that generates a simulated robot device model and a workpiece model based on three-dimensional shape data of the robot device and three-dimensional shape data of a workpiece; a simulation execution unit that simulates the actions of the robot device; an action information setting unit that sets action information determined in an action program based on the simulation of the robot device's actions; a position information generation unit that generates three-dimensional position information of the surface of the workpiece based on the output of a vision sensor that captures an actual image of the workpiece; and a position detection unit that detects the actual position of the workpiece by matching reference data of the workpiece with the three-dimensional position information of the workpiece's surface; the action information setting unit corrects the action information contained in the action program based on the position of the simulated workpiece model and the actual position of the workpiece detected by the position detection unit, so that it corresponds to the position of the actual workpiece.

5. The simulation device of claim 4, wherein a workpiece coordinate system is preset, the workpiece coordinate system having an origin for the workpiece, and the motion information determined in the motion program includes: the position and orientation of the workpiece coordinate system; and the position of the teaching point determined by the workpiece coordinate system and the orientation of the robot at the teaching point; the motion information setting unit corrects the position and orientation of the workpiece coordinate system set in the workpiece model in the simulation so that it corresponds to the position and orientation of the workpiece coordinate system of the actual workpiece calculated from the three-dimensional position information.

6. The simulation device as requested in item 4 or 5, wherein the aforementioned motion information setting unit has the function of converting the coordinate value of a point in the simulation from one coordinate system to the coordinate value of another coordinate system.

Images