Operation method of robot operation system

Abstract

To provide an operation method of a robot operation system allowing for operating a robot by easy operation.SOLUTION: When a three-dimensional model displayed on a first display unit 7 is moved to a predetermined position, in response to the move, a robot 2 is caused to move to a predetermined position. When displaying the three-dimensional model on the first display unit 7, it is determined from which position an image of the robot 2 to be displayed on the first display unit 7 is captured; a three-dimensional model of the robot 2 viewed from the determined position is generated; and the generated three-dimensional model of the robot 2 is superimposed onto the image of the robot 2 and displayed on the first display unit 7.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method of operating a robotic manipulation system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, when an industrial robot is used to perform an operation such as welding on a workpiece, a teaching process is carried out as a preparation for the robot to memorize in advance the operations required for moving the robot to a destination.

[0003] Generally, two teaching methods are known: an online teaching method using a teach pendant (see FIG. 1 of Patent Document 1) and an offline teaching method (Patent Document 2).

[0004] However, all of these teaching methods have the problem that they are dangerous because they ultimately require work to be done at the actual work site.

[0005] In order to solve such problems, the invention of Patent Document 3 has been proposed. The invention described in Patent Document 3 involves capturing images of the robot's work site with multiple multi-viewpoint image capturing cameras, synthesizing the captured images to generate an arbitrary viewpoint image, and using the generated arbitrary viewpoint image to instruct the robot's operation. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2013-132728 A [Patent Document 2] Japanese Patent Application Publication No. 9-179624 [Patent Document 3] JP 2011-251395 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, although the above-mentioned inventions are easy to operate, it takes a very long time to synthesize the images, and even if they are successfully synthesized, the resulting images are often unclear, making them unsuitable for practical use.

[0008] In view of the above problems, the present invention aims to provide a method for operating a robot operation system that enables a user to operate a robot through simple operations while viewing clear images obtained in a short processing time. [Means for solving the problem]

[0009] The above object of the present invention can be achieved by the following means. Note that the parentheses indicate reference symbols of the embodiments described below, but the present invention is not limited thereto.

[0010] The method for operating the robot operation system according to claim 1 comprises: A robot (e.g., robot 2 shown in FIG. 2) that performs a predetermined processing operation on a workpiece (e.g., workpiece W shown in FIG. 2); A single first camera (e.g., the first camera 34 shown in FIG. 1) capable of capturing images of the robot (e.g., the robot 2 shown in FIG. 2) and the workpiece (e.g., the workpiece W shown in FIG. 2); a three-dimensional model storage unit (e.g., the ROM 52 shown in FIG. 2) for storing a three-dimensional model (e.g., the three-dimensional model 9 shown in FIG. 5(a) and the three-dimensional model 9A shown in FIG. 5(b)) of the robot (e.g., the robot 2 shown in FIG. 2); a first display unit (e.g., the first display unit 7 shown in FIG. 2) that displays an image (e.g., an image P of a work site shown in FIG. 4(b)) of the robot (e.g., the robot 2 shown in FIG. 2) captured by the first camera (e.g., the first camera 34 shown in FIG. 1) superimposed on a three-dimensional model (e.g., the three-dimensional model 9 shown in FIG. 5(a) and the three-dimensional model 9A shown in FIG. 5(b)); a control unit that, when the three-dimensional model (e.g., the three-dimensional model 9 shown in FIG. 5(a) or the three-dimensional model 9A shown in FIG. 5(b)) displayed on the first display unit (e.g., the first display unit 7 shown in FIG. 2) moves to a predetermined position, moves the robot (e.g., the robot 2 shown in FIG. 2) to a predetermined position accordingly; A method for operating a robot manipulation system comprising: a step of determining from which position an image (e.g., an image P of a work site captured by the first camera 34 shown in FIG. 4(b)) of the robot (e.g., the robot 2 shown in FIG. 2) to be displayed on the first display unit (e.g., the first display unit 7 shown in FIG. 2) has been captured (e.g., step S105 shown in FIG. 3); a step of generating a three-dimensional model of the robot (e.g., the robot 2 shown in FIG. 2) viewed from the determined position (e.g., a three-dimensional model 9A of the robot 2 viewed from the first camera 34 shown in FIG. 5(b)) (e.g., step S3 shown in FIG. 3); a step of superimposing the generated three-dimensional model (e.g., three-dimensional model 9A of robot 2 seen from first camera 34 shown in FIG. 5(b)) of the robot (e.g., robot 2 shown in FIG. 2) on an image (e.g., image P of the work site captured by first camera 34 shown in FIG. 4(b)) of the robot (e.g., robot 2 shown in FIG. 2) and displaying it on the first display unit (e.g., first display unit 7 shown in FIG. 2) (e.g., first display unit 7 shown in FIG. 6) (e.g., step S107 shown in FIG. 3); The present invention is characterized in that it comprises:

[0011] The method for operating a robot operation system according to claim 2 is the method for operating a robot operation system according to claim 1, further comprising the steps of: The first cameras (e.g., first cameras 31 to 39 shown in FIG. 2) are a plurality of first cameras (e.g., first cameras 31 to 39 shown in FIG. 2) fixedly arranged so as to be able to image the robot (e.g., robot 2 shown in FIG. 2) and the workpiece (e.g., workpiece W shown in FIG. 2) from different positions, The image (e.g., image P of the work site shown in FIG. 4(b)) of the robot (e.g., robot 2 shown in FIG. 2) displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) is characterized in that it is an image (e.g., image P of the work site shown in FIG. 4(b)) of one first camera (e.g., first camera 34 shown in FIG. 1) selected from the plurality of first cameras (e.g., first cameras 31 to 39 shown in FIG. 2).

[0012] The method for operating a robot operation system according to claim 3 is the method for operating a robot operation system according to claim 2, further comprising the steps of: Furthermore, the robot operation system includes a first camera position storage unit (e.g., RAM 53 shown in FIG. 2) that stores the position from which the first camera (e.g., first cameras 31 to 39 shown in FIG. 2) images the robot and the workpiece, When determining from which position the image (e.g., image P of the work site shown in FIG. 4(b)) of the robot (e.g., robot 2 shown in FIG. 2) to be displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) was captured, the position from which the selected one of the first cameras (e.g., first cameras 31-39 shown in FIG. 2) that captured the image (e.g., image P of the work site shown in FIG. 4(b)) captured the image of the robot and the workpiece is obtained from the first camera position memory unit (e.g., RAM 53 shown in FIG. 2).

[0013] The method for operating a robot operation system according to claim 4 is the method for operating a robot operation system according to claim 1, further comprising the steps of: Further, a second camera (for example, the second camera 4 shown in FIG. 2) capable of capturing an image of the entire work site including the robot (for example, the robot 2 shown in FIG. 2) and the work (for example, the work W shown in FIG. 2); The display device further comprises a second display unit (for example, second display unit 6 shown in the figure) for displaying an image captured by the second camera in real time. Effect of the Invention

[0014] Next, the effects of the present invention will be described with reference to the drawings. Note that the reference symbols in the parentheses are those of the embodiments described below, but the present invention is not limited thereto.

[0015] According to the invention of claim 1, the position from which the work site was captured for an image (e.g., image P of the work site captured by first camera 34 shown in FIG. 4(b)) to be displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) is determined, and a three-dimensional model reproducing the robot as seen from that position in a virtual space (e.g., three-dimensional model 9A of robot 2 as seen by first camera 34 shown in FIG. 5(b)) is displayed superimposed on the image displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) (e.g., image P of the work site captured by first camera 34 shown in FIG. 6).

[0016] Then, while viewing an image (e.g., image P of the work site captured by first camera 34 shown in FIG. 6) on which the three-dimensional model (e.g., three-dimensional model 9A shown in FIG. 6) is superimposed, the worker moves the three-dimensional model (e.g., three-dimensional model 9A shown in FIG. 6) to a desired position by dragging or the like. Then, the worker checks from the image (e.g., image P of the work site captured by first camera 34 shown in FIG. 6) whether or not a problem such as interference will occur if the robot (e.g., robot 2 shown in FIG. 2) moves in the same manner as the three-dimensional model (e.g., three-dimensional model 9A shown in FIG. 6), and once it is determined that there is no problem, the worker can move the actual robot (e.g., robot 2 shown in FIG. 2). Thus, the robot can be operated by simple operations.

[0017] Here, if an image of the work site is generated according to the three-dimensional model, it is necessary to generate the image by synthesis or the like according to the position from which the three-dimensional model depicts the robot, and display it on the first display unit (for example, the first display unit 7 shown in FIG. 2). Therefore, synthesis takes time, and the image is not clear because it is a synthesis. However, according to the present invention, a three-dimensional model (for example, the three-dimensional model 9A of the robot 2 seen from the first camera 34 shown in FIG. 5(b)) is generated according to an image (for example, the image P of the work site captured by the first camera 34 shown in FIG. 4(b)). Therefore, there is no need to synthesize the images, and the processing time is shortened. In addition, the image (for example, the image P of the work site captured by the first camera 34 shown in FIG. 6) displayed on the first display unit (for example, the first display unit 7 shown in FIG. 2) is clear because it is not an image generated by synthesis or the like.

[0018] Therefore, according to the present invention, it is possible to provide an operating method for a robot operation system that can operate a robot (e.g., robot 2 shown in FIG. 2) with simple operations while viewing a clear image (e.g., image P of the work site captured by first camera 34 shown in FIG. 6) obtained in a short processing time.

[0019] According to the invention of claim 2, images captured from a plurality of positions can be displayed with a simple configuration. That is, when there is only one first camera, images captured from a plurality of positions cannot be displayed, resulting in low convenience. Therefore, a plurality of first cameras (for example, first cameras 31-39 shown in FIG. 2) are provided that are fixedly arranged so as to be able to capture images from different positions. In this way, a desired first camera (for example, first cameras 31-39 shown in FIG. 2) can be selected with a simple configuration, and an image of the selected first camera (for example, image P of the work site captured by first camera 34 shown in FIG. 4(b)) can be displayed on the first display unit (for example, first display unit 7 shown in FIG. 2).

[0020] According to the invention of claim 3, the position from which the first camera captures the robot and the workpiece can be stored in the first camera position storage unit (for example, the RAM 53 shown in FIG. 2) in advance by first performing calibration of the first camera (for example, the first camera 31-39 shown in FIG. 2). Therefore, every time an image (for example, the image P of the work site shown in FIG. 4(b)) to be displayed on the first display unit (for example, the first display unit 7 shown in FIG. 2) is selected, the position from which the selected first camera (for example, the first camera 31-39 shown in FIG. 2) that captured the image (for example, the image P of the work site shown in FIG. 4(b)) captures the robot and the workpiece can be determined by simply reading information from the first camera position storage unit (for example, the RAM 53 shown in FIG. 2), and there is no need to perform calculations each time, making the processing easier.

[0021] According to the invention of claim 4, the entire work site including the movement of the robot (for example, robot 2 shown in FIG. 2) can be viewed in real time on the second display unit (for example, second display unit 6 shown in the figure). Therefore, even if the entire work site cannot be viewed from the image (for example, image P of the work site shown in FIG. 4(b)) captured by one first camera (for example, first cameras 31-39 shown in FIG. 2), the entire work site can be confirmed on the second display unit (for example, second display unit 6 shown in the figure). [Brief description of the drawings]

[0022] [Figure 1] 1 is an overall view of a robot operation system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is an overall schematic diagram of the robot operation system according to the embodiment. [Diagram 3] FIG. 4 is a flowchart showing the operation of the robot operation system according to the embodiment. [Figure 4] (a) is the image used for calibration, and (b) is the display screen of the overlay application when the first camera is selected. [Diagram 5](a) is a display screen of the robot operation program showing the three-dimensional model before redrawing, and (b) is a display screen of the robot operation program when the three-dimensional model is redrawn to match the selected first camera. [Figure 6] 13 is a display screen of an overlay application showing a redrawn three-dimensional model overlaid on an image captured by a first selected camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of a robot manipulation system using a method for operating a robot manipulation system according to the present invention will be described in detail with reference to the drawings. In the following description, when directions such as up, down, left, and right are indicated, they refer to up, down, left, and right when viewed from the front of the figure.

[0024] <Outline of the robot operation system> First, an overview of the robot operation system 1 in this embodiment will be described. As shown in Fig. 1, in the robot operation system 1 in this embodiment, a turntable T on which a workpiece W (see Fig. 2) is placed, and a robot 2 that performs a predetermined processing operation such as welding or polishing on the workpiece W are captured by a plurality of first cameras 31-39. A worker operates the robot 2 using an operation terminal 5 (see Fig. 2) while viewing the images of the robot 2, the workpiece W, and the surrounding situation captured by the first cameras 31-39.

[0025] The configuration of the robot operation system 1 in this embodiment will be described more specifically with reference to Fig. 2. First, as shown in Fig. 2, the robot 2 is connected to an operation terminal 5 via a programmable logic controller (hereinafter referred to as PLC 29) through a network N. Meanwhile, the first cameras 31-39 are connected to the operation terminal 5 through the network N. Furthermore, the second camera 4 is also connected to the operation terminal 5 through the network N.

[0026] Furthermore, the operation terminal 5 is connected to the second display unit 6, the first display unit 7, and the robot operation screen 8 via a monitor cable such as an HDMI (registered trademark) cable.

[0027] Here, a method of moving the robot 2 by the robot operation system 1 configured as described above will be briefly described. First, when the worker selects one of the first cameras 31 to 39 on the operation terminal 5, an image P of the work site captured by the selected first camera is displayed on the first display unit 7 (see FIG. 4(b)). Next, the operation terminal 5 converts information on the position from which the work site was captured in the image P into a position and direction in the virtual space K (see FIG. 5(a)). Then, in the virtual space K, the three-dimensional model 9A is re-rendered when the image P is captured and viewed in the same direction from the same position (FIG. 5(b)). Then, the operation terminal 5 superimposes the re-rendered three-dimensional model 9A on the image P of the work site displayed on the first display unit 7 (FIG. 6). Finally, the worker drags the three-dimensional model 9A superimposed on the image P of the work site to operate the robot (not shown).

[0028] The outline of the robot operation system 1 in this embodiment has been described above. Next, each part in FIG. 2 will be described in detail.

[0029] <Robot description> The robot 2 is a multi-jointed general-purpose robot with six degrees of freedom. More specifically, as shown in Fig. 2, the robot 2 includes a base 20 fixed to the floor or the like of a work area, an arm 22 whose base end is rotatably connected to the base 20, and a work tool 24 attached to a tip 22a of the arm 22 and performing work such as welding, sealing, painting, cutting, polishing, handling, etc. on a workpiece W placed on a turntable T. Note that, in this embodiment, a six-axis multi-jointed general-purpose robot is exemplified as the robot 2, but the present invention is not limited to this and any robot may be used.

[0030] <Explanation of the first camera> The first cameras 31 to 39 are digital cameras capable of taking color images. In this embodiment, when the first cameras 31 to 39 are installed at the work site, as shown in FIG. 1, a camera installation frame 30 is installed within a safety fence D in which the robot 2 and the turntable T are installed. The camera installation frame 30 includes a first frame 30b, a second frame 30c, and a third frame 30d that are provided at a fixed interval from the right side to the left side in FIG. 1, and further includes a common horizontal member 30a that connects the first frame 30b, the second frame 30c, and the third frame 30d. As shown in FIG. 1, the first frame 30b, the second frame 30c, and the third frame 30d each include a pair of vertical members that are provided at a fixed interval in the front-rear direction in the figure, and a horizontal member that connects the upper ends of both vertical members. Furthermore, the common horizontal member 30a is provided so as to connect the centers of the horizontal members of the first frame 30b, the second frame 30c, and the third frame 30d.

[0031] The first cameras 31 and 32 are installed vertically at a distance from each other on the vertical member on the rear side of the first frame 30b shown in Fig. 1, and the first cameras 33 and 34 are installed vertically at a distance from each other on the vertical member on the front side of the figure. The first camera 36 is installed on the vertical member on the rear side of the second frame 30c shown in Fig. 1, and the first camera 37 is installed on the vertical member on the front side of the figure. Furthermore, a pair of vertical members installed at a certain distance from each other in the front-to-rear direction of the second frame 30c shown in Fig. 1 are provided with members 30c1 and 30c2 extending toward the left side in the figure, and first cameras 38 and 39 are installed at the tip portions of the members.

[0032] <Explanation of the second camera> The second camera 4 is a web camera that can move the lens direction up, down, left, and right and can enlarge and reduce the angle of view. As shown in Fig. 1, the second camera 4 is installed near the intersection of the first frame 30b and the frame member 30a, and since it is installed at a high position, it can capture an image of the entire work site.

[0033] 2, the second camera 4 is connected to the operation terminal 5 via a dedicated line L separate from the network N of the first cameras 31-39. This allows the second camera 4 to transmit captured images without delay, regardless of the traffic conditions of the network N connecting the first cameras 31-39 and the operation terminal 5.

[0034] Incidentally, the image captured by the second camera 4 is displayed on the second display unit 6 (see FIG. 2) of the operation terminal 5. Note that a terminal other than the operation terminal 5 may be prepared and connected to the second camera 4 and the second display unit 6. In this way, the image captured by the second camera 4 can be displayed on the second display unit 6 without delay even while the operation terminal 5 is performing the process described below (see FIG. 3).

[0035] <Operation terminal description> The operation terminal 5 is an information processing device, for example, a PC (Personal Computer). Specifically, as shown in Fig. 2, the operation terminal 5 is composed of a CPU 50, an input unit 51 that can input predetermined data from the outside to the operation terminal 5 using a mouse, a keyboard, a touch panel, or a USB connector, a Lightning connector, etc., a ROM 52 consisting of a writable flash ROM or the like that stores predetermined application programs, etc., a RAM 53 that functions as a working area, a buffer memory, etc., a display unit 54 that can display various images, and a communication unit 55 that can be connected to a network N by a communication means such as a wireless LAN, a wired LAN, or a dial-up.

[0036] The ROM 52 stores an overlay application AP1 (see FIGS. 4(b) and 6) that overlays and displays an image P (see FIG. 4(b)) of the work site and the three-dimensional models 9, 9a (see FIG. 5(a)), a robot operation program AP2 (see FIG. 5) that the worker uses to move the robot 2, and the three-dimensional models 9, 9a (see FIG. 5(a)) used in the program. The RAM 53 also stores the positions from which the first cameras 31-39 capture the image of the robot 2 (see step S102 in FIG. 3).

[0037] <Display section explanation> The second display unit 6, the first display unit 7, and the robot operation screen 8 are connected to the operation terminal 5, as shown in Fig. 2. The second display unit 6, the first display unit 7, and the robot operation screen 8 are each an independent liquid crystal monitor.

[0038] The robot operation screen 8 displays the robot operation program AP2 (see FIG. 5). The robot operation program AP2 displays three-dimensional models 9, 9a (see FIG. 5(a)) used for operating the robot.

[0039] The first display unit 7 displays the overlay application AP1 (see FIG. 4(b) and FIG. 6). The overlay application AP1 overlays and displays the image P received from the first camera 31-39, the three-dimensional model 9 of the robot 2 displayed on the robot operation program AP2 (robot operation screen 8), and the three-dimensional model 9a of the turntable T (see FIG. 6).

[0040] The second display unit 6 displays an image overlooking the entire work site, which the operation terminal 5 receives from the second camera 4 (not shown).

[0041] <Explanation of usage example of robot operation system> An example of how the robot operation system 1 configured as above is used will be specifically described with reference to Fig. 3. Note that the following is merely an example, and the present invention is not limited to this.

[0042] 2 (hereinafter, may be abbreviated as operation terminal 5) receives instructions from an operator via input unit 51 and starts up overlay application AP1 and robot operation program AP2. The operation terminal 5 displays the started overlay application AP1 on the first display unit 7 (see FIG. 4(b)), and displays the robot operation program AP2 on the robot operation screen 8 (see FIG. 5).

[0043] Next, in the robot operation program AP2, the operation terminal 5 reads out the three-dimensional models 9, 9a from the ROM 52 (step S1). After that, the operation terminal 5 puts the robot operation program AP2 on standby until it receives information from the overlay application AP1 (step S2).

[0044] On the other hand, the operation terminal 5 starts receiving images captured by the first cameras 31 to 39 and the second camera 4 (see FIG. 2) shown in FIG. 2 in the overlay application AP1 (step S101).

[0045] Next, in the overlay application AP1, the operation terminal 5 calibrates the first cameras 31-39 based on the images received from the first cameras 31-39 shown in FIG. 2, and stores the positions from which the first cameras 31-39 are capturing the images of the work site (step S102). A specific process will be described with reference to FIG. 4(a). FIG. 4(a) is an example of an image P of the work site received by the operation terminal 5 from the first cameras 31-39. As shown in FIG. 4(a), a plurality of marks M1 are provided on the turntable T, and a mark M2 is provided on the tip 22a of the arm 22 of the robot 2. The operation terminal 5 determines information on the position from which the work site is imaged for each of the first cameras 31-39 (see FIG. 1) (for example, three-dimensional coordinates indicating the positions of the first cameras 31-39 within the work site and three-dimensional vectors indicating the line-of-sight directions of the first cameras 31-39) based on the appearance of the marks M1 and M2 in the image P of the work site captured by each of the first cameras 31-39 (see FIG. 1), and stores the information in the RAM 53. Note that determining the position from which an image was captured based on the captured marks is a conventionally known technique, and therefore a detailed description thereof will be omitted.

[0046] Next, the operation terminal 5 waits for the worker to select one of the first cameras 31-39 in the overlay application AP1 (step S103). More specifically, the worker first clicks the "Find Camera" button 70 shown in FIG. 4(b) using the input unit 51 of the operation terminal 5. This causes the operation terminal 5 to display the first cameras 31-39 (see FIG. 1) connected to the operation terminal 5 in a first drop-down menu 71. The worker selects the desired first camera 31-39 from the first drop-down menu 71 shown in FIG. 4(b). Here, it is assumed that the first camera 34 (see FIG. 1) is selected.

[0047] When the worker clicks the "Find Snap Window" button 72 (see FIG. 4(b)) using the input unit 51 of the operation terminal 5, the operation terminal 5 displays all applications running on the operation terminal 5 in a second drop-down menu 73 (see FIG. 4(b)). When the worker selects the robot operation program AP2 (see FIG. 5) from the second drop-down menu 73, the operation terminal 5 acquires an identifier of the window of the robot operation program AP2. Thereafter, the operation terminal 5 continues to display an image depicted in a predetermined color (e.g., green) from the display contents of the window at that position, superimposed on the superimposition application AP1. Hereinafter, the explanation will be continued on the assumption that the worker selects the robot operation program AP2 (see FIG. 5) from the second drop-down menu 73.

[0048] Next, the operation terminal 5 causes the image P of the work site captured by the first camera 34 (see FIG. 1) selected in the first drop-down menu 71 (see FIG. 4(b)) to be displayed in the overlay application AP1 (see FIG. 4(b), step S104).

[0049] Next, the operation terminal 5 determines information on the position from which the selected first camera 34 (see FIG. 1) captures an image (step S105). More specifically, the operation terminal 5 reads out the information on the position from which the first camera 34 captures an image, which information was stored in the RAM 53 (see FIG. 2) in step S102 (see FIG. 3).

[0050] Next, the operation terminal 5 notifies the robot operation program AP2 of the information read out from the RAM 53 (see FIG. 2) (step S106).

[0051] The purpose of the process in step S106 is as follows: Fig. 5(a) shows a three-dimensional model 9 of the robot 2 (see Fig. 1) and a three-dimensional model 9a of the turntable T, which are depicted in the virtual space K immediately after the robot operation program AP2 is started. However, this three-dimensional model 9 is depicted by viewing the robot 2 (see Fig. 4(b)) displayed in the overlay application AP1 from a different angle, and further, the three-dimensional model 9a is also depicted by viewing the turntable T (see Fig. 4(b)) displayed in the overlay application AP1 from a different angle.

[0052] Therefore, the operation terminal 5 notifies the robot operation program AP2 of the information read out from the RAM 53 (see FIG. 2), that is, the information on the position from which the first camera 34 is imaging the work site (step S106). Then, when the operation terminal 5 receives the information in the robot operation program AP2, it re-renders the three-dimensional models 9, 9a (see FIG. 5(a)) (step S3). More specifically, the operation terminal 5 converts the information on the position from which the first camera 34 is imaging the work site, received from the overlay application AP1, into the position and direction in the virtual space K (see FIG. 5(b)) from which the first camera 34 is viewing in the robot operation program AP2. Then, as shown in FIG. 5(b), the operation terminal 5 re-renders the three-dimensional models 9A and 9aA when viewed in the same direction from the same position as the first camera 34 in the virtual space K, and displays them in the robot operation program AP2.

[0053] On the other hand, in the superimposition application AP1, the three-dimensional models 9A, 9aA displayed in the window of the robot operation program AP2 are superimposed and displayed (step S107). That is, the operation terminal 5 continues to display an image depicted in a predetermined color (e.g., green) among the display contents of the window of the robot operation program AP2, superimposed on the superimposition application AP1. Therefore, the re-rendered three-dimensional models 9A, 9aA (see FIG. 5(b)) are displayed as the three-dimensional model 9A of the robot 2 and the three-dimensional model 9aA of the turntable T, superimposed on the superimposition application AP1, as shown in FIG. 6.

[0054] Next, the operation terminal 5 moves the three-dimensional model 9A according to the operation of the worker by the robot operation program AP2 (step S4). More specifically, the worker uses the input unit 51 (see FIG. 2) of the operation terminal 5 to move the three-dimensional model 9A (FIG. 5(b)) displayed in the robot operation program AP2 on the robot operation screen 8 by an operation such as dragging (not shown). At this time, the operation terminal 5 continues to display an image depicted in a predetermined color (e.g., green) among the display contents of the window of the robot operation program AP2, superimposed on the superimposition application AP1. Therefore, the three-dimensional model 9A moved in the robot operation program AP2 by an operation such as dragging is also displayed in the superimposition application AP1 superimposed on the image P of the work site (not shown). By doing so, the worker can check, while looking at the first display unit 7, by the superimposition application AP1 whether or not a problem such as interference will occur if the robot 2 moves in the same way as the three-dimensional model 9A.

[0055] Next, the operation terminal 5 moves the robot 2 (step S5). Specifically, if the operator determines that there is no problem in moving the robot 2 to the position and orientation of the three-dimensional model 9A, he or she presses an OK button (not shown). This causes the operation terminal 5 to transmit the position and orientation of the three-dimensional model 9A to the PLC 29 via the communication unit 55. The PLC 29 moves the robot 2 based on the received instruction (not shown).

[0056] If the worker selects another first camera 31-39 in the overlay application AP1 in step S4 or step S5, the operation terminal 5 detects this in step S103 and executes steps S104-106. Then, when the robot operation program AP2 receives the position and the like from the overlay application AP1 (step S2), it redraws the three-dimensional model (step S3).

[0057] Meanwhile, the operation terminal 5 displays in real time on the second display unit 6 an image of the entire work site captured by the second camera 4 (see FIG. 2). When the first cameras 31-39 capture images of only a portion of the work site, the worker cannot get an overview of the entire work site on the first display unit 7. However, because the second camera 4 captures images of the entire work site from a high position, the worker can check an overview of the entire work site on the second display unit 6, which displays the image captured by the second camera 4.

[0058] According to the present embodiment described above, when the robot 2 and the three-dimensional model 9A in the image P of the work site are superimposed and displayed on the first display unit 7, it is necessary to view them from the same direction. If the image P to be displayed on the first display unit 7 is generated according to the position from which the three-dimensional model 9 depicts the robot 2, it is necessary to synthesize the images P of the work site captured by the first cameras 31 to 39. However, according to the present embodiment, the three-dimensional model 9 is re-rendered to become the three-dimensional model 9A (see FIG. 5(b)) according to the position from which the image P of the work site was captured. Therefore, since there is no need for synthesis processing, the processing time is shortened. Also, a clear image P of the work site that is not synthesized can be displayed on the first display unit 7.

[0059] Then, when directly operating the robot without teaching, the worker can move the robot 2 by dragging or otherwise manipulating the 3D model 9A on the robot operation screen 8 while checking on the clear image P of the work site on the first display unit 7 whether or not moving the robot 2 in the same way as the 3D model 9A will cause problems such as interference.

[0060] As described above, according to this embodiment, it is possible to provide an operating method of the robot operation system 1 that enables the robot 2 to be operated with simple operations while viewing a clear image P obtained in a short processing time.

[0061] <Description of Modifications> It should be noted that the robot operation system 1 shown in this embodiment is merely an example, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. For example, in this embodiment, the first cameras 31-39 are used, but it is also possible to use a single first camera, for example, only the first camera 34. However, using multiple first cameras 31-39 is preferable in that images captured from multiple positions can be displayed with a simple configuration.

[0062] Furthermore, in this embodiment, calibration of each of the first cameras 31-39 is first performed, and the position from which the robot 2 is to be imaged is stored in the RAM 53, but it may also be done as follows. That is, each time the worker selects one of the first cameras 31-39, the image (see FIG. 4(a)) in which the mark M1 and the mark M2 are captured by that camera may be analyzed, and the position from which the robot 2 is to be imaged may be calculated. However, it is preferable to store this in the RAM 53 in advance, since it is not necessary to execute the calculation process each time a first camera is selected.

[0063] In this embodiment, the overlay application AP1 and the robot operation program AP2 are separated, but the processes performed by these may be performed by one application. In this embodiment, the overlay application AP1 and the robot operation program AP2 are displayed on separate display units, but they may be displayed on one display unit. [Explanation of symbols]

[0064] 2. Robot 31~39 1st Camera 4. Second Camera 52 ROM (3D model storage section) 53 RAM (first camera position memory unit) 6 Second display 7 1st display section 9, 9A 3D model W work P Image (work site image)

Claims

1. A robot that performs a predetermined processing operation on a workpiece; A single first camera capable of capturing images of the robot and the workpiece; a three-dimensional model storage unit for storing a three-dimensional model of the robot; a first display unit that displays an image of the robot captured by the first camera and a three-dimensional model in a superimposed manner; a control unit that, when the three-dimensional model displayed on the first display unit moves to a predetermined position, moves the robot to a predetermined position accordingly; A method for operating a robot manipulation system comprising: determining from which position an image of the robot to be displayed on the first display unit has been captured; generating a three-dimensional model of the robot as viewed from the determined position; a step of superimposing the generated three-dimensional model of the robot on an image of the robot and displaying the three-dimensional model on the first display unit; A method of operating a robotic manipulation system comprising:

2. The first camera includes a plurality of first cameras fixedly disposed so as to be able to image the robot and the workpiece from different positions, The method for operating a robot manipulation system according to claim 1 , wherein the image of the robot displayed on the first display unit is an image of one first camera selected from the plurality of first cameras.

3. Furthermore, the robot operation system includes a first camera position storage unit that stores a position from which the first camera images the robot and the workpiece, 3. The method of claim 2, wherein when determining from which position the image of the robot to be displayed on the first display unit was taken, the position from which the selected one of the first cameras that took the image took the image of the robot and the workpiece is obtained from a first camera position memory unit.

4. Further, a second camera capable of capturing an image of the entire work site including the robot and the workpiece; The method for operating a robot operation system according to claim 1 , further comprising: a second display unit that displays an image captured by the second camera in real time.

Images