How the robotic operation system works

Abstract

To provide an operation method of a robot operation system allowing for operating a robot by easy operation.SOLUTION: When a three-dimensional model displayed on a first display unit 7 is moved to a predetermined position, in response to the move, a robot 2 is caused to move to a predetermined position. When displaying the three-dimensional model on the first display unit 7, it is determined from which position an image of the robot 2 to be displayed on the first display unit 7 is captured; a three-dimensional model of the robot 2 viewed from the determined position is generated; and the generated three-dimensional model of the robot 2 is superimposed onto the image of the robot 2 and displayed on the first display unit 7.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method of operating a robotic manipulation system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, when using an industrial robot to perform work such as welding on a workpiece, a teaching process is carried out as a preparation for the robot to memorize in advance the operations required for moving to a destination.

[0003] Generally, two teaching methods are known: an online teaching method using a teach pendant (see FIG. 1 of Patent Document 1) and an offline teaching method (Patent Document 2).

[0004] However, all teaching methods have the problem of being dangerous because they ultimately require work to be done at the actual work site.

[0005] In order to solve this problem, the invention of Patent Document 3 has been proposed. The invention described in Patent Document 3 involves capturing images of the robot's work site using multiple multi-viewpoint image capturing cameras, synthesizing the captured images to generate an arbitrary viewpoint image, and using the generated arbitrary viewpoint image to instruct the robot to operate. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-132728 [Patent Document 2] Japanese Patent Application Publication No. 9-179624 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-251395 Summary of the Invention [Problem to be solved by the invention]

[0007] However, although the above-mentioned inventions are easy to operate, it takes a very long time to synthesize the images, and even if they are successfully synthesized, the resulting images are often unclear, making them unsuitable for practical use.

[0008] In view of the above problems, the present invention aims to provide an operating method for a robot operation system that allows a user to operate a robot with simple operations while viewing clear images obtained in a short processing time. [Means for solving the problem]

[0009] The above object of the present invention can be achieved by the following means: Note that the parentheses indicate reference symbols of embodiments to be described later, but the present invention is not limited to these.

[0010] The method for operating a robot operation system according to claim 1 comprises: a robot (for example, robot 2 shown in FIG. 2) that performs a predetermined processing operation on a workpiece (for example, workpiece W shown in FIG. 2); Capable of capturing images of the robot (for example, robot 2 shown in FIG. 2) and the workpiece (for example, workpiece W shown in FIG. 2) No. one camera (e.g., the first camera 34 shown in FIG. 1); a three-dimensional model storage unit (for example, the ROM 52 shown in FIG. 2) that stores a three-dimensional model (for example, the three-dimensional model 9 shown in FIG. 5(a) and the three-dimensional model 9A shown in FIG. 5(b)) of the robot (for example, the robot 2 shown in FIG. 2); a first display unit (for example, the first display unit 7 shown in FIG. 2) that displays an image (for example, an image P of the work site shown in FIG. 4(b)) of the robot (for example, the robot 2 shown in FIG. 2) captured by the first camera (for example, the first camera 34 shown in FIG. 1) superimposed on a three-dimensional model (for example, the three-dimensional model 9 shown in FIG. 5(a) or the three-dimensional model 9A shown in FIG. 5(b)); The three-dimensional model (for example, the three-dimensional model 9 shown in FIG. 5(a) or the three-dimensional model 9A shown in FIG. 5(b)) displayed on the first display unit (for example, the first display unit 7 shown in FIG. 2) is displayed at a predetermined position. postureWhen the robot moves to a predetermined position, the robot (for example, the robot 2 shown in FIG. 2) moves to a predetermined position. posture a control unit for moving the A method for operating a robot manipulation system comprising: In order to superimpose an image of the robot (e.g., image P of the work site shown in FIG. 4(b)) captured by the first camera (e.g., first camera 34 shown in FIG. 1) on a three-dimensional model (e.g., three-dimensional model 9 shown in FIG. 5(a) or three-dimensional model 9A shown in FIG. 5(b)) and display the superimposed image on the first display unit (e.g., first display unit 7 shown in FIG. 2), first, a step of determining from what position an image of the robot (e.g., robot 2 shown in FIG. 2) (e.g., image P of the work site captured by first camera 34 shown in FIG. 4(b)) to be displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) has been captured (e.g., step S105 shown in FIG. 3); a step of generating a three-dimensional model of the robot (for example, the robot 2 shown in FIG. 2) viewed from the determined position (for example, a three-dimensional model 9A of the robot 2 viewed from the first camera 34 shown in FIG. 5(b)) (for example, step S3 shown in FIG. 3); a step (e.g., step S107 shown in FIG. 3) of superimposing the generated three-dimensional model of the robot (e.g., robot 2 shown in FIG. 2) (e.g., three-dimensional model 9A of robot 2 seen from first camera 34 shown in FIG. 5(b)) on an image of the robot (e.g., robot 2 shown in FIG. 2) (e.g., image P of the work site captured by first camera 34 shown in FIG. 4(b)) so as to match the shape of the robot (e.g., robot 2 shown in FIG. 4(b)), and displaying the image on the first display unit (e.g., first display unit 7 shown in FIG. 2) (e.g., first display unit 7 shown in FIG. 6); Including the law of nature, The first cameras (for example, first cameras 31 to 39 shown in FIG. 2) are a plurality of first cameras (for example, first cameras 31 to 39 shown in FIG. 2) fixedly arranged so as to be able to capture images of the robot (for example, robot 2 shown in FIG. 2) and the workpiece (for example, workpiece W shown in FIG. 2) from different positions, The image of the robot (for example, the robot 2 shown in FIG. 2) (for example, the image P of the work site shown in FIG. 4(b)) displayed on the first display unit (for example, the first display unit 7 shown in FIG. 2) is an image (for example, the image P of the work site shown in FIG. 4(b)) of one first camera (for example, the first camera 34 shown in FIG. 1) selected from the plurality of first cameras (for example, the first cameras 31 to 39 shown in FIG. 2). It is characterized by the following.

[0012] Claim 2 The method for operating the robot operation system described in the claim 1 In the method for operating the robot operation system described in Furthermore, the robot operation system includes a first camera position storage unit (for example, RAM 53 shown in FIG. 2) that stores the position from which the first camera (for example, first cameras 31 to 39 shown in FIG. 2) images the robot and the workpiece, When determining from which position the image (e.g., image P of the work site shown in Figure 4(b)) of the robot (e.g., robot 2 shown in Figure 2) to be displayed on the first display unit (e.g., first display unit 7 shown in Figure 2) was taken, the position from which the selected first camera (e.g., first cameras 31 to 39 shown in Figure 2) that took the image (e.g., image P of the work site shown in Figure 4(b)) took the image of the robot and the workpiece is obtained from the first camera position memory unit (e.g., RAM 53 shown in Figure 2).

[0013] Claim 3 The method for operating a robot operation system according to claim 1 further comprises: Furthermore, a second camera (for example, the second camera 4 shown in FIG. 2) capable of capturing an image of the entire work site including the robot (for example, the robot 2 shown in FIG. 2) and the work (for example, the work W shown in FIG. 2); The device is characterized by further comprising a second display unit (for example, second display unit 6 shown in the figure) that displays the image captured by the second camera in real time. [Effects of the Invention]

[0014] Next, the effects of the present invention will be described with reference to the drawings. Note that the reference symbols in parentheses are those of the embodiments described below, but the present invention is not limited to these.

[0015] According to the invention of claim 1, the position from which the image (e.g., image P of the work site captured by the first camera 34 shown in FIG. 4(b)) to be displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) was captured is determined, and a three-dimensional model (e.g., three-dimensional model 9A of the robot 2 viewed from the first camera 34 shown in FIG. 5(b)) reproducing the robot as seen from that position in a virtual space is superimposed on the image displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2) (e.g., image P of the work site captured by the first camera 34 shown in FIG. 6).

[0016] Then, while viewing an image (for example, an image P of the work site captured by the first camera 34 shown in FIG. 6) on which the three-dimensional model (for example, the three-dimensional model 9A shown in FIG. 6) is superimposed, the worker can move the three-dimensional model (for example, the three-dimensional model 9A shown in FIG. 6) to a desired position by dragging or the like. posture Then, when a robot (for example, robot 2 shown in FIG. 2) moves in the same way as a three-dimensional model (for example, three-dimensional model 9A shown in FIG. 6), it is checked using an image (for example, image P of the work site captured by first camera 34 shown in FIG. 6) whether or not problems such as interference will occur, and once it is determined that there will be no problems, the actual robot (for example, robot 2 shown in FIG. 2) can be moved. Therefore, the robot can be operated with simple operations.

[0017] Here, if an image of the work site is to be generated in accordance with the 3D model, it is necessary to generate the image by compositing or the like in accordance with the position from which the 3D model depicts the robot, and display it on the first display unit (for example, the first display unit 7 shown in FIG. 2). Therefore, compositing or the like takes time, and the image is not clear because it is a composite image. However, according to the present invention, a 3D model (for example, the 3D model 9A of the robot 2 seen from the first camera 34 shown in FIG. 5(b)) is generated in accordance with an image (for example, the image P of the work site captured by the first camera 34 shown in FIG. 4(b)). Therefore, there is no need to composite or the like the images, and processing time is shortened. Furthermore, the image (for example, the image P of the work site captured by the first camera 34 shown in FIG. 6) displayed on the first display unit (for example, the first display unit 7 shown in FIG. 2) is clear because it is not an image generated by compositing or the like. Furthermore, according to the present invention, images captured from multiple positions can be displayed with a simple configuration. That is, when a single first camera is used, images captured from multiple positions cannot be displayed, which is inconvenient. Therefore, multiple first cameras (e.g., first cameras 31 to 39 shown in FIG. 2) are provided that are fixedly positioned so that images can be captured from different positions. In this way, with a simple configuration, a desired first camera (e.g., first cameras 31 to 39 shown in FIG. 2) can be selected and the image of the selected first camera (e.g., image P of the work site captured by first camera 34 shown in FIG. 4(b)) can be displayed on the first display unit (e.g., first display unit 7 shown in FIG. 2).

[0018] Therefore, according to the present invention, it is possible to provide an operating method for a robot operation system that allows a robot (for example, robot 2 shown in Figure 2) to be operated with simple operations while viewing a clear image (for example, image P of the work site captured by first camera 34 shown in Figure 6) obtained in a short processing time.

[0020] Claim 2 According to the invention of (1), the position from which the first camera images the robot and the workpiece can be stored in a first camera position storage unit (for example, RAM 53 shown in FIG. 2) in advance by first calibrating the first camera (for example, first cameras 31 to 39 shown in FIG. 2). Therefore, every time an image (for example, image P of the work site shown in FIG. 4(b)) to be displayed on a first display unit (for example, first display unit 7 shown in FIG. 2) is selected, the position from which the selected first camera (for example, first cameras 31 to 39 shown in FIG. 2) that captured the image (for example, image P of the work site shown in FIG. 4(b)) images the robot and the workpiece can be determined by simply reading the information from the first camera position storage unit (for example, RAM 53 shown in FIG. 2), and there is no need to perform calculations or the like each time, which simplifies processing.

[0021] Claim 3 According to the invention, the entire work site including the movement of the robot (for example, robot 2 shown in FIG. 2) can be viewed in real time on the second display unit (for example, second display unit 6 shown in the figure). Therefore, even if it is not possible to view the entire work site from an image (for example, image P of the work site shown in FIG. 4(b)) captured by one first camera (for example, first cameras 31 to 39 shown in FIG. 2), the entire work site can be confirmed on the second display unit (for example, second display unit 6 shown in the figure). [Brief explanation of the drawings]

[0022] [Figure 1] 1 is an overall view of a robot operation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is an overall schematic diagram of the robot operation system according to the embodiment. [Figure 3] FIG. 10 is a flowchart showing the operation of the robot operation system according to the embodiment. [Figure 4] (a) is the image used for calibration, and (b) is the display screen of the overlay application when the first camera is selected. [Figure 5] (a) is the display screen of the robot operation program showing the three-dimensional model before redrawing, and (b) is the display screen of the robot operation program when the three-dimensional model is redrawn to match the selected first camera. [Figure 6] 10 is a display screen of an overlay application showing a redrawn three-dimensional model overlaid on an image captured by a selected first camera. DETAILED DESCRIPTION OF THE INVENTION

[0023] An embodiment of a robot manipulation system using a method for operating a robot manipulation system according to the present invention will be described in detail below with reference to the drawings. In the following description, when directions such as up, down, left, and right are indicated, they refer to up, down, left, and right when viewed from the front of the illustration.

[0024] <Outline of the robot operation system> First, an overview of the robot operation system 1 in this embodiment will be described. As shown in Fig. 1, the robot operation system 1 in this embodiment includes a turntable T on which a workpiece W (see Fig. 2) is placed, and a robot 2 that performs predetermined processing work on the workpiece W, such as welding or polishing, which are imaged by a plurality of first cameras 31 to 39. An operator operates the robot 2 using an operation terminal 5 (see Fig. 2) while viewing the images of the robot 2, the workpiece W, and the surrounding situation captured by the first cameras 31 to 39.

[0025] The configuration of the robot operation system 1 in this embodiment will be described more specifically with reference to Fig. 2. First, as shown in Fig. 2, the robot 2 is connected to an operation terminal 5 via a network N through a programmable logic controller (hereinafter referred to as PLC 29). Meanwhile, the first cameras 31 to 39 are connected to the operation terminal 5 via the network N. Furthermore, the second camera 4 is also connected to the operation terminal 5 via the network N.

[0026] Furthermore, the operation terminal 5 is connected to the second display unit 6, the first display unit 7, and the robot operation screen 8 via a monitor cable such as an HDMI (registered trademark) cable.

[0027] Here, a method for moving the robot 2 using the robot operation system 1 configured as described above will be briefly described. First, the worker selects one of the first cameras 31 to 39 on the operation terminal 5, and an image P of the work site captured by the selected first camera is displayed on the first display unit 7 (see FIG. 4(b)). Next, the operation terminal 5 converts information about the position from which the work site was captured in the image P into a position and direction in the virtual space K (see FIG. 5(a)). Then, the operation terminal 5 re-renders a three-dimensional model 9A in the virtual space K as if it were viewed from the same position and in the same direction as the position at which the image P was captured (FIG. 5(b)). The operation terminal 5 then overlays the re-rendered three-dimensional model 9A on the image P of the work site displayed on the first display unit 7 (FIG. 6). Finally, the worker operates the robot (not shown) by, for example, dragging the three-dimensional model 9A overlaid on the image P of the work site.

[0028] The outline of the robot operation system 1 in this embodiment has been described above. Next, each part in FIG. 2 will be described in detail.

[0029] <Robot description> The robot 2 is a multi-joint general-purpose robot with six degrees of freedom. More specifically, as shown in Fig. 2, the robot 2 includes a base 20 fixed to the floor or the like of a work area, an arm 22 whose base end is rotatably connected to the base 20, and a work tool 24 attached to a tip 22a of the arm 22 and performing work such as welding, sealing, painting, cutting, polishing, handling, etc. on a workpiece W placed on a turntable T. Note that, although a six-axis multi-joint general-purpose robot has been exemplified as the robot 2, this is not limiting and any robot may be used.

[0030] <Explanation of the first camera> The first cameras 31-39 are digital cameras capable of capturing color images. To install the first cameras 31-39 at the work site, in this embodiment, a camera installation frame 30 is installed within a safety fence D where the robot 2 and turntable T are installed, as shown in FIG. 1. The camera installation frame 30 includes a first frame 30b, a second frame 30c, and a third frame 30d, which are spaced apart from each other from right to left as shown in FIG. 1. The camera installation frame 30 further includes a common horizontal member 30a connecting the first frame 30b, the second frame 30c, and the third frame 30d. As shown in FIG. 1, the first frame 30b, the second frame 30c, and the third frame 30d each include a pair of vertical members spaced apart from each other in the front-to-rear direction and a horizontal member connecting the upper ends of the vertical members. The common horizontal member 30a is also provided to connect the centers of the horizontal members of the first frame 30b, the second frame 30c, and the third frame 30d.

[0031] First cameras 31 and 32 are installed vertically spaced apart on the vertical member on the rear side of first frame 30b shown in Fig. 1, and first cameras 33 and 34 are installed vertically spaced apart on the vertical member on the front side of the frame. A first camera 36 is installed on the vertical member on the rear side of second frame 30c shown in Fig. 1, and a first camera 37 is installed on the vertical member on the front side of the frame. Furthermore, a pair of vertical members installed at a fixed distance in the front-to-rear direction of second frame 30c shown in Fig. 1 are provided with members 30c1 and 30c2 extending toward the left side of the frame, and first cameras 38 and 39 are installed at the tips of the members.

[0032] <Explanation of the second camera> The second camera 4 is a webcam that can move the lens direction up, down, left, and right, and can expand and contract the angle of view. As shown in Fig. 1, the second camera 4 is installed near the intersection of the first frame 30b and the frame member 30a, and because it is installed at a high position, it can capture images of the entire work site.

[0033] 2, second camera 4 is connected to operation terminal 5 via a dedicated line L that is separate from network N for first cameras 31-39. This allows second camera 4 to transmit captured images without delay, regardless of the traffic conditions of network N that connects first cameras 31-39 and operation terminal 5.

[0034] Incidentally, the image captured by the second camera 4 is displayed on the second display unit 6 (see FIG. 2) of the operation terminal 5. Note that a terminal other than the operation terminal 5 may be prepared and connected to the second camera 4 and the second display unit 6. In this case, the image captured by the second camera 4 can be displayed on the second display unit 6 without delay, even while the operation terminal 5 is performing the processing described below (see FIG. 3).

[0035] <Operation terminal description> The operation terminal 5 is an information processing device, such as a PC (Personal Computer). Specifically, as shown in Fig. 2, the operation terminal 5 is configured with a CPU 50, an input unit 51 that can input predetermined data to the operation terminal 5 from the outside using a mouse, keyboard, touch panel, USB connector, Lightning connector, etc., a ROM 52 that is a writable flash ROM or the like that stores predetermined application programs, etc., a RAM 53 that functions as a work area, buffer memory, etc., a display unit 54 that can display various images, and a communication unit 55 that can connect to the network N by communication means such as wireless LAN, wired LAN, dial-up, etc.

[0036] The ROM 52 stores an overlay application AP1 (see FIGS. 4(b) and 6) that displays an image P (see FIG. 4(b)) of the work site superimposed on the three-dimensional models 9, 9a (see FIG. 5(a)), a robot operation program AP2 (see FIG. 5) that the worker uses to move the robot 2, and the three-dimensional models 9, 9a (see FIG. 5(a)) used in the program. The RAM 53 also stores information about the positions from which the first cameras 31-39 capture images of the robot 2 (see step S102 in FIG. 3).

[0037] <Display section explanation> The second display unit 6, the first display unit 7, and the robot operation screen 8 are connected to the operation terminal 5, as shown in Fig. 2. The second display unit 6, the first display unit 7, and the robot operation screen 8 are each an independent liquid crystal monitor.

[0038] The robot operation screen 8 displays the robot operation program AP2 (see FIG. 5). The robot operation program AP2 displays three-dimensional models 9, 9a (see FIG. 5(a)) used for operating the robot.

[0039] The first display unit 7 displays the overlay application AP1 (see FIG. 4(b) and FIG. 6). The overlay application AP1 displays the image P received from the first camera 31-39, the three-dimensional model 9 of the robot 2 displayed on the robot operation program AP2 (robot operation screen 8), and the three-dimensional model 9a of the turntable T in an overlaid manner (see FIG. 6).

[0040] The second display unit 6 displays an image overlooking the entire work site, which the operation terminal 5 receives from the second camera 4 (not shown).

[0041] <Explanation of usage examples of the robot operation system> An example of how to use the operating method of the robot operation system 1 configured as above will be specifically described with reference to Fig. 3. Note that the following content is merely an example, and is not intended to be limiting.

[0042] 2 (hereinafter, may be abbreviated as operation terminal 5) receives instructions from the worker via input unit 51 and starts up superimposition application AP1 and robot operation program AP2. The operation terminal 5 displays the started superimposition application AP1 on the first display unit 7 (see FIG. 4(b)), and displays the robot operation program AP2 on the robot operation screen 8 (see FIG. 5).

[0043] Next, in the robot operation program AP2, the operation terminal 5 reads out the three-dimensional models 9, 9a from the ROM 52 (step S1). After that, the operation terminal 5 puts the robot operation program AP2 on standby until it receives information from the overlay application AP1 (step S2).

[0044] Meanwhile, the operation terminal 5 starts receiving images captured by the first cameras 31 to 39 and the second camera 4 (see FIG. 2) shown in FIG. 2 in the overlay application AP1 (step S101).

[0045] Next, the operation terminal 5, in the overlay application AP1, calibrates the first cameras 31 to 39 based on the images received from the first cameras 31 to 39 shown in FIG. 2, and stores the positions from which the first cameras 31 to 39 are capturing images of the work site (step S102). Specific processing will be described with reference to FIG. 4(a). FIG. 4(a) is an example of an image P of the work site received by the operation terminal 5 from the first cameras 31 to 39. As shown in FIG. 4(a), a plurality of marks M1 are provided on the turntable T, and a mark M2 is provided on the tip 22a of the arm 22 of the robot 2. The operation terminal 5 determines information about the position from which the work site is being imaged for each of the first cameras 31-39 (see FIG. 1) (for example, three-dimensional coordinates indicating the position of the first cameras 31-39 within the work site and three-dimensional vectors indicating the line-of-sight direction of the first cameras 31-39) based on how these marks M1 and M2 appear in the image P of the work site captured by each of the first cameras 31-39 (see FIG. 1), and stores this information in the RAM 53. Note that determining the position from which an image was captured based on the captured marks is a conventionally known technique, and therefore a detailed description thereof will be omitted.

[0046] Next, the operation terminal 5 waits for the worker to select one of the first cameras 31 to 39 in the overlay application AP1 (step S103). More specifically, the worker first clicks the "Find Camera" button 70 shown in FIG. 4(b) using the input unit 51 of the operation terminal 5. This causes the operation terminal 5 to display the first cameras 31 to 39 (see FIG. 1) connected to the operation terminal 5 in a first drop-down menu 71. The worker selects the desired first camera 31 to 39 from the first drop-down menu 71 shown in FIG. 4(b). Here, it is assumed that the first camera 34 (see FIG. 1) has been selected.

[0047] When the worker clicks the "Find Snap Window" button 72 (see FIG. 4(b)) using the input unit 51 of the operation terminal 5, the operation terminal 5 displays all applications running on the operation terminal 5 in a second drop-down menu 73 (see FIG. 4(b)). When the worker selects the robot operation program AP2 (see FIG. 5) from the second drop-down menu 73, the operation terminal 5 acquires the identifier of the window of the robot operation program AP2. Thereafter, the operation terminal 5 continues to display an image depicted in a predetermined color (e.g., green) from the display content of the window at that position, superimposed on the superimposition application AP1. The following description will be continued assuming that the worker selects the robot operation program AP2 (see FIG. 5) from the second drop-down menu 73.

[0048] Next, the operation terminal 5 displays an image P of the work site captured by the first camera 34 (see Figure 1) selected in the first drop-down menu 71 (see Figure 4(b)) in the overlay application AP1 (see Figure 4(b), step S104).

[0049] Next, the operation terminal 5 determines information on the position from which the selected first camera 34 (see FIG. 1) is capturing an image (step S105). More specifically, the operation terminal 5 reads out the information on the position from which the first camera 34 is capturing an image, which information was stored in the RAM 53 (see FIG. 2) in step S102 (see FIG. 3).

[0050] Next, the operation terminal 5 notifies the robot operation program AP2 of the information read out from the RAM 53 (see FIG. 2) (step S106).

[0051] The purpose of the processing in step S106 is as follows: Fig. 5(a) shows a three-dimensional model 9 of the robot 2 (see Fig. 1) and a three-dimensional model 9a of the turntable T, which are rendered in the virtual space K immediately after the robot operation program AP2 is launched. However, this three-dimensional model 9 is a rendering of the robot 2 (see Fig. 4(b)) displayed in the overlay application AP1, viewed from a different angle, and furthermore, the three-dimensional model 9a is a rendering of the turntable T (see Fig. 4(b)) displayed in the overlay application AP1, viewed from a different angle.

[0052] Therefore, the operation terminal 5 notifies the robot operation program AP2 of the information read out from the RAM 53 (see FIG. 2), i.e., information about the position from which the first camera 34 is capturing an image of the work site (step S106). Then, when the operation terminal 5 receives the information in the robot operation program AP2, it re-renders the three-dimensional models 9, 9a (see FIG. 5(a)) (step S3). More specifically, the operation terminal 5 converts the information about the position from which the first camera 34 is capturing an image of the work site, received from the overlay application AP1, into a position and a direction in the virtual space K (see FIG. 5(b)) from which the first camera 34 is viewing the image, in the robot operation program AP2. Then, as shown in FIG. 5(b), the operation terminal 5 re-renders the three-dimensional models 9A and 9aA in the virtual space K when viewed in the same direction from the same position as the first camera 34, and displays them in the robot operation program AP2.

[0053] Meanwhile, the overlay application AP1 displays the three-dimensional models 9A, 9aA displayed in the window of the robot operation program AP2 in an overlaid manner (step S107). That is, the operation terminal 5 continues to display, in the overlay application AP1, an image depicted in a predetermined color (e.g., green) from among the display contents of the window of the robot operation program AP2. Therefore, the re-rendered three-dimensional models 9A, 9aA (see FIG. 5(b)) are displayed in the overlay application AP1, with the three-dimensional model 9A of the robot 2 and the three-dimensional model 9aA of the turntable T being overlaid, as shown in FIG. 6.

[0054] Next, the operation terminal 5 moves the three-dimensional model 9A according to the operator's operation using the robot operation program AP2 (step S4). More specifically, the operator uses the input unit 51 (see FIG. 2) of the operation terminal 5 to move the three-dimensional model 9A (FIG. 5(b)) displayed in the robot operation program AP2 on the robot operation screen 8 by dragging or other operations (not shown). At this time, the operation terminal 5 continues to display an image depicted in a predetermined color (e.g., green) from the display contents of the window of the robot operation program AP2, superimposed on the superimposition application AP1. Therefore, the three-dimensional model 9A moved by dragging or other operations in the robot operation program AP2 is also displayed in the superimposition application AP1, superimposed on the image P of the work site (not shown). This allows the operator to check, while looking at the first display unit 7, using the superimposition application AP1 whether or not a problem such as interference will occur if the robot 2 moves in the same way as the three-dimensional model 9A.

[0055] Next, the operation terminal 5 moves the robot 2 (step S5). Specifically, if the operator determines that there is no problem in moving the robot 2 to the position and orientation of the three-dimensional model 9A, he or she presses an OK button (not shown). This causes the operation terminal 5 to transmit the position and orientation of the three-dimensional model 9A to the PLC 29 via the communication unit 55. The PLC 29 moves the robot 2 (not shown) based on the received instruction.

[0056] If the worker selects another first camera 31-39 in the overlay application AP1 in step S4 or step S5, the operation terminal 5 detects this in step S103 and executes steps S104 to S106. Then, when the robot operation program AP2 receives the position and the like from the overlay application AP1 (step S2), it re-renders the three-dimensional model (step S3).

[0057] Meanwhile, the operation terminal 5 displays in real time an image of the entire work site captured by the second camera 4 (see FIG. 2) on the second display unit 6. If the first cameras 31-39 are capturing images of only a portion of the work site, the worker cannot get a bird's-eye view of the entire work site on the first display unit 7. However, because the second camera 4 captures images of the entire work site from a high position, the worker can check an image of the entire work site on the second display unit 6, which displays the image captured by the second camera 4.

[0058] According to the present embodiment described above, when the robot 2 and the three-dimensional model 9A in the image P of the work site are superimposed and displayed on the first display unit 7, it is necessary to view both of them from the same direction. If the image P to be displayed on the first display unit 7 were to be generated according to the position from which the three-dimensional model 9 depicts the robot 2, it would be necessary to composite the images P of the work site captured by the first cameras 31 to 39. However, according to the present embodiment, the three-dimensional model 9 is re-rendered to become the three-dimensional model 9A (see FIG. 5(b)) according to the position from which the image P of the work site was captured. This eliminates the need for composite processing, thereby shortening the processing time. Furthermore, a clear image P of the work site that has not been composited can be displayed on the first display unit 7.

[0059] When operating the robot directly without teaching, the worker can move the robot 2 by dragging or otherwise manipulating the three-dimensional model 9A on the robot operation screen 8 while checking on the clear image P of the work site on the first display unit 7 whether or not moving the robot 2 in the same way as the three-dimensional model 9A will cause problems such as interference.

[0060] As described above, according to this embodiment, it is possible to provide an operating method of the robot operation system 1 that allows the robot 2 to be operated with simple operations while viewing a clear image P obtained in a short processing time.

[0061] <Description of Modifications> It should be noted that the robot operation system 1 shown in this embodiment is merely an example, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims. For example, in this embodiment, the first cameras 31 to 39 are used, but it is also possible to use a single first camera, for example, only the first camera 34. However, using multiple first cameras 31 to 39 is preferable because it allows images captured from multiple positions to be displayed with a simple configuration.

[0062] Furthermore, in this embodiment, calibration of each of the first cameras 31 to 39 is first performed, and the position from which the robot 2 is to be imaged is stored in the RAM 53. However, it may also be possible to do as follows. That is, each time the worker selects one of the first cameras 31 to 39, the image (see FIG. 4(a)) capturing marks M1 and M2 captured by that camera may be analyzed, and the position from which the robot 2 is to be imaged may be calculated. However, it is preferable to store this information in the RAM 53 in advance, since it is not necessary to execute the calculation process each time a first camera is selected.

[0063] In this embodiment, the superimposition application AP1 and the robot operation program AP2 are separated, but the processes performed by these may be performed by a single application. In this embodiment, the superimposition application AP1 and the robot operation program AP2 are displayed on separate display units, but they may be displayed on a single display unit. [Explanation of symbols]

[0064] 2. Robot 31~39 Camera 1 4. Second Camera 52 ROM (3D model storage area) 53 RAM (first camera position memory unit) 6 Second display 7 1st display section 9, 9A 3D model double work P Image (Image of the work site)

Claims

1. a robot that performs a predetermined processing operation on a workpiece; a first camera capable of capturing images of the robot and the workpiece; a three-dimensional model storage unit for storing a three-dimensional model of the robot; a first display unit that displays an image of the robot captured by the first camera and a three-dimensional model in a superimposed manner; a control unit that, when the three-dimensional model displayed on the first display unit moves to a predetermined position and posture, moves the robot to a predetermined position and posture accordingly; A method for operating a robot manipulation system comprising: a step of first determining from what position the image of the robot to be displayed on the first display unit was captured, in order to superimpose the image of the robot captured by the first camera on a three-dimensional model and display the superimposed image on the first display unit; generating a three-dimensional model of the robot as seen from the determined position; a step of displaying the generated three-dimensional model of the robot on the first display unit by superimposing the generated three-dimensional model of the robot on an image of the robot so as to match the shape of the robot; 1. A method of operating a robotic manipulation system, comprising: The first camera includes a plurality of first cameras fixedly disposed so as to be able to capture images of the robot and the workpiece from different positions, A method for operating a robot operation system, wherein the image of the robot displayed on the first display unit is an image of one first camera selected from the plurality of first cameras.

2. The robot operation system further includes a first camera position storage unit that stores a position from which the first camera captures an image of the robot and the workpiece, 2. A method for operating a robot operation system as described in claim 1, wherein, when determining from which position the image of the robot to be displayed on the first display unit was taken, the position from which the selected one first camera that took the image took the image of the robot and the workpiece is obtained from a first camera position memory unit.

3. Furthermore, a second camera capable of capturing an image of the entire work site including the robot and the workpiece; 2. The method for operating a robot manipulation system according to claim 1, further comprising: a second display unit that displays an image captured by the second camera in real time.

Images