Augmented reality simulation device and robot system

Abstract

Simulation device (10) comprising: an image sensor (11) that captures an image of a real space including an actual robot (20) and a peripheral device (22) arranged in the vicinity of the actual robot (20); an augmented reality display section (12) that displays a virtual robot (23) superimposed on the actual robot (20) shown in the captured image and a virtual workpiece (W1; W2); a workpiece management section (13) that manages a position of a moving workpiece (W1; W2), wherein: the moving workpiece (W1; W2) is the virtual workpiece (W1), and the workpiece management section (13) manages the constantly changing position of the virtual workpiece (W1) based on a preset initial position, movement direction and movement amount of the virtual workpiece (W1), or the moving workpiece (W1; W2) is the actual workpiece (W2), and the workpiece management section (13) manages the constantly changing position of the actual workpiece (W2) based on information from a first detection sensor (31) and / or a second detection sensor (32); and a motion control section (14) that controls a motion of the virtual robot (23) based on the position of the moving workpiece (W1; W2).

Description

[0001] The present invention relates to a simulation apparatus and a robot system, and more particularly to an augmented reality simulation apparatus and a robot system.

[0002] When performing a given operation using a robot, it is common practice to perform an offline simulation beforehand and then perform motion confirmation on-site using an actual robot. For example, the literature described below is well-known regarding such robot simulation.

[0003] JP 2004 - 243 516 A discloses a method for superimposing computer-generated information into an image of the real environment captured by an image receiving device of a viewing device. In this method, an assessment is made regarding the position and orientation or posture of an image receiving unit, and robot-specific information (reference coordinate system, robot axis, etc.) is superimposed onto the image of the real environment according to this provision.

[0004] JP 6 385 627 B1 discloses that three-dimensional data of an orientation coordinate axis, which includes a straight line passing through control points set on a robot, is displayed in a virtual space defined by three-dimensional data of a structure based on the three-dimensional data of the robot and the three-dimensional data of the structure.

[0005] JP 2019 - 025620 A and JP 2019 - 042843 A disclose that a virtual robot is displayed superimposed on an actual robot shown in an image, the virtual robot is moved, and then the actual robot is moved. Furthermore, the display of a virtual workpiece is also described.

[0006] DE 10 2019 006 800 A1, DE 10 2019 002 928 A1, DE 10 2019 002 898 A1, DE 10 2016 123 945 A1, WO 2021 / 101 522 A1 and US 2018 / 0 311 825 A1 disclose further relevant prior art.

[0007] The situation at the location where the robot is used does not always correspond to the situation when the simulation is performed offline. Thus, when the robot is actually moved, it may affect peripheral devices. In particular, when a robot performs an operation on a moving workpiece, it is difficult to predict in advance in a simulation how the robot will actually move, because the pre-taught robot trajectory is corrected based on the position of the workpiece.

[0008] In addition, when a robot is moved for teaching purposes, an operation button may be pressed incorrectly or the robot may be moved without realizing that the robot's movement speed was set to a high speed, which may also cause the robot to interfere with peripheral devices.

[0009] The task is to provide a technology that can detect whether a robot interferes with peripheral devices in a simulation of a robot performing an operation on a moving workpiece.

[0010] One solution to the problem consists in a simulation device having the features of patent claim 1, which comprises an image sensor that captures an image of a real space with an actual robot and a peripheral device arranged in the vicinity of the actual robot, an augmented reality display section that displays a virtual robot in a superimposed manner on the actual robot shown in the captured image, a workpiece management section that manages a position of a moving workpiece, and a motion control section that controls a motion of the virtual robot based on the position of the workpiece.

[0011] A further solution to the problem consists in a simulation device having the features of claim 5, which comprises an image sensor that captures an image of a real space with an actual robot and a peripheral device arranged in the vicinity of the actual robot, an augmented reality display section that displays a virtual robot in a superimposed manner on the actual robot shown in the captured image, an indexing section that indexes a movement of the virtual robot in the image, and a movement control section that controls the movement of the virtual robot based on the indexing of the indexing section.

[0012] A further solution to the problem consists in a robot system having the features of claim 10, which comprises an actual robot, an image sensor that captures an image of a real space including the actual robot and a peripheral device arranged in the vicinity of the actual robot, an augmented reality display section that displays a virtual robot superimposed on the actual robot shown in the captured image, a workpiece management section that manages a position of a moving workpiece, and a motion control section that controls a motion of the virtual robot based on the position of the workpiece.

[0013] A further solution to the problem consists in a robot system having the features of patent claim 13, which comprises an actual robot, an image sensor which captures an image of a real space including the actual robot and a peripheral device arranged in the vicinity of the actual robot, an augmented reality display section which displays a virtual robot in a superimposed manner on the actual robot shown in the captured image, an indexing section which indexes a movement of the virtual robot in the image, and a movement control section which controls the movement of the virtual robot based on the indexing of the indexing section. Fig. 1 is a schematic structural view of a simulation device according to an embodiment. Fig. 2 is a schematic structural view showing a modified example of the simulation device. Fig. 3 is a functional block diagram of a robot system according to an embodiment. Fig. 4 is a view showing an operator cursor for indexing the movement of a virtual robot. Fig. 5 is a functional block diagram of a modified example of the robot system.

[0014] Embodiments of the present invention will be described below with reference to the accompanying drawings. In the drawings, like or similar components are designated by like or similar reference numerals. Furthermore, the embodiments described below do not limit the technical scope of the invention or the definitions of the terms described in the claims. Note that in this specification, the term "workpiece" includes both an actual workpiece and a virtual workpiece, and the term "robot" includes both an actual robot and a virtual robot.

[0015] Fig. 1 shows the schematic structure of a simulation device 10 according to the present embodiment. The simulation device 10 includes a computing device including a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The simulation device 10 may be a mobile terminal such as a tablet, a smartphone, a wearable device, or a laptop, or a stationary terminal such as a desktop. The simulation device 10 is an augmented reality display device that superimposes an image of a real space captured by an image sensor (not shown) with a virtual object, and may also function as a robot teaching device that generates a motion program of the robot.The simulation device 10 includes an augmented reality display section 12 including a processor performing augmented reality processing, a display panel, etc. To superimpose a captured image with a virtual object, the augmented reality display section 12 preliminarily sets three-dimensional data of the virtual image, sets a characteristic portion or characteristic mark of an actual object, calibrates the image sensor, and preliminarily detects the relative relationship (relative position and relative posture) between the characteristic portion or characteristic mark and the image sensor.Furthermore, the augmented reality display section 12 detects the position and posture of the characteristic portion or the characteristic mark from an actually captured image and displays a virtual object superimposed on the image based on the previously detected relative relationship between the detected position and posture of the characteristic portion or the characteristic mark and the image sensor.

[0016] The augmented reality display device 12 of the present example captures an image showing an actual robot 20 and a peripheral device 22 arranged around the actual robot 20, and displays a virtual robot 23 superimposed on the actual robot 20 shown in the captured image. The actual robot 20 and the virtual robot 23 are industrial robots, such as articulated robots or parallel-link robots, and may be humanoid or similar. Furthermore, the peripheral device 22 is, for example, a machine tool and includes any objects arranged around the actual robot 20. A transport section 30 that transports actual workpieces (not shown) may be shown in the captured image. The transport section 30 includes, for example, a conveyor belt, an AGV (Automated Guided Vehicle), etc.

[0017] The augmented reality display section 12 of the present example also displays a virtual workpiece W1 being transported by the transport section 30 and moving in the direction of an arrow X superimposed on the image, and the movement of the virtual robot 23 with respect to the virtual workpiece W1. At this time, virtual workpieces W1 can flow at the same movement speed and in the same feed amount just as in actual operation. Therefore, even if it is difficult to reproduce the actual workpiece flow just as in actual operation, since the movement of the virtual robot 23 with respect to the virtual workpiece W1 can be reproduced in the real space image, the presence or absence of interference with the peripheral device 22 by the actual robot 20 can be detected in advance.Note that when performing a simulation, a robot controller 21 can mechanically lock the actual robot 20. This can prevent interference with the peripheral device 22 by the actual robot 20 even if the actual robot 20 is moved incorrectly for teaching purposes.

[0018] Fig. 2 shows a modified example of the simulation device 10. The simulation device 10 of the present example differs from the one described above in that an image obtained by flowing actual workpieces W2 is captured instead of virtual workpieces W1. The simulation device 10 moves the virtual robot 23 with respect to the actual workpiece W2 shown in the image. The simulation device 10 can detect the position and, if necessary, the movement amount of the actual workpiece W2 based on the information from a first detection sensor 31 and / or the information from a second detection sensor 32. The first detection sensor 31 is a fixed camera capable of detecting, for example, the position and movement amount of the actual workpiece W2, but it may also be a photoelectric sensor, a contact sensor, etc.that detects the arrival of the actual workpiece W2 when the movement amount of the actual workpiece W2 is known. Furthermore, the second detection sensor 32 is a rotary encoder mounted, for example, on the rotary shaft of the transport section 30, and can be combined with the first detection sensor 31, such as a photoelectric sensor or contact sensor, when the movement amount of the actual workpiece W2 is unknown. By detecting the position and, if applicable, the movement amount of the actual workpiece W2, the movement of the virtual robot 23 with respect to the actual workpiece W2 can be displayed.When the flow of actual workpieces W2 can be reproduced in the same way as in an actual operation, the presence or absence of interference of the peripheral device 22 by the actual robot 20 can be confirmed more accurately by displaying the movement of the virtual robot 23 with respect to the actual workpiece W2.

[0019] Fig. Figure 3 shows the functional blocks of a robot system 1 according to the present embodiment. Note that the "sections" in the functional blocks are formed by hardware and / or software. The robot system 1 includes, in addition to the simulation device 10, an actual robot 20 and a robot controller 21. The simulation device 10 sends the simulated robot motion program to the robot controller 21. The robot controller 21 controls the actual robot 20 according to the received motion program.

[0020] In addition to the image sensor 11 and augmented reality display section 12, the simulation device 10 includes a workpiece management section 13 that manages the position of the moving workpiece, and a motion control section 14 that controls the movement of the virtual robot based on the position of the workpiece. When, as shown in Fig. 1, virtual workpieces W1 flow, the workpiece management section 13 manages the constantly changing position of the virtual workpiece W1 based on the preset home position, the preset movement direction, the preset movement amount, etc. of the virtual workpiece W1. On the other hand, as shown in Fig. 2, actual workpieces W2 flow, the workpiece management section 13 manages the constantly changing position of the actual workpiece W2 based on the information of the first detection sensor 31 and / or the second detection sensor 32. The motion control section 14 controls the movement of the virtual robot 23 based on the position of the virtual workpiece W1 or the actual workpiece W2.

[0021] The simulation device 10 may further include an interference detection section 15 that detects the presence or absence of interference with the peripheral device 22 by the virtual robot 23. Before detecting the presence or absence of interference, three-dimensional data of the peripheral device 22 may be set in advance, and if the image sensor 11 is a three-dimensional sensor such as a ToF (Time-of-Flight) sensor or laser scanner, the three-dimensional data of the peripheral device 22 may be acquired in advance based on information from the image sensor 11. The interference detection section 15 is capable of detecting the presence or absence of interference with the peripheral device 22 by the virtual robot 23 using a known interference detection method.For example, the interference detection section 15 may project a convex polyhedron having a simplified shape of the peripheral device 22 and a convex polyhedron having a simplified shape of the virtual robot 23 onto the XYZ planes, determine the intersection of the two convex polyhedrons formed by the projection using a plane scanning method, and, when the intersection is detected on all of the XYZ planes, determine that interference occurs.

[0022] The simulation device 10 may further include a color-changing section 16 that changes the color of the virtual robot 23 when the interference is detected. For example, the color-changing section 16 changes the color of the virtual robot 23 to red, and the augmented reality display section 12 displays the virtual robot 23, whose color has been changed to red, superimposed on the image. This allows the presence or absence of interference with the peripheral device 22 by the actual robot 20 to be easily visually recognized.

[0023] The simulation device 10 may further include an indexing section 17 that indicates the movement of the virtual robot 23 in the image. The indexing section 17 is formed, for example, by a touch panel, a processor, etc., and displays an operating cursor that indicates the movement of the virtual robot 23 superimposed on the image. Fig. 4 shows an operation cursor 40. The operation cursor 40 is formed, for example, using the origin of the tool coordinate system (e.g., the tool center point (TCP) or the like) or any control point as the origin through the XYZ axes. The indexing section 17 moves the operation cursor 40 based on information from the touch panel. The indexing of the indexing section 17 includes the position and posture of the virtual robot 23, and the motion control section 14 controls the movement of the virtual robot 23 based on the indexing of the indexing section 17. The position of the virtual robot 23 may be, for example, the origin of a flange coordinate system, and the posture of the virtual robot 23 may be, for example, the orientation of the tool coordinate system with respect to the origin of a base coordinate system.

[0024] Referring again to Fig. 3, the simulation device 10 may further include a motion recording section 18 that records the motion of the virtual robot 23 based on the indexing of the indexing section 17, and a motion program generation section 19 that generates a motion program of the robot based on the recorded motion. The simulation device 10 may, in turn, simulate the generated motion program in an augmented reality or send the generated motion program to the robot controller 21. The robot controller 21 controls the actual robot 20 based on the received motion program.

[0025] Fig.5 shows a modified example of the robot system 1. The robot system 1 of the present example differs from the above-described one in that the workpiece management section 13, the motion control section 14, and the interference detection section 15 are provided not in the simulation device 10 but in the robot controller 21. By adopting such a configuration, the simulation device 10 can perform augmented reality simulation by utilizing the existing functions of the robot controller 21.

[0026] According to the embodiments described above, the presence or absence of interference with the peripheral device 22 by the actual robot 20 can be detected in advance in a simulation in which a robot performs an operation on a moving workpiece. This reduces operational man-hours because layout confirmation tasks can be performed in the same situation as in on-site operation.

[0027] The program for executing the flowchart described above may be recorded and provided in a computer-readable non-volatile recording medium, e.g., a CD-ROM, etc.

Claims

[1] Simulation device (10) comprising: an image sensor (11) that captures an image of a real space including an actual robot (20) and a peripheral device (22) arranged in the vicinity of the actual robot (20); an augmented reality display section (12) that displays a virtual robot (23) superimposed on the actual robot (20) shown in the captured image and a virtual workpiece (W1; W2); a workpiece management section (13) that manages a position of a moving workpiece (W1; W2), wherein: the moving workpiece (W1; W2) is the virtual workpiece (W1), and the workpiece management section (13) manages the constantly changing position of the virtual workpiece (W1) based on a preset initial position, movement direction and movement amount of the virtual workpiece (W1), or the moving workpiece (W1; W2) is the actual workpiece (W2), and the workpiece management section (13) manages the constantly changing position of the actual workpiece (W2) based on information from a first detection sensor (31) and / or a second detection sensor (32); and a motion control section (14) that controls a motion of the virtual robot (23) based on the position of the moving workpiece (W1; W2). [2] The simulation device (10) according to claim 1, further comprising an interference detection section (15) that detects the presence or absence of interference of the peripheral device (22) by the virtual robot (23). [3] The simulation device (10) according to claim 2, further comprising a color changing section (16) that changes a color of the virtual robot (23) when the impairment is detected. [4] Simulation device (10) according to one of claims 1 to 3, wherein a transport section (30) which transports the workpiece (W1; W2) is shown in the image. [5] Simulation device (10) comprising: an image sensor (11) that captures an image of a real space including an actual robot (20) and a peripheral device (22) arranged in the vicinity of the actual robot (20); an augmented reality display section (12) that displays a virtual robot (23) superimposed on the actual robot (20) shown in the captured image; an indexing section (17) that indexes a movement of the virtual robot (23) in the image, wherein the indexing section (17) comprises a touch panel, wherein the indexing section (17) displays an operation cursor (40) that indicates the movement of the virtual robot (23) superimposed on the touch panel, and moves the operation cursor (40) based on information from the touch panel; and a motion control section (14) that controls the motion of the virtual robot (23) based on the indexing of the indexing section (17). [6] The simulation device (10) according to claim 5, further comprising an interference detection section (15) that detects interference of the peripheral device (22) by the virtual robot (23). [7] The simulation device (10) according to claim 6, further comprising a color changing section (16) that changes a color of the virtual robot (23) when the impairment is detected. [8] Simulation device (10) according to one of claims 5 to 7, wherein the indexing of the indexing section (17) comprises a position and posture of the virtual robot (23). [9] Simulation device (10) according to one of claims 5 to 8, further comprising: a motion recording section (18) which records the motion of the virtual robot based on the indexing of the indexing section, and a movement program generating section (19) that generates a movement program based on a recording of the movement. [10] Robot system (1) comprising: an actual robot (20); an image sensor (11) that captures an image of a real space including the actual robot (20) and a peripheral device (22) arranged in the vicinity of the actual robot (20); an augmented reality display section (12) that displays a virtual robot (23) superimposed on the actual robot (20) shown in the captured image and a virtual workpiece (W1; W2); a workpiece management section (13) that manages a position of a moving workpiece (W1; W2), wherein: the moving workpiece (W1; W2) is the virtual workpiece (W1), and the workpiece management section (13) manages the constantly changing position of the virtual workpiece (W1) based on a preset initial position, movement direction and movement amount of the virtual workpiece (W1), or the moving workpiece (W1; W2) is the actual workpiece (W2), and the workpiece management section (13) manages the constantly changing position of the actual workpiece (W2) based on information from a first detection sensor (31) and / or a second detection sensor (32); and a motion control section (14) that controls a motion of the virtual robot (23) based on the position of the moving workpiece (W1; W2). [11] Robot system (1) according to claim 10, wherein the workpiece (W1; W2) is an actual workpiece (W2), and the robot system (1) further comprises a detection sensor (31, 32) that detects a position of the actual workpiece (W2). [12] The robot system (1) according to claim 11, wherein the detection sensor (32) further detects a movement amount of the actual workpiece (W2). [13] Robot system (1) comprising: an actual robot (20); an image sensor (11) that captures an image of a real space including the actual robot (20) and a peripheral device (22) arranged in the vicinity of the actual robot (20); an augmented reality display section (12) that displays a virtual robot (23) superimposed on the actual robot (20) shown in the captured image; an indexing section (17) that indexes a movement of the virtual robot (23) in the image, wherein the indexing section (17) comprises a touch panel, wherein the indexing section (17) displays an operation cursor (40) that indicates the movement of the virtual robot (23) superimposed on the touch panel, and moves the operation cursor (40) based on information from the touch panel; and a motion control section (14) that controls the motion of the virtual robot (23) based on the indexing of the indexing section (17).

Images