Teaching system, teaching method, and teaching program

Abstract

To provide a teaching system by which content to be processed can be taught to a robot easily and sensuously.SOLUTION: A teaching system includes: a display part which displays an image; an imaging part which images a visual field direction of a user; an image generation part which generates a synthesized image, which is formed by synthesizing a virtual robot and an imaged image imaged by the imaging part and is displayed on the display; a detection part which detects operation to the virtual robot by a hand of the user from the image generated by the image generation part; an instruction generation part which generates an operation instruction of an actual robot on the basis of operation detected by the detection part; and a transmission part which transmits the operation instruction to the actual robot.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a teaching system, a teaching method, and a teaching program for teaching a process to be executed by a robot. [Background technology]

[0002] Conventionally, various teaching methods have been used to make a robot perform a given task. Patent Document 1 discloses a teaching device for teaching a robot, and a method for correcting a teaching point by hand guide or jog feed. Patent Document 2 discloses a technology for providing a robot user with information on the state of the robot and operation guides in real images using an augmented reality compatible display. Patent Document 3 discloses a technology in which a user creates an operation program for a robot, and the robot in a virtual space is made to perform operations according to the program to check the operation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-093491 A [Patent Document 2] JP 2016-107379 A [Patent Document 3] JP 2014-180707 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, teaching a normal robot generally involves problems such as the complication of the user having to actually move the robot as in Patent Document 1, and the need for specialized knowledge to create an operating program as in Patent Document 3. However, in the future, there is a demand for a system that enables general users to easily operate such robots and issue instructions to the robot to execute.

[0005] Therefore, the present invention has been made in consideration of the above-mentioned demand, and aims to provide a teaching system, a teaching method, and a teaching program that can intuitively teach a robot the processing to be executed, even without specialized knowledge. [Means for solving the problem]

[0006] In order to solve the above problems, the teaching system of the present invention includes a display unit that displays an image, an imaging unit that images the user's field of view, an image generation unit that generates a composite image to be displayed on the display unit by combining a virtual robot with the image captured by the imaging unit, a detection unit that detects operations performed on the virtual robot by the user's hand from the image generated by the image generation unit, an instruction generation unit that generates operation instructions for the actual robot based on the operations detected by the detection unit, and a transmission unit that transmits the operation instructions to the actual robot.

[0007] Furthermore, a teaching method according to one aspect of the present invention is a teaching method for a robot executed by a teaching system, and includes a display step of displaying an image, an imaging step of imaging the user's field of view, an image generation step of generating a composite image by combining a virtual robot and the imaged image captured by the imaging step, which is displayed in the display step, a detection step of detecting an operation of the virtual robot by the user's hand from the image generated by the image generation step, an instruction generation step of generating an operation instruction for an actual robot based on the operation detected by the detection step, and a transmission step of transmitting the operation instruction to the actual robot.

[0008] In addition, a teaching program according to one embodiment of the present invention provides a computer of the teaching system with a display function for displaying images, an imaging function for imaging the user's field of view, an image generation function for generating a composite image by combining the virtual robot displayed by the display function with the image imaged by the imaging function, a detection function for detecting an operation of the virtual robot by the user's hand from the image generated by the image generation function, an instruction generation function for generating an operation instruction for an actual robot based on the operation detected by the detection function, and a transmission function for transmitting the operation instruction to the actual robot.

[0009] In the above teaching system, the robot has a gripping unit that grips an item, the image generation unit further generates a composite image that combines a virtual item, the detection unit identifies the item that the user has gripped by operating the gripping unit of the virtual robot, and identifies the location or target at which the gripper has released the item, and the command generation unit generates an action command for the actual robot to transport the item identified by the detection unit to the location or target.

[0010] In the above teaching system, the robot may be a robot capable of moving independently, the detection unit may detect a path of movement along which a user moves a virtual robot within the synthetic image, and the command generation unit may generate an action command for the actual robot to move along the path of movement identified by the detection unit.

[0011] In the above teaching system, the imaging unit may capture an image including an actual robot as the captured image, and the composite image may be generated by superimposing a virtual robot on the actual robot in the captured image.

[0012] In the teaching system, the robot may be a seven-axis robot. Effect of the Invention

[0013] A teaching system according to one aspect of the present invention can interpret sensory input from a user to a virtual robot, and generate and output teaching commands for processing to be executed by an actual robot. Therefore, teaching of the robot can be executed without the hassle of operating an actual robot or specialized knowledge for operating a robot. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a teaching system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of each device in the teaching system. [Diagram 3] FIG. 13 is a data conceptual diagram showing an example of the configuration of an object management table. [Figure 4] FIG. 13 is a screen diagram showing an example of a video display screen, illustrating an example of a robot operation by a user. [Diagram 5] 5 is an example of the image display screen following FIG. 4, showing an example of the operation of the robot by the user. FIG. [Figure 6] 6 is an example of the image display screen following FIG. 5, showing an example of the operation of the robot by the user. FIG. [Figure 7] 7 is an example of the image display screen following FIG. 6, showing an example of the operation of the robot by the user. FIG. [Figure 8] 8 is an example of the image display screen following FIG. 7, showing an example of the operation of the robot by the user. FIG. [Figure 9] FIG. 2 is a sequence diagram showing interactions between devices in the teaching system. [Figure 10] 10 is a flowchart showing an example of the operation of the head mounted display. [Figure 11] 11 is a flowchart illustrating an example of the operation of the information processing device. [Figure 12] 11 is a flowchart showing an example of the operation of the robot. [Figure 13] FIG. 13 is a screen diagram showing an example of operating another robot. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, a teaching system according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0016] <Embodiment> <Summary> FIG. 1 is a schematic diagram showing an outline of a teaching system 1 according to the present invention. As shown in FIG. 1, the teaching system 1 includes an information processing device 120, a head-mounted display 110 capable of viewing an image provided by the information processing device 120, and a robot 200. The information processing device 120 displays an image including the robot 200 in a virtual space, a trash can 20, and a plastic bottle 30 to a user 10 wearing the head-mounted display 110. The image may be an AR (Augumented Reality) image, a VR (Virtual Reality) image, or an MR (Mixed Reality) image. An imaging unit (camera) is mounted on the head-mounted display 110, and the image obtained from the imaging unit is transmitted to the information processing device 120. The user 10 operates the robot as an AR image by operating a virtual robot present within the field of view with his / her own finger. This operation is detected as an operation on the robot when the user's hand is included in the image captured by the imaging unit and the shape of the hand is performing a predetermined motion. For example, suppose that the user manipulates the virtual robot's hands to grip a plastic bottle 30, bring it to the trash can 20, and release it. The information processing device 120 analyzes the series of operations and interprets the user's instruction as "throw the plastic bottle in the trash can." Based on this interpretation, the information processing device 120 generates and transmits a teaching command to the actual robot 200, stating, "search for the plastic bottle and throw it in the specified trash can." This makes it possible to provide the teaching system 1 that makes it easy and convenient to teach a robot.

[0017] The information processing device 120 may provide the other user 10a with an image similar to the image being viewed by the user 10 at an angle viewable from the position of the user 10a by the other user 10a wearing the head mounted display 110a. This will be described in detail below.

[0018] <Configuration> Fig. 2 is a block diagram showing an example of a functional configuration of each device in the teaching system 1. As shown in Fig. 2, the teaching system 1 includes a head mounted display 110, an information processing device 120, and a robot 200. The head mounted display 110 and the information processing device 120 are communicatively connected to each other, and similarly, the information processing device 120 and the robot 200 are communicatively connected to each other. These connections may be wireless connections or wired connections.

[0019] 2, the head mounted display 110 includes a communication unit 111, a storage unit 112, a control unit 113, an imaging unit 114, and a display unit 115. The head mounted display 110 may be realized by a device other than a head mounted display, as long as it has a function of providing an image to the user and a function of capturing an image, and may be realized by wearable glasses, for example. Also, a configuration may be adopted in which only a camera capable of capturing an image from the user's viewpoint is attached to the user's head, and the image is provided by a projector, etc.

[0020] The communication unit 111 is a communication interface for communicating with the information processing device 120. The communication unit 111 may execute communication by either wired or wireless means, and may use any communication protocol, as long as it can communicate with the information processing device 120. The communication unit 111 receives AR video, VR video, or MR video from the information processing device 120 and transmits the video to the control unit 113, or transmits video captured by the imaging unit 114 to the information processing device 120 according to an instruction from the control unit 113.

[0021] The storage unit 112 is a storage medium having a function of storing various programs and data required for the operation of the head mounted display 110. The storage unit 112 can be realized by, for example, an HDD, an SSD, a flash memory, etc., but is not limited to these. The storage unit 112 may store a program for displaying an image provided by the information processing device 120, or the data itself of an AR image, a VR image, or an MR image.

[0022] The control unit 113 is a processor having a function of controlling each unit of the head mounted display 110. The control unit 113 controls each unit of the head mounted display 110 by executing an operation program stored in the storage unit 112. For example, the control unit 113 causes the display unit 115 to display AR (VR, MR) images transmitted from the communication unit 111, and causes the communication unit 111 to transmit images captured by the imaging unit 114 to the information processing device 120.

[0023] The imaging unit 114 is a so-called camera, and is arranged so as to be able to capture an image in the front direction of the user (the user's field of vision). The imaging unit 114 transmits the captured image to the communication unit 111. The captured images captured by the imaging unit 114 are sequentially transmitted to the information processing device 120 in real time, but this is not limited thereto, and the captured images may be transmitted after a captured image having a certain time length is obtained or after the capturing is completed.

[0024] The display unit 115 displays the AR (VR, MR) video provided by the information processing device 120 in accordance with instructions from the control unit 113. The display unit 115 can be realized by, for example, a liquid crystal monitor or an organic EL display.

[0025] The above is the configuration of the head mounted display. Next, the information processing device 120 will be described.

[0026] 2, the information processing device 120 includes a communication unit 121, a storage unit 122, and a control unit 123. The information processing device 120 is a server, a computer system, or the like that provides an image to the head mounted display 110 and generates and transmits commands to the robot 200.

[0027] The communication unit 121 is a communication interface for communicating with the head mounted display 110 and the robot 200. The communication unit 121 may execute communication by wire or wirelessly as long as it can communicate with the head mounted display 110 and the robot 200, and may use any communication protocol. The communication unit 121 transmits AR (VR, MR) images to the head mounted display 110, receives captured images captured by the imaging unit 114 from the head mounted display 110 and transmits them to the control unit 123, and transmits teaching commands to the robot 200 that define its operation. The communication unit 121 may be configured with separate circuits for communicating with the head mounted display 110 and with the robot 200, or may communicate by sharing a common communication circuit. The communication unit 121 may also communicate with devices other than the head mounted display 110 and the robot 200.

[0028] The storage unit 122 is a storage medium having a function of storing various programs and data required for the operation of the information processing device 120. The storage unit 122 can be realized by, for example, an HDD, an SSD, a flash memory, etc., but is not limited to these. The storage unit 122 may store an analysis program for analyzing the contents of instructions given by the user to the robot from the captured image provided by the head mounted display 100, and a teaching command generation program for generating teaching commands for the robot based on the analysis results.

[0029] The control unit 123 is a processor having a function of controlling each unit of the information processing device 120. The control unit 123 controls each unit of the information processing device 120 by executing an operation program stored in the storage unit 122.

[0030] The control unit 123 functions as an analysis unit that analyzes the captured image transmitted from the head mounted display 110. Specifically, the control unit 123 analyzes whether or not the user has performed some operation on each object (robot, plastic bottle, trash can) included in the image generated by the control unit 123. The control unit 123 analyzes whether or not the user's fingers are included in the captured image transmitted from the head mounted display 110. This can be realized by existing image analysis technology, and as an example, it can be realized by using pattern matching or the like. Then, when the user's fingers are included, it analyzes what kind of gesture the fingers are making. In addition, it analyzes what the fingers are operating in the user's field of view. That is, it identifies the position of the user's fingers in the captured image and the object that exists at that position. Therefore, the control unit 123 analyzes the captured image to analyze what operation the user 10 performed on which object (object).

[0031] The control unit 123 interprets what the user 10 wants the robot to do based on the analysis of the captured image. The control unit 123 then functions as a generation unit that generates a teaching command for the actual robot 200 based on the interpretation. The teaching command here may be a command including the content of what to do, where to do it, and how to do it. The control unit 123 transmits the generated teaching command to the actual robot 200 via the communication unit 121.

[0032] The control unit 123 also functions as a generating unit that generates the next AR image, VR image, or MR image from the analysis result of the captured image. When the user 10 performs some operation on an object provided from the analysis result, the control unit 123 generates an AR image or a VR image including a state after the operation. In the case of an AR (MR) image, the control unit 123 generates an AR (MR) image by synthesizing the AR (MR) image in a mode according to the position of the head mounted display 110 with the captured image received from the head mounted display 110. Then, the control unit 123 transmits the generated image to the head mounted display 110 via the communication unit 121. At this time, when the captured image includes an actual object to be generated, the control unit 123 may generate an AR image of a virtual object so as to be superimposed on the actual object.

[0033] This concludes the description of the information processing device 120. Next, the robot 200 will be described.

[0034] As shown in FIG. 2, the robot 200 includes a communication unit 201, a storage unit 202, a control unit 203, a driving unit 204, and a detection unit 205. The robot 200 may be any robot that has a communication function and can execute a function required for the purpose. The robot 200 may be a 7-axis robot, for example. The 7-axis robot is a multi-joint robot having 7 rotation axes, and is formed as a robot arm by connecting the ends of each axis. For example, the robot 200 may transport items or execute a predetermined process in a factory. The robot 200 is not limited to a 7-axis robot, and may be a robot of another form, for example, a 6-axis robot or a 5-axis robot, or may be a mobile robot or a guide robot that moves according to a predetermined algorithm or instruction, or may be a communication robot that communicates with a human being such as a user.

[0035] The communication unit 201 is a communication interface for communicating with the information processing device 120. As long as the communication unit 201 can communicate with the information processing device 120, the communication unit 201 may execute communication either wired or wirelessly, and may use any communication protocol. The communication unit 201 receives a teaching command from the information processing device 120 and transmits it to the control unit 203. In addition, the communication unit 201 may transmit sensing data collected using the operating state of each part of its own device (robot 200) and sensors, etc., to the information processing device 120 in accordance with an instruction from the control unit 203.

[0036] The storage unit 202 is a storage medium having a function of storing various programs and data required for the operation of the robot 200. The storage unit 202 can be realized by, for example, an HDD, an SSD, a flash memory, etc., but is not limited to these. The storage unit 202 may store a teaching command issued by the information processing device 120, and an execution program that interprets the teaching command and executes the command content. The storage unit 202 may also hold reference data used for pattern matching and the like required for operation.

[0037] The control unit 203 is a processor having a function of controlling each part of the robot 200. The control unit 203 controls each part of the robot 200 by executing a program stored in the storage unit 202. The control unit 203 may operate the robot 200 according to a predetermined standard. When the control unit 203 receives a teaching command, the control unit 203 executes an operation according to the situation detected by the detection unit 205 according to the teaching command. For example, when the teaching command is "throw the plastic bottle in the blue trash can," the detection unit 205 analyzes the surrounding image acquired to determine whether or not there is a plastic bottle and where the trash can is located. Then, when there is a plastic bottle, the control unit 203 drives the driving unit 204 to control the robot 200 so that the robot 200 grasps the plastic bottle and moves it to the position of the trash can 20.

[0038] Furthermore, when the control unit 203 receives a direct instruction from a user via the information processing device 120 or a remote controller, the control unit 203 may operate the robot 200 according to the instruction.

[0039] The driving unit 204 has a function of driving each part of the robot 200. If the robot 200 is a seven-axis robot, the driving unit 204 is realized by a motor that rotates each axis, and drives according to instructions from the control unit 203. If the robot 200 is a mobile robot that moves by wheels, the driving unit 204 is realized by a motor that rotates the wheels, and drives according to instructions from the control unit 203.

[0040] The detection unit 205 has a function of the robot 200 detecting information about the surroundings of the robot 200 and the state of the robot 200. The detection unit 205 can be realized by various sensors. Examples of sensors include a distance sensor (e.g., an ultrasonic sensor) capable of measuring the distance to another object, a human sensor (e.g., an infrared sensor) capable of detecting a human, and an image sensor capable of acquiring an image in a predetermined direction (e.g., the direction in which the tip of a seven-axis robot faces, or the direction of travel in the case of a mobile robot). The type of sensor to be used is selected and installed according to the function to be performed by the robot 200. The above is the configuration of the robot 200.

[0041] <Data> 3 is a conceptual data diagram showing an example of the configuration of an object data management table 300 for managing each object, which is data held by the information processing device 120. As shown in FIG 3, the object data management table 300 is information in which an object ID 301, coordinates 302, and attribute information 303 are associated with each other.

[0042] The object ID 301 is identification information for identifying each object present in an image provided to the user as an AR image or a VR image. The object ID 301 may be assigned by the information processing device 120 or its operator when, for example, the robot 200 or the plastic bottle 30 is present in a virtual space (including an augmented reality space).

[0043] The coordinates 302 are information indicating the position in the virtual space where the object indicated by the corresponding object ID 301 exists. The coordinates 302 may indicate the coordinate position of a specific point (for example, a center point) of the object, may indicate the coordinate positions of each of a plurality of feature points of the object, or may indicate the coordinate range in which the object exists. The coordinates 302 are linked to the coordinate positions of the real space in which the actual robot 200 exists. Here, linking the coordinates means establishing a correspondence relationship between the positions of each object and the positions of the walls of a room. In this case, as long as this correspondence is established, the scale of the coordinate system for defining the object, which is a virtual object, and the coordinate system of the real space may be different.

[0044] When the object indicated by the corresponding object ID 301 is operated and moved by the user 10, the coordinates 302 are corrected by the control unit 123 to the coordinate position after the movement.

[0045] The attribute information 303 is information indicating the attributes of the object indicated by the corresponding object ID 301. The attributes are information indicating the nature of the object, such as shape, color, what the object is, and state, but are not limited to these. Any other information may be used as long as it characterizes the object.

[0046] 3, an object with object ID 301 "B011" is located at coordinates (X1, Y1, Z1), and it can be seen from its attribute information 303 that it is a cylindrical blue plastic bottle. If this is robot 200, the position and shape of each arm constituting robot 200 may be specified, or information such as the position of each axis and the shaft angle (which may be called a rotation angle or a rotation angle) of drive unit 204 (motor) that drives each axis may be defined as a single piece of information in attribute information 303.

[0047] Although not shown in FIG. 3, each object ID (object) may further be associated with display data that is actually used as an image.

[0048] <Example of operation> From here, specific examples of actual operations will be described with reference to Fig. 4 to Fig. 8. Fig. 4 is an example of an image provided to the head mounted display 110 from the information processing device 120 and displayed on the head mounted display 110. Fig. 4 is also an example of an image visible to the user 10 in Fig. 1.

[0049] FIG. 4 first shows a display screen 400 on the display unit 115 of the head mounted display 110, on which an AR image generated by the information processing device 120 based on an image captured by the imaging unit 114 of the head mounted display 110 is displayed.

[0050] As shown in FIG. 4, the display screen 400 displays a robot 200a, a trash can 20, and three plastic bottles 30. Of these objects, at least the robot 200a and the plastic bottle 30 are virtual objects, and are virtual data generated by the information processing device 120. Meanwhile, the trash can 20 may be an actual object captured by the imaging unit 114, or may be virtual data generated by the information processing device 120. Although not shown in FIG. 4, the robot 200a may be displayed as a virtual robot 200a in such a manner that the actual robot 200 is actually placed at the position and the placed robot 200 is superimposed on the captured image.

[0051] Under the circumstances shown in FIG. 4, the user 10 operates the robot 200a. For example, as shown in FIG. 5, the user 10 operates the robot 200a with his / her own finger 501. FIG. 5 is a view following the display screen 400 shown in FIG. 4, and is a view showing an example of a display screen 500 displayed on the display unit 115 of the head mounted display 110. As shown in FIG. 5, in the display screen 500, it can be seen that the finger 501 of the user 10 is pinching (holding) the tip 201a of the robot 200a. This does not mean that the actual robot 200 is being grasped with the finger, but means that the user's own finger 501 is moving in the image provided by the information processing device 120 so as to appear as if the user is pinching the virtual robot 200a with the finger. The information processing device 120 detects a predetermined movement of the user's fingers (here, the movement of pinching the tip 201a of the robot 200a between two fingers) from the image captured by the imaging unit 114, and interprets this as the user grasping the tip of the robot 200a. Thereafter, the information processing device 120 generates an AR image in which the robot 200a is moved in accordance with the user's movement.

[0052] Fig. 6 shows a display screen 600 after the robot 200a has been moved in this way. As shown in Fig. 6, the user 10 can virtually move the robot 200a by moving his / her hand while keeping the tip 201a pinched between his / her fingers 501. Fig. 6 shows a display screen 600 following the display screen 500 shown in Fig. 5. Fig. 6 shows a state in which the tip 201a has been moved to the position of the plastic bottle 30a.

[0053] Here, assume that user 10 wants to have the gripping portion of tip 201a of robot 200a grip PET bottle 30a. As an example, the state shown in Fig. 6 is maintained for a predetermined time, thereby making it possible to make the gripping portion of tip 201a grip PET bottle 30a. Alternatively, although not shown, the configuration may be such that user 10 pinches fingers 501 again to make the gripping portion grip PET bottle 30a.

[0054] When the user 10 has the plastic bottle 30a held by the gripper, he or she moves the robot 200a in this state to move the tip portion 201a to above the trash can 20, as shown in a display screen 700 shown in Fig. 7. Fig. 7 shows a display screen 700 following the display screen 600 shown in Fig. 6. As shown in Fig. 7, an image of the gripper of the tip portion 201a holding the plastic bottle 30a is displayed on the display unit 115 of the head mounted display 110 of the user 10.

[0055] Then, in the state shown in FIG. 7, the user 10 moves, for example, a finger so as to move it away from the tip 201a of the robot 200a. FIG. 8 shows a display screen 800 following the display screen 700 shown in FIG. 7. As shown in FIG. 8, the user 10 moves his / her finger 501 so as to move it away from the tip 201a of the robot 200a. As can be understood by comparing FIG. 7 and FIG. 8, the user 10 moves his / her finger away from the tip 201a. The information processing device 120 analyzes the captured image showing this movement, accepts it as an input from the user 10, and releases the grip of the plastic bottle 30a by the gripping part of the robot 200a. Then, as a result, the information processing device 120 provides the head mounted display 110 with an image of the plastic bottle 30a falling into the trash can 20.

[0056] On the other hand, the information processing device 120 can recognize the process that the user 10 wants the robot 200 to execute by having the user 10 virtually operate the robot 200a in the manner as shown in Fig. 4 to Fig. 8. When the user 10 causes the virtual robot 200a to perform the actions shown in Fig. 4 to Fig. 8, the information processing device 120 can interpret, as an example, that the user 10 wants the robot 200 to execute a process of throwing a plastic bottle in a trash can. Then, according to this interpretation, the information processing device 120 can transmit a teaching command to the robot 200 to throw the plastic bottle in a trash can, and the robot 200 can learn the process of throwing the plastic bottle in a trash can.

[0057] <Operation> Next, a description will be given of the operation of each device in the teaching system 1. First, a description will be given of interactions between each device in the teaching system 1, that is, the head mounted display 110, the information processing device 120, and the robot 200, with reference to FIG.

[0058] Fig. 9 is a sequence diagram showing an example of communication between devices related to the teaching system 1. As shown in Fig. 9, the head mounted display (HMD) 110 transmits a captured video (image) captured by the imaging unit 114 to the information processing device 120 (step S901).

[0059] When the information processing device 120 receives the captured image, it generates an AR image including the virtual robot 200a from the captured image and transmits the AR image to the head mounted display 110 (step S902).

[0060] The head mounted display 110 displays the received AR image (step S903). The head mounted display 110 transmits the captured image including the action (operation) of the user 10 with respect to the virtual robot 200a to the information processing device 120 (step S904).

[0061] The information processing device 120 analyzes the content of the received captured image (step S905). Then, from the analyzed content, it interprets the process that the user wants the actual robot 200 to execute, and generates a teaching command according to the interpretation (step S906). The information processing device 120 transmits the generated teaching command to the robot 200 (step S907).

[0062] When the robot 200 receives the teaching command, the robot 200 executes the command indicated by the teaching command (step S908). As a result, in the teaching system 1, the user can teach the robot 200 to execute a process by simply intuitively operating the virtual robot 200a generated by augmented reality or virtual reality.

[0063] From here, an example of the operation of each device for realizing the sequence shown in Fig. 9 will be described. Fig. 10 is a flowchart showing an example of the operation of the head mounted display 110.

[0064] 10, the imaging unit 114 performs shooting (capturing images) according to instructions from the control unit 113 (step S1001). The imaging unit 114 transmits the captured image (or video, i.e., video) to the control unit 113. The control unit 113 transmits the captured image to the information processing device 120 via the communication unit 111 (step S1002).

[0065] The communication unit 111 of the head mounted display 110 receives the image including the virtual robot. The communication unit 111 transmits the received image to the control unit 113. The control unit 113 then causes the display unit 115 to display the transmitted image (step S1003). This allows the head mounted display 110 to provide the user 10 wearing the display with an AR image (or a VR image) including the virtual robot 200a.

[0066] The control unit 113 of the head mounted display 110 judges whether or not an end input has been received from the user (step S1004). The end input from the user is, for example, pressing a power button (not shown) or an input to end the provision of AR or VR images, but is not limited to these. If an end input from the user has been received (YES in step S1004), the process ends. If an end input from the user has not been received (NO in step S1004), the head mounted display 110 returns to the process of step S1001 and executes the subsequent processes.

[0067] By repeating the processing of steps S1001 to S1003, the head mounted display 110 can transmit to the information processing device 120 an image captured of an input being made to a displayed virtual object while displaying an AR image or a VR image including an object to be operated.

[0068] Fig. 11 is a flowchart showing an example of the operation of the information processing device 120 for realizing the exchange shown in Fig. 9. The flowchart shown in Fig. 11 starts from a state in which the AR video or VR video generated by the control unit 123 has already been provided.

[0069] 11, the communication unit 121 of the information processing device 120 receives a captured image captured by the head mounted display 110 from the head mounted display 110 (step S1101). The communication unit 121 transmits the received captured image to the control unit 123.

[0070] When the captured image is transmitted from the communication unit 121, the control unit 123 analyzes the content of the captured image (step S1102). It is preferable to use a general image recognition technique for analyzing the image.

[0071] The control unit 123 determines whether or not there is an operation on the virtual robot 200a based on the analysis result (step S1103). An operation on the virtual robot 200a may be, for example, a designation of a predetermined position of the robot by the finger of the user 10, an instruction to move the robot arm, or a designation of execution of a specific process that the robot can execute, and these can be identified by detecting a specific motion by the user's finger.

[0072] If it is determined that there is an operational input to the virtual robot 200a (YES in step S1103), the control unit 123 generates a composite image including the virtual robot 200a in a state after the operation (step S1104).If it is determined that there is no operational input to the virtual robot 200a (NO in step S1103), the control unit 123 generates a composite image in which the state of the virtual robot 200a is maintained (step S1105).

[0073] Then, the control unit 123 transmits the generated composite image to the head mounted display 110 via the communication unit 121 (step S1106).

[0074] The control unit 123 determines whether the input to the virtual robot 200a has ended (step S1107). The control unit 123 may determine that the input to the virtual robot 200a has ended by, for example, receiving an end input from the user to the information processing device 120, detecting an end input from the user 10 to the head mounted display 110, or detecting a specific operation of the user's hand or finger in the captured video.

[0075] If it is determined that the input to the virtual robot 200a is not complete (NO in step S1107), the control unit 123 returns to the process of step S1101 and executes the subsequent processes. If it is determined that the input to the virtual robot 200a is complete (YES in step S1107), the control unit 123 interprets, from a series of operations by the user 10 on the virtual robot 200a, what process the user 10 has caused the robot 200 to execute. Then, from the interpretation result, a teaching command, which is an operation command for the robot 200, is generated (step S1108). The control unit 123 transmits the generated teaching command to the actual robot 200 via the communication unit 121 (step S1109), and ends the process.

[0076] This enables the information processing device 120 to output instructions for the actual robot 200 without detailed input from the user (teaching by actually moving the robot 200 by hand or teaching by detailed specification of each drive axis at each time).

[0077] Finally, FIG. 12 is a flow chart showing the operation of the actual robot 200.

[0078] 12, the communication unit 201 of the robot 200 receives a teaching command from the information processing device 120 (step S1201). The communication unit 201 transmits the received teaching command to the control unit 203.

[0079] The control unit 203 stores the received teaching command in the storage unit 202 (step S1202).

[0080] The control unit 203 interprets the contents of the teaching command and specifies the processing contents to be executed (step S1203). The detection unit 205 transmits sensing data indicating the detected surrounding situation to the control unit 203. In this way, the control unit 203 grasps the surrounding situation of the robot 200 (step S1204).

[0081] The control unit 203 judges whether the transmitted sensing data matches the situation indicated by the teaching command (S1205). If it is judged that the surrounding situation matches the situation indicated by the teaching command (YES in step S1205), that is, if it is judged that the teaching command should be executed, the control unit 123 controls the drive unit 204 to execute the command contents indicated by the teaching command (step S1206), and ends the process.

[0082] This allows the robot 200 to execute an action desired by the user in accordance with a teaching command generated according to intuitive instructions, without detailed instructions from the user.

[0083] <Summary> Conventional robot teaching involves difficulties for ordinary users, such as heavy labor when operating an actual robot, or highly specialized knowledge when teaching by program. In contrast, according to the teaching system 1 of the present embodiment, a process to be executed by an actual robot can be taught by periodically operating a virtual robot, so that robot teaching can be performed more simply and intuitively than before. Therefore, the information processing device 120 of the present embodiment and the teaching system 1 including the information processing device 120 can promote the improvement of convenience in robot teaching.

[0084] <Modification> The devices according to the above-described embodiments are not limited to the above-described embodiments, and may be realized by other methods. Various modified examples will be described below.

[0085] (1) In the above embodiment, the information processing device 120 and the head mounted display 110 have been described as separate devices, but this is not limited thereto. The head mounted display 110 and the information processing device 120 may be formed integrally. In other words, the head mounted display 110 may have all the functions that the information processing device 120 has.

[0086] The head mounted display 110 may also have some of the functions of the information processing device 120 (for example, a function to generate an image). In this case, the control unit 113 of the head mounted display 110 analyzes the operation contents being performed by the user 10 based on the captured image captured by the imaging unit 114, generates an image, and displays the image on the display unit 115. On the other hand, the control unit 113 may transmit the captured image to the information processing device 120 via the communication unit 111, and the control unit 123 of the information processing device 120 may interpret the processing contents that the user 10 wants the robot to execute from the captured image and generate a teaching command.

[0087] (2) In the above embodiment, an example of operating the tip 201a of the robot 200a has been shown, but the operation of the robot 200a is not limited to the operation of the tip 201a. For example, the operation of instructing the rotation of a joint may be performed by pointing one of the joints constituting the robot 200a with one finger (e.g., index finger) and rotating the joint around the one finger with the other two fingers (e.g., thumb and middle finger). Similarly, the operation of instructing the rotation of an axis constituting the robot by specifying one of the axes instead of a joint may be performed by a similar operation.

[0088] (3) In the above embodiment, as a specific example, an example was shown in which a seven-axis robot was taught to grasp and carry an object, but as mentioned above, the robot is not limited to a seven-axis robot, and the work it performs is not limited to grasping and carrying. Fig. 13 shows an example in which a so-called automatic cleaning robot is used as another example of robot 200.

[0089] FIG. 13 shows an image visually recognized by the user 10a and displayed on the display unit 115 of the head-mounted display 110. Here, an example of using an automatic cleaning robot 200b is shown as an example of the robot 200. Some automatic cleaning robots 200b have a function of checking the surrounding environment with their own sensors and automatically generating a movement path, but cleaning along the generated movement path may be inefficient. Therefore, the user can specify a movement path using the teaching system 1. In the example of FIG. 13, the user points to the robot 200b at the position indicated by the dotted line with his / her finger 1301 for a predetermined time. At this time, the user does not need to touch the actual robot 200, and it is sufficient that the user appears to be touching the virtual robot 200b. When the information processing device 120 detects that the user has touched the robot 200b for a predetermined time or more, it generates an image in which the robot 200b moves to follow the user's finger. The user moves the finger 1301 to instruct the movement path along which the robot 200b should move. Here, it is assumed that the finger is moved as indicated by the arrow 1302. Then, the information processing device 120, based on the operation, determines the correspondence between the display screen shown in FIG. 13 and the actual position so as to move along the position indicated by the arrow 1302, specifies the actual position (actual coordinate system) of the path indicated by the arrow 1302, and generates a teaching command to perform cleaning along the movement path in the specified coordinate system. Then, the information processing device 120 transmits the generated teaching command to the robot 200. This allows the automatic cleaning robot 200 to perform cleaning along the specified path, thereby enabling efficient cleaning.

[0090] In this embodiment, the robot 200b can be operated by pointing at the robot 200b with a finger for a predetermined period of time or more, but this is merely an example, and the robot 200b may be held and moved by moving the finger as if pinching the robot 200b as in the above embodiment, and the movement path of the robot 200b may be specified by moving the finger as if releasing the pinching motion. The method of operating the virtual robot 200 may be any method as long as the user can operate it at intervals.

[0091] (4) In the above embodiment, it has been explained that an AR image in which the virtual robot 200a is superimposed on the real robot 200 may be displayed. Furthermore, it may be configured so that each time a user operates the virtual robot 200a, the real robot 200 is simultaneously made to move in accordance with the operation (a time lag for transmission may occur).

[0092] (5) Although not particularly described in the above embodiment, when more detailed operation is desired in teaching the robot, the information processing device 120 may provide the head mounted display 110 with an enlarged image of the location where the user is working (for example, the location where the user is operating the virtual robot 200a by performing a predetermined input). This enlarged image may be displayed as an enlarged image of the operation location in a separate window using VR video, or may be displayed as an enlarged image in a manner that zooms in on the currently viewed image. Therefore, by providing the enlarged image, the user can easily execute instruction input to the robot 200a. Note that the teaching system 1 may include an imaging camera for imaging the actual robot's working location (the gripping portion in the above embodiment) in order to provide an enlarged image of the working location. As an example, this imaging camera may be provided at the tip 201a of the robot 200, or may be provided separately from the robot 200 so as to be able to capture an image from outside. Furthermore, in the case where the imaging camera is provided separately from the robot 200, a moving mechanism may be provided for continuing to capture an image of the work location of the robot (following the work location) when the work location moves. For example, in the teaching system 1, the imaging camera may be attached to a robot arm separate from the robot 200, and the robot arm may move so that the imaging of the work location can be continued. This modification is particularly useful when operating the actual robot 200 in real time using a virtual robot 200a, as shown in the above modification (4).

[0093] (6) In the above embodiment, the teaching system generates a teaching command by executing a teaching program or the like by the information processing device 120, but this may be realized by a logic circuit (hardware) or a dedicated circuit formed in an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)) or the like in the device. These circuits may be realized by one or more integrated circuits, and the functions of the multiple functional units shown in the above embodiment may be realized by one integrated circuit. LSIs are sometimes called VLSIs, super LSIs, ultra LSIs, etc., depending on the degree of integration.

[0094] The teaching program may be recorded on a processor-readable recording medium, and the recording medium may be a "non-transient tangible medium", such as a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The teaching program may be supplied to the processor via any transmission medium (such as a communication network or a broadcast wave) capable of transmitting the teaching program. The present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the walking teaching program is embodied by electronic transmission.

[0095] The teaching program can be implemented using, for example, a scripting language such as C, C++, ActionScript, or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), or a markup language such as HTML5.

[0096] (7) The above-described embodiment and modifications may be combined as appropriate as necessary to achieve desired functions. [Explanation of symbols]

[0097] 1 Teaching system 110, 110a Head-mounted display 111 Communications Department 112 Storage section 113 Control Unit 114 Imaging unit 115 Display section 120 Information processing device 121 Communications Department 122 Storage section 123 Control Unit 200 Robots 201 Communications Department 202 Storage section 203 Control Unit 204 Drive unit 205 Detection unit

Claims

1. A display unit for displaying an image; An imaging unit that captures an image in the user's visual field; an image generating unit that generates a composite image to be displayed on the display unit by combining a virtual robot with an image captured by the imaging unit; a control unit that detects an operation of the virtual robot by the user's hand from the image generated by the image generation unit; a command generation unit that generates an actual robot operation command based on the operation detected by the control unit; a transmission unit that transmits the operation command to the actual robot, The image generation unit further generates the composite image by combining a virtual object, The control unit analyzes an operation performed by the user on the virtual object, The physical robot acquires and analyzes images of its surroundings to identify objects physically present in its surroundings; The command generation unit interprets an operation performed by the user on the virtual object as an action to be performed by the actual robot, and generates an action command for causing the actual robot to perform an action based on the interpretation when an object identical to the virtual object actually exists. Teaching system.

2. The imaging unit captures an image including the actual robot as the captured image, The synthetic image is generated by superimposing the virtual robot on the actual robot in the captured image.

2. The teaching system according to claim 1 .

3. A teaching method for a robot executed by a teaching system, comprising: a display step of displaying the image; An imaging step of imaging a user's visual field direction; an image generating step of generating a composite image by combining a virtual robot to be displayed in the display step with the captured image captured in the imaging step; a control step of detecting an operation of the virtual robot by the user's hand from the image generated by the image generation step; a command generation step of generating an actual robot operation command based on the operation detected by the control step; and a transmitting step of transmitting the operation command to the actual robot, The image generating step further includes generating the composite image by combining a virtual object, The control step includes analyzing an operation performed by the user on the virtual object, The physical robot acquires and analyzes images of its surroundings to identify objects physically present in its surroundings; The command generation step interprets an operation performed by the user on the virtual object as an action to be performed by the actual robot, and generates a command for causing the actual robot to perform an action based on the interpretation when an object identical to the virtual object actually exists. Teaching methods.

4. The computer of the teaching system A display function for displaying an image; An imaging function for imaging a user's visual field; an image generating function that generates a composite image by combining a virtual robot displayed by the display function with an image captured by the imaging function; a control function for detecting an operation of the virtual robot by the user's hand from the image generated by the image generation function; a command generation function that generates a command for an actual robot operation based on the operation detected by the control function; A transmission function of transmitting the operation command to the actual robot is realized, The image generation function further generates the composite image by combining a virtual object, The control function analyzes an operation performed by the user on the virtual object, The physical robot acquires and analyzes images of its surroundings to identify objects physically present in its surroundings; The command generation function interprets an operation that the user wants the actual robot to perform based on the operation performed on the virtual object, and generates a command for causing the actual robot to perform an operation based on the interpretation when an object identical to the virtual object actually exists. Teaching program.

Images