Control system, control device, and control method

Abstract

To enable an operator to control a robot remotely according to their intentions. [Solution] A control system for a robot, comprising: an environmental sensor; a first instruction generation unit that generates a first instruction which is an operation instruction for the robot in real space; a virtual space generation unit that generates a virtual space based on the detection result of the environmental sensor and the state of the robot; a second instruction generation unit that generates a second instruction which is an operation instruction for the robot in the virtual space; a mode control unit that controls the system to switch between a first mode in which the robot is operated based on the first instruction and a second mode in which the robot is operated based on the second instruction; and a robot operation unit that operates the robot based on the first instruction or the second instruction.

Description

Technical Field

[0001] The present invention relates to a control system, a control device, and a control method.

Background Art

[0002] Conventionally, a technique is known in which an operator remotely operates a robot using a virtual space. For example, Patent Document 1 describes a technique that generates a virtual image in which a virtual tool corresponding to a robot tool and a virtual workpiece corresponding to a workpiece are arranged in a virtual space, and generates robot teaching points for executing processing on the workpiece by the tool in response to the virtual tool processing the virtual workpiece in the virtual space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technique, when an operator remotely operates a robot in real time, delays occur such as the delay before an operation instruction from the operator reaches the robot, and the delay before the operator recognizes a state change that has occurred in the robot. As a result, with the conventional technique, the operator may not be able to operate the robot as intended in some cases.

[0005] The present invention has been made in consideration of such circumstances, and one of the objects thereof is to provide a control system, a control device, and a control method that can enable an operator to operate a robot as intended when the operator remotely operates the robot.

Means for Solving the Problems

[0006] The control system, control device, and control method according to this invention employ the following configuration. (1) A control system according to one aspect of the present invention is a robot control system comprising: one or more environmental sensors installed on the robot or the surrounding environment of the robot and detecting the surrounding environment; an instruction generation unit that generates a first instruction which is an operation instruction for the robot in real space; a virtual space generation unit that generates a virtual space based on the detection results of the one or more environmental sensors and the state of the robot; a second instruction generation unit that generates a second instruction which is an operation instruction for the robot in the virtual space; a mode control unit that controls the system to switch between a first mode in which the robot is operated based on the first instruction and a second mode in which the robot is operated based on the second instruction; and a robot operation unit that operates the robot based on the first instruction in the first mode or the second instruction in the second mode switched by the mode control unit.

[0007] (2): In the embodiment of (1) above, the first instruction generation unit generates the first instruction based on the detection result of the operator sensor that detects the movements of the robot operator.

[0008] (3) In the embodiment of (2) above, the first instruction generation unit modifies the first instruction based on the detection results of one or more environmental sensors in addition to the detection results of the operator sensor.

[0009] (4): In the embodiment of (1) above, the control system further comprises a display unit for displaying images to the operator of the robot, wherein in the first mode the display unit displays real-space images obtained from one or more environmental sensors in real time, and in the second mode a virtual space image representing the virtual space is displayed statically.

[0010] (5) In the embodiment of (1) above, when the first instruction or the second instruction is generated, the first instruction or the second instruction is converted into an operation instruction for the robot in the robot's workspace, and the robot is operated based on the converted operation instruction.

[0011] (6): A control device according to another aspect of the present invention is a control device for a robot, comprising: an instruction receiving unit that receives a first instruction which is an instruction to operate the robot in real space, or a second instruction which is an instruction to operate the robot generated in a virtual space; a virtual space generation unit that generates the virtual space based on the detection results of one or more environmental sensors installed in the robot or the surrounding environment of the robot and detecting the surrounding environment, and the state of the robot; a mode control unit that controls the robot to switch between a first mode in which the robot is operated based on the first instruction and a second mode in which the robot is operated based on the second instruction; and a robot operation unit that operates the robot based on the first instruction in the first mode or the second instruction in the second mode switched by the mode control unit.

[0012] (7) Another aspect of the present invention is a control method for a robot, which receives a first instruction, which is an operation instruction for the robot in real space, or a second instruction, which is an operation instruction for the robot generated in a virtual space, generates the virtual space based on the detection results of one or more environmental sensors installed in the robot or the surrounding environment of the robot and detecting the surrounding environment, and the state of the robot, controls to switch between a first mode in which the robot is operated based on the first instruction and a second mode in which the robot is operated based on the second instruction, and operates the robot based on the first instruction in the switched first mode or the second instruction in the second mode. [Effects of the Invention]

[0013] According to embodiments (1) to (7), when an operator remotely controls a robot, it is possible to operate the robot according to the operator's intentions.

[0014] According to embodiments (4) and (5), the operator can direct the robot's operation on a static virtual space image, thereby enabling more precise control of the robot. [Brief explanation of the drawing]

[0015] [Figure 1] This diagram illustrates the interaction between the robot 2, controlled by the control system 1, and the operator Us's HMD4. [Figure 2] This is an illustrative configuration diagram of control system 1. [Figure 3] This diagram illustrates the transition relationship between the real space mode and the virtual space mode as controlled by the mode control unit 120. [Figure 4] This is a sequence diagram showing an example of the operation method for robot 2 in direct mode. [Figure 5] This figure shows an example of how to operate Robot 2 in teaching mode. [Figure 6] This figure shows another example of how to operate Robot 2 in teaching mode. [Figure 7] This is a sequence diagram showing an example of the operation method for robot 2 in teaching mode. [Figure 8] This figure shows an example of how to operate Robot 2 in virtual space support mode. [Figure 9] This figure shows another example of how to operate Robot 2 in virtual space support mode. [Figure 10] This figure shows another example of how to operate Robot 2 in virtual space support mode. [Figure 11] This is a sequence diagram showing an example of how to operate Robot 2 in virtual space support mode. [Modes for carrying out the invention]

[0016] [Summary] Hereinafter, embodiments of the control system, the control device, and the control method of the present invention will be described with reference to the drawings. FIG. 1 is a diagram for explaining the interaction between a robot 2 controlled by a control system 1 and the HMD 4 of an operator Us. As shown in FIG. 1, in the remote operation space, the operator Us wears, for example, an HMD (Head Mount Display) 4 on the head, and wears an operation unit 5 (5L, 5R) such as a data glove on the hand. The robot 2 includes end effectors 21 (a first end effector 21L, a second end effector 21R), and hands 211 (211L, 211R). An environment sensor 3E is installed in the robot work space. Note that the environment sensor may be attached to the robot 2. In the example of FIG. 1, the environment sensor 3L is attached to the hand 211L, and the environment sensor 3R is attached to the hand 211R. In the following description, when the environment sensor 3E, the environment sensor 3L, and the environment sensor 3R are not distinguished, they are also collectively referred to as the environment sensor 3.

[0017] The target object obj is an object to be operated by the robot 2. The target object obj is any arbitrary object such as a plastic bottle, a bottle, a wire, a belt, or a lever, for example. FIG. 1 shows an example in which the target object obj is a plastic bottle, which includes a body part and a cap. The operator Us, for example, moves the hand or fingers wearing the operation unit 5 while viewing the image displayed on the HMD 4, thereby remotely operating the robot 2 to manipulate the target object obj.

[0018] An example of the operation content is, when the target object obj is a plastic bottle or a bottle, as shown in FIG. 1, attaching the cap to the body part to close it, or unscrewing the cap from the body part, and the like. As another example, when the target object obj is a wire, the operation content is hooking the wire onto a hook, or unhooking the wire from the hook, and the like. As still another example, when the target object obj is a tie-down belt, the operation content is tightening the tie-down belt, or loosening the tie-down belt, and the like.

[0019] [Control system configuration] Next, an example of the configuration of the control system 1 will be described. Figure 2 is an exemplary configuration diagram of the control system 1. As shown in Figure 2, the control system 1 includes, for example, a robot 2, an environmental sensor 3, an HMD 4, an operating unit 5, and a control device 100.

[0020] Robot 2 includes, for example, a first end effector 21L and a second end effector 21R. The first end effector 21L includes, for example, a hand 211L, an actuator 212L, and a sensor 213L. The second end effector 21R includes, for example, a hand 211R, an actuator 212R, and a sensor 213R. Robot 2 may also include a communication unit, a power supply unit, a body, legs, a head, etc., which are not shown. In the following description, when the first end effector 21L and the second end effector 21R are not distinguished, they will also be referred to as end effector 21. Similarly, when the hand 211L and hand 211R are not distinguished, they will also be referred to as hand 211, when the actuator 212L and actuator 212R are not distinguished, they will also be referred to as actuator 212, and when the sensor 213L and sensor 213R are not distinguished, they will also be referred to as sensor 213. Robot 2 transmits and receives various information with the control device 100 via a wired or wireless network NW.

[0021] The hand 211 is, for example, a gripper. Alternatively, the hand 211 may have multiple fingers, and each finger may have a joint. Actuators 212 are attached to each joint. Sensors 213 are, for example, a 6-axis sensor attached to a joint, a tactile sensor attached to a finger, etc. The 6-axis sensor detects force in 3 axes (x, y, z) and moment in 3 axes (α, β, γ). The robot 2 may also have a drive circuit to drive the actuators 212.

[0022] As shown in Figure 1, the environmental sensor 3 is installed, for example, on the hand 211 of the robot 2 and in the robot's workspace. The environmental sensor 3 is, for example, an RGB-D camera and acquires RGB (red, green, blue) information and depth information. The information is acquired, for example, at predetermined time intervals. The environmental sensor 3 transmits the acquired information to the HMD4 or control device 100 directly or indirectly via the robot 2 through a wired or wireless network NW.

[0023] The HMD4 includes, for example, an image display unit 41 and a gaze detection unit 42. The image display unit 41 acquires and displays real-space images obtained by the environmental sensor 3 in real time via the control device 100. If information is acquired at predetermined time intervals, the image display unit 41 may acquire and display images at the same predetermined time intervals. In addition, for example, the image display unit 41 can also display virtual space images generated by the control device 100, which will be described later. By wearing the HMD4, the operator Us can visually confirm the robot work environment in which the robot 2 is working. The gaze detection unit 42 detects the direction and movement of the operator Us's gaze. The HMD4 also includes a communication unit, a power supply unit, etc., which are not shown. The HMD4 transmits and receives various information with the control device 100 via a wired or wireless network NW.

[0024] The control unit 5 includes, for example, a sensor 51. The control unit 5 is, for example, a data glove. The sensor 51 is, for example, an accelerometer, a gyroscope, a pressure sensor, a tactile sensor, etc. These sensors track the movements of the operator's hands and fingers and detect them as operation instructions for the robot 2. In other words, the control unit 5 functions as an input device for transmitting the hand movements of the operator Us to the robot 2 and is used to reflect the hand movements of the operator Us in the operation of the robot 2.

[0025] The operator Us can manipulate the target object obj through the robot 2 by moving their hands and fingers while wearing a data glove and viewing a real-space or virtual-space image displayed on the HMD4. The operation unit 5 may further include motion capture sensors and an IMU (inertial measurement unit) for tracking the operator Us's body movements (e.g., 3D position). The operation unit 5 also includes a communication unit (not shown). The operation unit 5 transmits operation instructions to the control device 100 via a wired or wireless network NW. The operation unit 5 is an example of the "first instruction generation unit" and the "second instruction generation unit" in the claims.

[0026] The control device 100 operates the robot 2 based on operation instructions received via the network NW. More specifically, when the control device 100 receives an operation instruction from the operation unit 5, it converts the received operation instruction into the robot's coordinate system. Here, the operation instruction received from the operation unit 5 includes a first instruction given to the operator Us while they are viewing a real-space image displayed on the HMD4, and a second instruction given to the operator Us while they are viewing a virtual-space image displayed on the HMD4. Regardless of which instruction is given, the control device 100 converts the operation instruction in the real-space or virtual-space coordinate system into the robot's coordinate system and transmits it to the robot 2 via the network NW. The robot 2 performs actions based on the received operation instruction.

[0027] For the sake of explanation, the control device 100 is described as being composed of a single device (server), but the present invention is not limited to such processing, and the control device 100 may be composed of multiple devices for load balancing. For example, the control device 100 may be composed of a device responsible for virtual space-related processing that generates a virtual space based on real-space images received from the environmental sensor 3 and provides it to the HMD4, and a device responsible for robot operation-related processing that generates operation instructions for the robot 2 based on operation instructions received from the operation unit 5 and provides them to the robot 2. In another embodiment, the control device 100 may be integrated with the HMD4.

[0028] [Control device configuration] The control device 100 includes, for example, a transmitting / receiving unit 110, a mode control unit 120, an instruction receiving unit 130, a virtual space generation unit 140, and a robot operation unit 150. These components are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or flash memory (a storage device with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed when the storage medium is inserted into a drive device. The control device 100 also includes a communication unit, a power supply unit, etc., which are not shown.

[0029] The transmitting / receiving unit 110 transmits and receives various information with the robot 2 and HDM 4 via a wired or wireless network NW. The control device 100 receives various information from the environmental sensor 3 and the operation unit 5 via a wired or wireless network NW.

[0030] The mode control unit 120 controls the operation of the robot 2 to switch between a real-space mode, in which the robot 2 is operated based on a first instruction, which is an operation instruction for the robot 2 in real space, and a virtual-space mode, in which the robot 2 is operated based on a second instruction, which is an operation instruction for the robot 2 generated in virtual space. More specifically, for example, the gaze detection unit 42 detects when the operator Us directs their gaze to a predetermined area for mode switching among the images displayed on the HMD4, and at the same time, the mode control unit 120 switches between the real-space mode and the virtual-space mode when the operator Us performs an operation for mode switching using the operation unit 5 (for example, by moving a finger in the direction of the predetermined area for mode switching). The real-space mode is an example of the "first mode" in the claims, and the virtual-space mode is an example of the "second mode" in the claims.

[0031] Figure 3 is a diagram illustrating the transition relationship between the real-space mode and the virtual-space mode by the mode control unit 120. As shown in Figure 3, the real-space mode includes, for example, the direct mode and the real-space support mode, and the virtual-space mode includes, for example, the teaching mode and the virtual-space support mode. As shown in Figure 3, in this embodiment, the direct mode is activated in the initial stage when the operator Us puts on the HMD4 and the control unit 5 and starts operating the robot 2. Subsequently, in response to the operator Us's operation of the control unit 5, the mode control unit 120 transitions the direct mode to the real-space support mode or the teaching mode. When the direct mode transitions to the teaching mode, the mode control unit 120 transitions the teaching mode to the virtual-space support mode in response to further operation of the control unit 5 by the operator Us. The details of the direct mode, real-space support mode, teaching mode, and virtual-space support mode will be described below.

[0032] [Direct Mode] Direct mode is a mode in which the operator Us operates the robot 2 using the control unit 5 while checking real-time spatial images transmitted from the environmental sensor 3. Figure 4 is a sequence diagram showing an example of the flow of operation of the robot 2 in direct mode. In the sequence diagram shown in Figure 4, "operator device" represents at least one of the HMD 4 and the control unit 5, and is an example of "operator sensor" in the claims.

[0033] First, the operator device receives a selection of direct mode from operator Us (step S100). Next, the operator device notifies the control device 100 of the selection of direct mode received from operator Us (step S102). Next, the transmitting / receiving unit 110 receives a real-space image from the environment sensor 3 from the robot 2 (step S104). Next, the transmitting / receiving unit 110 displays the real-space image on the HMD 4 (step S106). Next, the operator device receives operation instructions for the robot 2 on the real-space image from operator Us (step S108).

[0034] Next, the operator device transmits the received operation instruction (first instruction) for robot 2 to the control device 100 (step S110). Next, the instruction receiving unit 130 receives the first instruction, and the robot operation unit 150 converts the first instruction into an operation instruction for robot 2 in the robot coordinate system (step S112). Next, the transmitting / receiving unit 110 transmits the converted operation instruction to robot 2 (step S114). Next, robot 2 operates based on the received operation instruction (step S116). This completes the processing shown in this sequence diagram.

[0035] Thus, in direct mode, through the processing from step S104 to step S116, the operator Us can check the real-space image from the environmental sensor 3 in real time and use the control unit 5 to give operation instructions to the robot 2 in the real-space image. However, for example, if the robot's workspace is in outer space and is far from the remote control space, a discrepancy may occur between the timing when the environmental sensor 3 transmits the real-space image and the timing when the operator Us checks the real-space image, gives operation instructions to the robot 2, and the robot 2 operates, due to information communication delays, for example.

[0036] For example, if operator Us moves their hand to have robot 2's hand 211 grasp a target object obj, even though robot 2's hand 211 and the target object obj are actually close enough, the real-space image may still display the hand 211 and the target object obj as being far apart. As a result, operator Us may move their hand more than necessary, causing hand 211 to collide with or pass through the target object obj. In other words, direct mode is suitable for rough operations such as moving robot 2 closer to a target object obj located at a distance, but may not be suitable for fine movements of the target object obj (e.g., grasping, tightening, loosening, hooking, detaching, rotating, etc.).

[0037] [Real-world support mode] The real-space support mode is a mode in which the operator Us operates the robot 2 using the control unit 5 while checking the real-space image transmitted in real time from the environmental sensor 3 and the support information provided on the real-space image. Here, the support information is, for example, information obtained by reading an AR marker attached to the target object obj. An AR marker is a tag with a specific pattern or code (a distinctive visual mark). When the operator Us selects the real-space support mode, the control device 100 can read the AR marker included in the real-space image transmitted in real time from the environmental sensor 3 and extract the position (relative position from the robot 2, etc.) and orientation (orientation, etc.) of the AR marker in three-dimensional space.

[0038] When an AR marker is read, the control device 100 can, for example, display the position information of the AR marker and / or the robot 2 (e.g., coordinate information and distance information) on the HMD4, or superimpose information about the target object obj to which the AR marker is attached as overlay information onto the real-space image. Furthermore, if the target object obj is, for example, a plastic bottle or glass bottle, the control device 100 can superimpose the rotation axis of the target object obj as overlay information on the HMD4, and also display options for constraining the robot 2 to rotate the hand 211 along that rotation axis (i.e., to open the lid of the target object obj). This is possible because the position and orientation of the target object obj are determined by reading the AR marker. Also, for example, the control device 100 can display options on the HMD4 for constraining the robot 2 to translate in the direction of the target object obj, regardless of the type of target object obj.

[0039] The direct mode and real-space support mode described above allow the operator Us to operate the robot 2 while viewing real-space images from the environmental sensor 3 in real time. However, as mentioned above, the direct mode may not be suitable for fine movements of the target object obj due to communication delays, for example, when the robot workspace and the remote control space are far apart. Furthermore, in this embodiment, the real-space support mode is a mode that utilizes AR markers, but AR markers are not necessarily provided to all target objects obj that are to be operated, and depending on the shape of the target object obj (for example, a flexible object that deforms easily), it may not be possible to attach AR markers to it.

[0040] [Teaching Mode] Against the backdrop of the above circumstances, the teaching mode and virtual space support mode described below generate a virtual space image from a real space image and display it on the HMD4. The operator Us then gives operation instructions on the virtual space image, and subsequently converts the instructions given on the virtual space image into instructions for robot 2 in the robot coordinate system. Unlike the real space image, which is updated in real time, the virtual space image generated at this time is a static image. Therefore, the operator Us can give robot 2 detailed instructions on the target object obj without being affected by communication delays.

[0041] Figure 5 shows an example of how to operate robot 2 in teaching mode. As an example, Figure 5 shows a situation in which operator Us is operating robot 2, which grasps wire W with hand 211L, to attach wire W to hook F. The left side of Figure 5 shows the real-space image displayed on HMD4 in direct mode. As mentioned above, in direct mode, HMD4 displays the real-space image in real time, so due to communication delays, there may be a discrepancy between the positional relationship of wire W and hook F displayed in the real-space image and the actual positional relationship of wire W and hook F. For this reason, direct mode may not be suitable for the operation of attaching wire W to hook F.

[0042] Therefore, the operator Us operates the control unit 5 to switch from direct mode to teaching mode. The control unit 5 notifies the control device 100 via the network NW that the user has switched from direct mode to teaching mode. When the control device 100 receives the information indicating that the user has switched from direct mode to teaching mode, the mode control unit 120 switches from direct mode to teaching mode, and the virtual space generation unit 140 generates a three-dimensional virtual space image corresponding to the latest real space image received from the environment sensor 3. More specifically, the virtual space generation unit 140 generates a virtual space by arranging objects depicted in the latest real space image as mesh objects in three-dimensional space, based on the state of the robot 2, such as its position, posture, and orientation. The right side of Figure 5 shows the virtual space image obtained by the virtual space generation unit 140 transforming the real space image on the left side of Figure 5.

[0043] When generating a virtual space, the virtual space generation unit 140 may extract only important objects from the objects depicted in the real-world image (for example, objects that the robot 2 will operate on, or objects that could become obstacles to the robot 2's movement) using known image recognition methods or pre-trained models. The transmission / reception unit 110 transmits the generated virtual space image to the HMD4 for display.

[0044] Operator Us operates robot 2 using the control unit 5 while observing the virtual space image displayed on the HMD4 (more precisely, simulating the movement of robot 2 on the virtual space image without actually moving robot 2). As described above, since the virtual space image displayed on the HMD4 is a static virtual space image, operator Us can simulate the movement of robot 2 as intended without being affected by the discrepancy between the positional relationship of wire W and hook F displayed on the HMD4 and the actual positional relationship of wire W and hook F.

[0045] When operator Us confirms an operation in teaching mode, the operation unit 5 transmits the operation confirmed in teaching mode (an operation on a static virtual space image) as a second instruction to the control device 100. When the transmitting / receiving unit 110 receives the second instruction, the robot operation unit 150 converts the second instruction into an operation instruction for robot 2 in the robot coordinate system. The transmitting / receiving unit 110 transmits the converted operation instruction to robot 2, and robot 2 operates based on the received operation instruction. Through the above process, the operation of attaching wire W to hook F, which was simulated on a static virtual space image in teaching mode, can be reflected as the operation of robot 2 in the robot workspace, enabling fine movements that are difficult to execute in direct mode.

[0046] Figure 6 shows another example of how to operate robot 2 in teaching mode. As an example, Figure 6 shows a situation where operator Us is operating robot 2 to grasp the tie-down belt B with both hands 211L and 211R and then attempting to loosen (pull) it. The left side of Figure 6 shows the virtual space image displayed on the HMD4 in teaching mode. The action of grasping the tie-down belt may also be difficult to perform in direct mode due to communication delays, which can cause a discrepancy between the positional relationship between both hands 211L and 211R and the tie-down belt B displayed in the real space image and the actual positional relationship between both hands 211L and 211R and the tie-down belt B.

[0047] Therefore, as shown on the left side of Figure 6, operator Us operates robot 2 using the control unit 5 while checking the virtual space image displayed on HMD4, and grasps the tie-down belt B with both hands 211L and 211R of robot 2 on the static virtual space image. Once robot 2 has successfully grasped the tie-down belt B with both hands 211L and 211R, operator Us confirms the operation using the control unit 5.

[0048] The operation unit 5 transmits the operation confirmed in teaching mode as a second instruction to the control device 100, and the robot operation unit 150 converts the second instruction into an operation instruction for robot 2 in the robot coordinate system. The transmitting / receiving unit 110 transmits the converted operation instruction to robot 2, and robot 2 operates based on the received operation instruction. As a result, robot 2 grasps the tie-down belt B in the robot workspace.

[0049] Subsequently, operator Us operates the control unit 5 to switch the teaching mode to direct mode. At this stage, since both hands 211L and 211R of robot 2 are gripping the tie-down belt B in real space, operator Us, in direct mode, operates the control unit 5 (for example, by extending both arms) to send a first instruction to the control device 100 to loosen the tie-down belt B. When the transmitting / receiving unit 110 receives the first instruction, the robot operating unit 150 converts the first instruction into an operation instruction for robot 2 in the robot coordinate system. The transmitting / receiving unit 110 transmits the converted operation instruction to robot 2, and robot 2 operates based on the received operation instruction.

[0050] Through the above process, the action of gripping the tie-down belt B, which was simulated on a static virtual space image in teaching mode, can be reflected as the action of robot 2 in the robot workspace, enabling fine movements that are difficult to perform in direct mode. Then, it is possible to seamlessly transition to direct mode and finally perform the desired operation (i.e., the operation of loosening the tie-down belt after gripping it).

[0051] Figure 7 is a sequence diagram showing an example of the operation method for robot 2 in teaching mode. First, the operator device receives the selection of the teaching mode from operator Us (step S200). Next, the operator device notifies the control device 100 of the selection of the teaching mode received from operator Us (step S202). Next, the transmitting / receiving unit 110 receives the real-space image from the environment sensor 3 from robot 2 (step S204).

[0052] Next, the virtual space generation unit 140 generates a virtual space image based on the received real space image (step S206). Next, the transmitting / receiving unit 110 displays the virtual space image on the HMD4 (step S208). Next, the operator device receives operation instructions for the robot 2 on the virtual space image from operator Us (step S210).

[0053] Next, the operator device transmits the received operation instruction (second instruction) for robot 2 to the control device 100 (step S212). Next, the instruction receiving unit 130 receives the second instruction, and the robot operation unit 150 converts the second instruction into an operation instruction for robot 2 in the robot coordinate system (step S214). Next, the transmitting / receiving unit 110 transmits the converted operation instruction to robot 2 (step S216). Next, robot 2 operates based on the received operation instruction (step S218). This completes the processing shown in this sequence diagram.

[0054] [Virtual Space Support Mode] As described above, the operator Us can switch from direct mode to teaching mode to simulate detailed movements in a static virtual space, and, if necessary, check a preview image before translating it into operation instructions for robot 2 in the robot workspace. However, even if simulation in a static virtual space is possible, it may be preferable to provide further support in the virtual space to more accurately realize the operation intended by the operator Us. For example, when attaching (or detaching) wire W to hook F, in teaching mode, wire W is also displayed as a static image, so the operator Us may not be confident that the operation simulated in teaching mode will cause robot 2 to perform the intended action (i.e., whether wire W can actually be attached to hook F).

[0055] In this embodiment, the virtual space support mode provides not only a simulation function in the virtual space, but also a function to draw a virtual object obj_v that can be referenced by the operator Us when manipulating the target object obj. One form of the virtual object obj_v is an object that moves in conjunction with the movement of the robot 2, even in a static virtual space. By manipulating the robot 2 while referencing the virtual object obj_v, the operator Us can grasp the movement trajectory (target trajectory) of the target object obj. This allows the operator Us to perform operations more accurately as intended.

[0056] Figure 8 shows an example of how to operate robot 2 in virtual space support mode. The situation shown on the left side of Figure 8 is the same as the situation shown on the right side of Figure 5. That is, in the situation shown on the left side of Figure 8, operator Us is operating robot 2, which is grasping wire W with hand 211L, to attach wire W to hook F. This is a fine movement, so the operator switches from direct mode to teaching mode. However, as mentioned above, the virtual space image generated in teaching mode is a static image. Therefore, when simulating the movement of robot 2 in teaching mode, wire W is also a static image, and thus wire W cannot be moved in teaching mode.

[0057] Therefore, as shown in the center of Figure 8, the virtual space support mode enables the drawing of a virtual object obj_v that is linked to the movement of the robot 2 in the virtual space. More specifically, for example, first, the gaze detection unit 42 detects that the operator Us has directed their gaze to a predetermined area for drawing virtual objects in the virtual space image displayed on the HMD4, and the operator Us operates the control unit 5 to draw the virtual object obj_v in the virtual space image by gesture. In the case of the center of Figure 8, the operator Us operates the control unit 5 to trace the wire W, and the HMD4 draws the virtual object obj_v accordingly.

[0058] In this case, the HMD4 (or virtual space generation unit 140) may display the virtual object obj_v drawn freehand by the operator Us as is, or it may correct and display the virtual object obj_v so that it matches an object whose degree of agreement with the virtual object obj_v drawn by the operator Us is at or above a predetermined level. For example, in the case of Figure 8, if the degree of agreement between the virtual object obj_v drawn by the operator Us and the wire W object is at or above a predetermined level, the HMD4 (or virtual space generation unit 140) may correct the virtual object obj_v so that it matches the wire W object.

[0059] When operator Us completes the drawing of the virtual object obj_v by, for example, operating the control unit 5, operator Us operates robot 2 on the virtual space image using the control unit 5 while confirming the virtual space image and virtual object obj_v displayed on the HMD4. As a result, as shown in the right side of Figure 8, the virtual object obj_v moves in conjunction with the movement of robot 2, and operator Us can grasp the movement trajectory of hook F. In other words, operating robot 2 while referring to the virtual object obj_v is equivalent to specifying the movement trajectory of the virtual object obj_v. When robot 2 successfully attaches hook F to wire W on the static virtual space image, operator Us confirms the operation using the control unit 5. At this time, before operator Us confirms the operation, the HMD4 may display a preview image of the operations from when robot 2 transitions to teaching mode until it attaches wire W to hook F, and operator Us may confirm the operation after confirming the preview image.

[0060] When operator Us confirms an operation in virtual space support mode, the operation unit 5 transmits the confirmed operation (an operation on a static virtual space image) in virtual space support mode to the control device 100 as a second instruction. When the transmitting / receiving unit 110 receives the second instruction, the robot operation unit 150 converts the second instruction into an operation instruction for robot 2 in the robot coordinate system. The transmitting / receiving unit 110 transmits the converted operation instruction to robot 2, and robot 2 operates based on the received operation instruction. Through the above process, the operation of attaching wire W to hook F, which was simulated on a static virtual space image while referring to the virtual object obj_v in virtual space support mode, can be reflected as the operation of robot 2 in the robot workspace, enabling fine movements that are difficult to execute in direct mode.

[0061] In the above explanation, the process from drawing the virtual object obj_v to simulating the operation of robot 2 is performed within the virtual space support mode. However, the virtual space support mode may be specialized solely for drawing the virtual object obj_v. In that case, the operator Us would operate the control unit 5 to switch the virtual space support mode to teaching mode after completing the drawing of the virtual object obj_v, and then simulate the operation of robot 2 in teaching mode.

[0062] Figure 9 shows another example of how to operate robot 2 in virtual space support mode. As an example, Figure 9 shows a situation in which operator Us is operating robot 2, which is gripping a wire W attached to hook F, in an attempt to remove the wire W.

[0063] As shown in the left part of Figure 9, first, operator Us operates the control unit 5 to trace the wire W, and the HMD4 (or virtual space generation unit 140) draws a virtual object obj_v accordingly. Once operator Us completes drawing the virtual object obj_v by, for example, operating the control unit 5, operator Us checks the virtual space image and the virtual object obj_v displayed on the HMD4, and uses the control unit 5 to operate the robot 2 on the virtual space image. As a result, as shown in the right part of Figure 9, the virtual object obj_v moves in conjunction with the movement of the robot 2, and operator Us can check whether the robot 2 can perform the intended action (i.e., whether the wire W can be released from the hook F).

[0064] Figures 8 and 9 illustrate an example of drawing a virtual object obj_v on a virtual space image that can be visually referenced by the operator Us. However, the form of the virtual object obj_v is not limited to this. Another form of the virtual object obj_v may be a virtual object obj_v that serves as a reference for constraining the movement of robot 2. In this case, the constraint on movement may be, for example, a constraint on rotational motion with the drawn virtual object obj_v as the axis of rotation, or a constraint on translational motion in the direction of the drawn virtual object obj_v.

[0065] Figure 10 shows another example of how to operate Robot 2 in virtual space support mode. Figure 10 shows a situation where operator Us is trying to get Robot 2 to open the lid of an object obj, such as a plastic bottle or glass bottle. The object obj may also be a valve, doorknob, etc.

[0066] In Figure 10, operator Us intends to rotate the hand 211R, which is grasping the lid, in a clockwise direction in order to open the lid. Therefore, in virtual space support mode, operator Us draws a virtual object obj_v in the direction that will be the axis of rotation, for example, with the lid to be rotated as the starting point. At this time, operator Us may specify the direction of rotation by gesture, for example.

[0067] When operator Us completes the drawing of the virtual object obj_v by, for example, operating the control unit 5, operator Us confirms the virtual space image and the virtual object obj_v displayed on the HMD4 and uses the control unit 5 to operate robot 2 on the virtual space image. At this time, since the operation of robot 2 is constrained, robot 2's hand 211R rotates around the virtual object obj_v in the virtual space. When operator Us confirms the rotation operation in, for example, the preview image, they confirm the operation using the control unit 5.

[0068] When operator Us confirms an operation in virtual space support mode, the operation unit 5 transmits the confirmed operation (an operation on a static virtual space image) in virtual space support mode to the control device 100 as a second instruction. When the transmitting / receiving unit 110 receives the second instruction, the robot operation unit 150 converts the second instruction into an operation instruction for robot 2 in the robot coordinate system. The transmitting / receiving unit 110 transmits the converted operation instruction to robot 2, and robot 2 operates based on the received operation instruction. This makes it possible to reflect the operation of rotating the lid cap, which was simulated on a static virtual space image with the virtual object obj_v as the reference in virtual space support mode, as the operation of robot 2 in the robot workspace, enabling fine movements that are difficult to execute in direct mode.

[0069] Figure 11 is a sequence diagram showing an example of the operation method for robot 2 in virtual space support mode. First, the operator device receives the selection of the teaching mode from operator Us (step S300). Next, the operator device notifies the control device 100 of the selection of the teaching mode received from operator Us (step S302). Next, the transmitting / receiving unit 110 receives the real-space image from the environment sensor 3 from robot 2 (step S304).

[0070] Next, the virtual space generation unit 140 generates a virtual space image based on the received real space image (step S306). Next, the transmitting / receiving unit 110 displays the virtual space image on the HMD4 (step S308). Next, the operator device accepts the selection of a virtual space support mode from operator Us (step S310). Next, the operator device accepts the drawing of a virtual object from operator Us (step S312). Next, the operator device accepts the operation of robot 2 on the virtual space image from operator Us (step S314).

[0071] Next, the transmitting / receiving unit 110 transmits the received operation instruction (second instruction) for the robot 2 to the control device 100 (step S316). Next, the instruction receiving unit 130 receives the second instruction, and the robot operation unit 150 converts the second instruction into an operation instruction for the robot 2 in the robot coordinate system (step S318). Next, the transmitting / receiving unit 110 transmits the converted operation instruction to the robot 2 (step S320). Next, the robot 2 operates based on the received operation instruction (step S322). This completes the processing shown in this sequence diagram.

[0072] The division of functions between the operator device, the control device 100, and the robot 2 as described above may be changed as appropriate. For example, information processing by the operator device may be transferred to the control device 100, and the operator device may only have information input / output functions. Alternatively, for example, the control device 100 may be mounted on the robot 2, and the functions of this embodiment may be realized only between the operator device and the robot 2.

[0073] As described above, this embodiment allows switching between a first mode in which the robot is operated based on a first instruction, which is an operation instruction for the robot in real space, and a second mode in which the robot is operated based on a second instruction, which is an operation instruction for the robot generated in virtual space. Furthermore, according to this embodiment, the second mode further includes a mode that allows the operator to draw virtual objects that can be referenced for robot operation in virtual space. This makes it possible for the operator to operate the robot as intended when remotely controlling it.

[0074] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, The system comprises one or more processors connected to the storage medium, The one or more processors execute the computer-readable instructions to: The system receives a first instruction, which is an operation instruction for the robot in real space, or a second instruction, which is an operation instruction for the robot, generated in virtual space. The virtual space is generated based on the detection results of one or more environmental sensors installed in the robot or the surrounding environment of the robot and detecting the surrounding environment, and the state of the robot. The system is controlled to allow switching between a first mode in which the robot is operated based on the first instruction and a second mode in which the robot is operated based on the second instruction. Based on the first instruction or the second instruction, the robot is operated. A control device configured in such a way.

[0075] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0076] 1. Control System 2 Robots 3. Environmental Sensors 4 HMD 5 Control section 100 Control device 110 Transmitter / Receiver 120 Mode Control Unit 130 Instruction Reception Department 140 Virtual Space Generation Unit 150 Robot Operation Unit

Claims

1. A robot control system, One or more environmental sensors installed in the robot or the surrounding environment of the robot to detect the surrounding environment, A first instruction generation unit that generates a first instruction, which is an operation instruction for the robot in real space, A virtual space generation unit that generates a virtual space based on the detection results of one or more environmental sensors and the state of the robot, In the virtual space, a second instruction generation unit generates a second instruction, which is an operation instruction for the robot. A mode control unit that controls the robot to switch between a first mode in which the robot is operated based on a first instruction and a second mode in which the robot is operated based on a second instruction, The system includes a robot operation unit that operates the robot based on the first instruction in the first mode or the second instruction in the second mode, which is switched by the mode control unit. Control system.

2. The first instruction generation unit generates the first instruction based on the detection result of the operator sensor that detects the movements of the robot operator. The control system according to claim 1.

3. The first instruction generation unit modifies the first instruction based on the detection results of one or more environmental sensors in addition to the detection results of the operator sensor. The control system according to claim 2.

4. The robot further includes a display unit that displays an image to the operator of the robot, In the first mode, the display unit displays real-space images obtained from one or more environmental sensors in real time, while in the second mode, it statically displays virtual space images representing the virtual space. The control system according to claim 1.

5. When the first instruction or the second instruction is generated, the robot operation unit converts the first instruction or the second instruction into an operation instruction for the robot in the robot's workspace, and operates the robot based on the converted operation instruction. The control system according to claim 1.

6. A robot control device, An instruction receiving unit that receives a first instruction, which is an operation instruction for the robot in real space, or a second instruction, which is an operation instruction for the robot generated in virtual space, A virtual space generation unit that generates the virtual space based on the detection results of one or more environmental sensors installed in the robot or the surrounding environment of the robot and detecting the surrounding environment, and the state of the robot, A mode control unit that controls the robot to switch between a first mode in which the robot is operated based on a first instruction and a second mode in which the robot is operated based on a second instruction, The system includes a robot operation unit that operates the robot based on the first instruction in the first mode or the second instruction in the second mode, which is switched by the mode control unit. Control device.

7. A method for controlling a robot, The system receives a first instruction, which is an operation instruction for the robot in real space, or a second instruction, which is an operation instruction for the robot, generated in virtual space. The virtual space is generated based on the detection results of one or more environmental sensors installed in the robot or the surrounding environment of the robot and detecting the surrounding environment, and the state of the robot. The system is controlled to allow switching between a first mode in which the robot is operated based on the first instruction and a second mode in which the robot is operated based on the second instruction. The robot is operated based on the first instruction in the switched first mode, or the second instruction in the second mode. Control method.

Images