Remote operation control system, remote operation control method, and program
The remote operation control system facilitates collaborative operation and feedback between multiple operators by integrating control and feedback mechanisms, allowing simultaneous control and correction of remote operations, thereby enhancing the operational experience and efficiency.
Patent Information
- Application Number
- JP2021069234
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-04-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-04-15
AI Technical Summary
Conventional remote control systems do not allow multiple operators to simultaneously experience and operate the same remote control object, limiting collaborative operation and feedback between operators.
A remote operation control system that integrates feedback mechanisms allowing multiple operators to control and experience the operation of a remote control object through first and second controllers, generating and transmitting control and feedback information based on the operations of both operators, with prioritization and synthesis options for combined control.
Enables multiple operators to collaboratively control and learn from each other's operations, providing real-time guidance and correction, and simulating the sensations of operating the remote control object, enhancing the operational experience and efficiency.
Smart Images

Figure 0007588545000001 
Figure 0007588545000002 
Figure 0007588545000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a remote operation control system, a remote operation control method, and a program. [Background technology]
[0002] In recent years, a technique has been known in which an operator remotely controls a remote control target such as a robot. For example, a technique is known in which multiple operators take turns remotely controlling the same robot by switching the authority of remote control (see Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2019-005819 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, with conventional technology, it has not been possible for a plurality of operators to operate the same remotely controlled object and experience the operation of the same remotely controlled object by the other operators.
[0005] The aspects of the present invention have been made in consideration of such circumstances, and aim to provide a remote operation control system, a remote operation control method, and a program that enable a user to experience operations performed by another operator on the same remote operation target. [Means for solving the problem]
[0006] The remote operation control system, the remote operation control method, and the program according to the present invention employ the following configuration. (1): A remote operation control system according to one embodiment of the present invention is a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world. The remote operation control system includes an information acquisition unit that acquires first operation information indicating an operation input by a first operator via a first controller that accepts operations on the remote operation target, and second operation information indicating an operation input by a second operator via a second controller that accepts operations on the remote operation target, a control information generation unit that generates control information for operating the remote operation target based on the first operation information and the second operation information acquired by the information acquisition unit, and a feedback information generation unit that generates first feedback information that feeds back the operation content of the remote operation target to the first controller and generates second feedback information that feeds back the operation content of the remote operation target to the second controller based on the control information.
[0007] (2): In the above aspect (1), the feedback information generation unit generates first feedback information for transmitting the operation content by the first operator to the second operator via the second controller based on the movement of the remotely operated object based on the operation input by the first operator, and generates second feedback information for transmitting the operation content by the second operator to the first operator via the first controller based on the movement of the remotely operated object based on the operation input by the second operator.
[0008] (3): In the above aspect (1) or (2), the system further includes a comparison image generating unit that generates a comparison image showing a comparison result between an operation input by the first operator and an operation input by the second operator, the comparison image being image data to be displayed on a first display unit for the first operator or a second display unit for the second operator.
[0009] (4): In any of the above aspects (1) to (3), the control information generation unit generates the control information based on a result of combining the content of the remote operation by the first operator and the content of the remote operation by the second operator.
[0010] (5): In the above aspect (4), the control information generation unit generates the control information by prioritizing the operation of the second operator over that of the first operator when synthesizing the content of the remote operation by the first operator and the content of the remote operation by the second operator.
[0011] (6): In any one of the above aspects (1) to (5), the feedback information generation unit generates third feedback information for transmitting a sensation or feel obtained at the remotely controlled target to the first operator via the first controller based on a detection result by a sensor mounted on the remotely controlled target, and generates first feedback information for transmitting a sensation or feel obtained at the remotely controlled target to the second operator via the second controller based on a detection result by a sensor mounted on the remotely controlled target.
[0012] (7): A remote operation control method according to another aspect of the present invention is a remote operation control method in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, by a computer, which acquires first operation information indicating an operation input by a first operator via a first controller that accepts operations on the remote operation target and second operation information indicating an operation input by a second operator via a second controller that accepts operations on the remote operation target, generates control information for operating the remote operation target based on the acquired first operation information and second operation information, generates first feedback information for feeding back the operation content of the remote operation target to the first controller based on the control information, and generates second feedback information for feeding back the operation content of the remote operation target to the second controller.
[0013] (8): A program according to another aspect of the present invention is a program for remote control in which an operator remotely controls a remotely controlled object, which is a real object or a virtual object in a virtual world, the program causing a computer to acquire first operation information indicating an operation input by a first operator via a first controller that accepts operations on the remotely controlled object and second operation information indicating an operation input by a second operator via a second controller that accepts operations on the remotely controlled object, generate control information for operating the remotely controlled object based on the acquired first operation information and second operation information, generate first feedback information for feeding back the operation content of the remotely controlled object to the first controller based on the control information, and generate second feedback information for feeding back the operation content of the remotely controlled object to the second controller. Effect of the Invention
[0014] According to the above aspects (1) to (8), it is possible to experience the operation of the same remotely operated object by another operator. According to the above (2), the operation of a robot or the like remotely operated by one operator can be transmitted to the other operator. According to (3) above, one operator can check, by means of an image, his / her own erroneous operation that has been corrected by the other operator, and the differences between his / her operation and the operation by the other operator. According to the above (4), a remote-operated object can be operated in accordance with remote operations by two or more operators, which makes it possible for an experienced operator to provide guidance to a novice operator in real time, for example. According to (5) above, the remote operation by one operator can be reflected in the behavior of the remotely controlled object rather than the remote operation by the other operator, and the remote operation by one operator can be corrected to the remote operation by the other operator. According to the above (6), it is possible to transmit to the operator the sensations and feelings that would be obtained if the operator were actually operating an object to be remotely operated, rather than remote operation using a remote-operated object. [Brief description of the drawings]
[0015] [Figure 1] 1 is a diagram showing an overview of a remote operation control system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a functional block diagram of a configuration installed in a work space where a first operator works. [Diagram 3] FIG. 2 is a functional block diagram of the robot. [Figure 4] FIG. 2 is a functional block diagram of a robot remote operation control device. [Diagram 5] FIG. 13 is a diagram showing an example of a comparison image. [Figure 6] FIG. 2 is a sequence diagram showing an example of processing in the robot remote operation control system. [Figure 7] 10 is a flowchart illustrating an example of a process performed by a robot remote operation control device. [Figure 8] FIG. 11 is a sequence diagram showing another example of the processing in the robot remote operation control system. [Figure 9] 13 is a flowchart showing another example of the process performed by the robot remote operation control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, an embodiment of a remote control system, a remote control method, and a program according to the present invention will be described with reference to the drawings. Note that in the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size.
[0017] <Summary> 1 is a diagram showing an overview of a remote operation control system 1 according to this embodiment. The remote operation control system 1 includes, for example, a first HMD (head mounted display) 10, a first controller 20, a first environmental sensor 30, a first terminal device 40, a second HMD 50, a second controller 60, a second environmental sensor 70, a second terminal device 80, a robot 100, a third environmental sensor 200, and a robot remote operation control device 300.
[0018] For example, the first HMD 10, the first controller 20, and the first environmental sensor 30 are connected to the first terminal device 40 via short-range wireless communication. The first terminal device 40 is connected to the robot 100, the third environmental sensor 200, and the robot remote operation control device 300 via a network NW. The second HMD 50, the second controller 60, and the second environmental sensor 70 are connected to the second terminal device 80 via short-range wireless communication, for example. The second terminal device 80 is connected to the robot 100, the third environmental sensor 200, and the robot remote operation control device 300 via the network NW. The network NW includes, for example, the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a provider device, a wireless base station, and the like.
[0019] However, this configuration is not limited to the above, and the first terminal device 40 and the second terminal device 80 may be omitted. In this case, the first HMD 10, the first controller 20, the first environmental sensor 30, the second HMD 50, the second controller 60, and the second environmental sensor 70 may be directly connected to the robot 100, the third environmental sensor 200, and the robot remote control device 300 via the network NW.
[0020] The robot 100, the third environmental sensor 200, and the robot remote operation control device 300 are connected, for example, via a wireless or wired communication network. However, the present invention is not limited to this, and the robot 100, the third environmental sensor 200, and the robot remote operation control device 300 may be directly connected by wire without using a communication network.
[0021] The first HMD 10 and the first controller 20 are devices worn by, for example, novice A (an example of a first operator), and the second HMD 50 and the second controller 60 are devices worn by, for example, expert B (an example of a second operator). The first controller 20 and the second controller 60 are, for example, tactile data gloves, and are worn on the hands of the operators. The first controller 20 and the second controller 60 detect the direction, the movement of each finger, the movement of the hand, the movement of the arm, and the like by the sensors provided therein, and transmit information indicating the detection results (hereinafter, referred to as operation information) to the robot remote operation control device 300. The operation information includes information indicating the operation by novice A, information indicating the operation by expert B, and the like.
[0022] In the work space where new employee A works, for example, a first environmental sensor 30 and a first terminal device 40 are installed. The first environmental sensor 30 is, for example, a dedicated device equipped with a communication function, an RBG camera, and a depth sensor. The first environmental sensor 30 is installed in a position where it can capture and detect the work of new employee A. The first terminal device 40 is, for example, a tablet terminal or a personal computer equipped with a communication function. Without being limited thereto, the first environmental sensor 30 may be, for example, attached to the first terminal device 40, or the first environmental sensor 30 may have the functions of the first terminal device 40.
[0023] In the work space where the expert B works, for example, a second environmental sensor 70 and a second terminal device 80 are installed. The second environmental sensor 70 is, for example, a dedicated device equipped with a communication function, an RBG camera, and a depth sensor. The second environmental sensor 70 is installed in a position where it can capture and detect the work of the expert B. The second terminal device 80 is, for example, a tablet terminal or a personal computer equipped with a communication function. Without being limited thereto, the second environmental sensor 70 may be, for example, attached to the second terminal device 80, or the second environmental sensor 70 may have the functions of the second terminal device 80.
[0024] The third environmental sensor 200 is, for example, a dedicated device equipped with a communication function, an RGB camera, and a depth sensor, and is installed at a position where it can capture and detect the work of the robot 100. The third environmental sensor 200 may be provided in the robot 100 or may be attached to the robot 100.
[0025] For example, novice A remotely controls the robot 100 by moving the hand or finger wearing the first controller 20 while viewing the image displayed on the first HMD 10. Moreover, expert B remotely controls the robot 100 by moving the hand or finger wearing the second controller 60 while viewing the image displayed on the second HMD 50. In this embodiment, novice A is a worker who remotely controls the robot 100 to create an object such as a craftwork that requires skill to make. Expert B is an instructor who remotely controls the robot 100 during or before and after the operation of novice A, allowing novice A to experience the example of expert B and providing instruction such as correcting the work of novice A. In addition, in remote operation, novice A and expert B cannot directly see the operation of the robot 100, but can indirectly see the image of the robot 100 side by the first HMD 10 or the second HMD 50.
[0026] The robot 100 is an example of a remote control target that is a real object. The remote control target is not limited to this, and may be a virtual object in a virtual world. For example, a robot or an avatar that is a virtual object that appears in a virtual world such as a two-dimensional or three-dimensional animation may be the remote control target. An example in which the remote control target is the robot 100 will be described below. In the example in which the remote control target is a virtual object, the process for operating the robot 100 may be read as a process for operating the virtual object in the virtual world. In other words, the robot remote control control device 300 may have a configuration such as an image processing unit for operating a virtual object in the virtual world, and the first controller 20 and the second controller 60 may have a configuration for operating a virtual object in the virtual world.
[0027] The robot remote operation control device 300 feeds back the motion of the robot 100 remotely operated by the novice A or the expert B to the first controller 20 or the second controller 60. As a result, the first controller 20 reproduces the motion of the robot 100 remotely operated by the expert B, and the second controller 60 reproduces the motion of the robot 100 remotely operated by the novice A. For example, in a scene in which the expert B instructs the novice A to apply more force when performing remote operation to transform an operation object, the robot 100 performs an action of applying more force to the operation object when the robot arm is placed on the operation object of the robot 100 according to the remote operation of the expert B. By feeding back such an action performed by the robot 100 to the first controller 20 or the second controller 60, the content of the remote operation by the expert B is fed back to the novice A, and the novice A can be given a feeling or sensation as if the expert B was directly instructing the novice A by placing his hand on the operation object.
[0028] In this way, the expert B can experience the operation of the novice A via the second controller 60, and the expert B can input an operation to correct the operation of the novice A into the second controller 60. Thus, the novice A can experience the operation of the expert B via the first controller 20, and remotely operate the robot 100 while receiving guidance from the expert B while being in a location away from the expert B and the robot 100.
[0029] Although details will be described later, the robot remote operation control device 300 may feed back the sensations and feelings obtained on the robot 100 side to the newbie A and the expert B.
[0030] 2 is a functional block diagram of the components installed in the work space where the first operator works. Note that the components installed in the work space where the expert B works also have the same functions, and detailed descriptions thereof will be omitted.
[0031] <hmd> The first HMD 10 includes, for example, a display unit 11, a gaze detection unit 12, a sensor 13, a control unit 14, a communication unit 15, and a sound collection unit 16. The first HMD 10 displays an image for remotely controlling the robot 100 based on image data received from the robot remote operation control device 300. For example, the first HMD 10 displays a status image of the robot 100. The status image of the robot 100 is an image showing the action of the robot 100, and includes, for example, an image of the robot 100's hand.
[0032] The display unit 11 is, for example, an LCD or an organic EL display device. The gaze detection unit 12 detects the movement of the gaze of newcomer A. The gaze detection unit 12 may include a needle detector that detects gaze information of newcomer A. The sensor 13 is, for example, an acceleration sensor, a gyroscope, etc. The sensor 13 detects the movement and tilt of newcomer A's head.
[0033] For example, the control unit 14 causes the display unit 11 to display, for example, an image of the state of the robot based on image data transmitted by the robot remote operation control device 300.
[0034] For example, the control unit 14 generates information indicating the movement of the line of sight of newcomer A (hereinafter referred to as line of sight information) based on the detection result by the line of sight detection unit 12. The control unit 14 also generates information indicating the movement and tilt of the head of newcomer A (hereinafter referred to as head movement information) based on the detection result by the sensor 13. The line of sight information and head movement information are information included in the operator sensor value described later. The control unit 14 causes the generated line of sight information and head movement information to be transmitted to the robot remote operation control device 300 via the first terminal device 40.
[0035] The communication unit 15 is, for example, a wireless module for using Bluetooth (registered trademark) or the like, and includes an antenna, a transmitting / receiving device, etc. The communication unit 15 communicates with the first terminal device 40 via short-range wireless communication in accordance with the control of the control unit 14. However, without being limited thereto, the communication unit 15 may include a communication interface such as a NIC and communicate with the first terminal device 40, the robot 100, or the robot remote operation control device 300 via a network NW.
[0036] The sound pickup unit 16 is, for example, a microphone, and outputs the picked-up sound signal to the control unit 14. The control unit 14 uses the communication unit 15 to transmit the sound signal to the robot remote operation control device 300. When the control unit 14 receives voice information from the robot remote operation control device 300, it outputs the sound from the speaker 17. When the robot remote operation control device 300 acquires, for example, a sound signal of the conversation of newcomer A via the first HMD 10, it provides the voice signal of the conversation of newcomer A to the expert B via the second HMD 50. Also, when the robot remote operation control device 300 acquires, for example, a sound signal of the conversation of expert B via the second HMD 50, it provides the voice signal of the conversation of expert B to newcomer A via the first HMD 10.
[0037] <Controller> The first controller 20 includes, for example, a sensor 21, a control unit 22, a communication unit 23, and a feedback means 24.
[0038] The sensor 21 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The control unit 22 tracks the movement of each finger based on the detection results detected by the multiple types of sensors included in the sensor 21. The control unit 22 also acquires information indicating the posture and position of the operator's arm, such as the direction, the movement of each finger, and the movement of the hand, based on the detection results by the sensor 21 (hereinafter, referred to as operator arm information). The operator arm information includes, for example, information covering the entire human arm, such as hand tip position and posture information, angle information of each finger, elbow position and posture information, and information tracking the movement of each part. The operator arm information is an example of operation information. The control unit 22 transmits the acquired operator arm information to the robot remote operation control device 300 via the first terminal device 40.
[0039] The communication unit 23 is a wireless module for using Bluetooth (registered trademark) or the like, and includes an antenna, a transmitting / receiving device, etc. The communication unit 23 communicates with the first terminal device 40 via short-range wireless communication in accordance with the control of the control unit 22. However, the communication unit 23 may include a communication interface such as a NIC and communicate with the first terminal device 40, the robot 100, or the robot remote operation control device 300 via the network NW.
[0040] The feedback means 24, under the control of the control unit 22, feeds back feedback information (pressure, temperature, etc.) from the end effector of the robot 100 to the operator.
[0041] The feedback means 24 includes, for example, a means for reproducing a sensation or feel obtained on the robot 100 side, and a means for reproducing an action performed by the robot 100. The means for reproducing a sensation or feel obtained on the robot 100 side is, for example, a means for reproducing a feel or feel when the robot hand comes into contact with an object or the like while the robot 100 is performing a task. The means for reproducing an action performed by the robot 100 is, for example, a means for reproducing a direction and speed in which the robot arm of the robot 100 moves.
[0042] For example, the feedback means 24 includes a means for applying vibration (not shown), a means for applying air pressure (not shown), a means for restricting hand movement (not shown), a means for making the user feel temperature (not shown), a means for making the user feel hardness or softness (not shown), etc. The feedback means 24 reproduces the sensations and touches obtained on the robot 100 side and the actions of the robot 100 that are fed back from the robot 100, according to the control of the control unit 22. In this way, the first controller 20 can transmit the remote operations of the newbie A and the expert B to the newbie A.
[0043] The control unit 22 controls the feedback means 24 based on the feedback information received from the robot remote operation control device 300. The feedback information will be described in detail later.
[0044] <Environmental sensor> The first environmental sensor 30 includes, for example, an image capture device 31, a sensor 32, an object position detection unit 33, a control unit 34, and a communication unit 35. The second environmental sensor 70 and the third environmental sensor 200 also have similar functions, and detailed descriptions thereof will be omitted.
[0045] The image capturing device 31 is, for example, an RGB camera, and the sensor 32 is, for example, a depth sensor. The image capturing device 31 and the sensor 32 may be distance sensors. In the first environmental sensor 30, the positional relationship between the image capturing device 31 and the sensor 32 is known.
[0046] The object position detection unit 33 detects the three-dimensional position, size, shape, etc. of a target object in a captured image by a known method based on the image captured by the image capture device 31 and the detection result detected by the sensor 32. For example, the object position detection unit 33 estimates the position of the object by performing image processing (edge detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, etc.) on the image captured by the image capture device 31 with reference to a pattern matching model stored in the object position detection unit 33. When multiple objects are detected from the captured image, the object position detection unit 33 detects the position of each object.
[0047] The control unit 34 transmits information indicating the position of the object detected by the object position detection unit 33, the image captured by the photographing device 31, and the sensor value detected by the sensor 32 to the robot remote operation control device 300 via the first terminal device 40. The information indicating the position of the object, the image captured by the photographing device 31, and the sensor value detected by the sensor 32 are included in the environmental sensor value described below.
[0048] The communication unit 35 is a wireless module for using Bluetooth (registered trademark) or the like, and includes an antenna, a transmitting / receiving device, etc. The communication unit 35 communicates with the first terminal device 40 via short-range wireless communication. However, the communication unit 35 may include a communication interface such as a NIC and communicate with the first terminal device 40, the robot 100, or the robot remote operation control device 300 via a network NW.
[0049] The first environment sensor 30 may be a motion capture device, and may detect the position information of an object by motion capture, or may have a GPS receiver (not shown) equipped with a position information transmission unit attached to the object. In this case, the GPS receiver may transmit the position information to the robot remote operation control device 300.
[0050] <Example of robot configuration> 3 is a functional block diagram of the robot 100. The robot 100 includes, for example, a communication unit 110, a sound collection unit 120, a sensor 130, a drive unit 140, a robot arm 150, a power source 160, a storage unit 170, and a control unit 180.
[0051] The communication unit 110 includes a communication interface such as a NIC. The communication unit 110 connects to the first terminal device 40, the second terminal device 80, the third environmental sensor 200, and the robot remote operation control device 300 via the network NW under the control of the control unit 180. The communication unit 110 is a wireless module for using Bluetooth (registered trademark) or the like, and has an antenna, a transmitting / receiving device, and the like, and may communicate with the third environmental sensor 200 and the robot remote operation control device 300 via short-range wireless communication.
[0052] The sound collection unit 120 is, for example, a microphone array including a plurality of microphones. The sound collection unit 120 outputs a collected acoustic signal to the control unit 180. The sound collection unit 120 may have a voice recognition processing function. In this case, the sound collection unit 120 outputs a voice recognition result to the control unit 180.
[0053] The sensor 130 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, an encoder for each joint, etc. For example, the sensor 130 is attached to each joint, the head, etc. of the robot 100. The sensor 130 outputs information indicating the detection result to the control unit 180.
[0054] The driving unit 140 drives each part of the robot 100 (arms, fingers, legs, head, torso, waist, etc.) according to the control of the control unit 180. The driving unit 140 drives, for example, the robot arm 150. The robot arm 150 includes, for example, an actuator, a gear, an artificial muscle, etc. The robot arm 150 also includes a hand sensor 151. The hand sensor 151 includes, for example, a means for sensing temperature (not shown), a means for sensing hardness or softness (not shown), etc.
[0055] The power source 160 supplies power to each component of the robot 100. The power source 160 may include, for example, a rechargeable battery or a charging circuit.
[0056] The storage unit 170 is realized by, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), etc. The storage unit 170 stores the programs executed by the processor, as well as various types of information such as operation parameters for operating the robot 100.
[0057] The control unit 180 executes control to operate the robot 100 by remote operation and control to operate the robot 100 without remote operation. In the former case, the control unit 180 operates the robot 100 according to control information generated by the robot remote operation control device 300.
[0058] In the latter case, the control unit 180 performs voice recognition processing (such as speech interval detection, sound source separation, sound source localization, noise suppression, and sound source identification) on the acoustic signal collected by the sound collection unit 120. If the result of the voice recognition includes an operation instruction for the robot 100, the control unit 180 controls the operation of the robot 100 based on the operation instruction by voice. In addition, the control unit 180 may perform image processing (such as edge detection, binarization processing, feature extraction, image enhancement processing, image extraction, and pattern matching processing) on images captured by the third environmental sensor 200 or the like based on the information stored in the storage unit 170, and control the operation of the robot 100 based on the voice recognition result and the image processing result.
[0059] <Configuration example of the robot remote operation control device 300> 4 is a functional block diagram of the robot remote operation control device 300. The robot remote operation control device 300 includes, for example, a communication unit 310, an input unit 320, an output unit 330, a storage unit 340, and a processing unit 350.
[0060] The communication unit 310 includes a communication interface such as a NIC. The communication unit 310 communicates with the first terminal device 40, the second terminal device 80, the robot 100, and the third environmental sensor 200 via the network NW under the control of the processing unit 350.
[0061] The input unit 320 includes, for example, some or all of various keys, buttons, dial switches, a mouse, etc. The output unit 330 includes, for example, an LCD or organic EL display device, a speaker, a projector, etc. The storage unit 340 is realized by, for example, a RAM, a ROM, a HDD, etc. The storage unit 340 stores the programs executed by the processor, as well as robot information 341, feedback parameter information 342, etc. Details of each piece of information will be described later.
[0062] The processing unit 350 includes, for example, an information acquisition unit 351, an intention estimation unit 352, a robot operation determination unit 353, a control information generation unit 354, a feedback information generation unit 355, a robot state image generation unit 356, a comparison image generation unit 357, and a communication control unit 358. Some or all of these functional units are realized by, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program (software). In addition, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array), or may be realized by cooperation between software and hardware. The program may be stored in advance in the storage unit 340 of the robot remote operation control device 300, or may be stored in a removable storage medium such as a DVD or a CD-ROM, and installed in the storage unit 340 by mounting the storage medium in a drive device.
[0063] For example, the information acquisition unit 351 acquires first operation information from the first terminal device 40 via the network NW. The first operation information is information indicating an operation input by the novice A via the first controller 20 that accepts an operation on the robot 100 that is the remote operation target. The information acquisition unit 351 also acquires second operation information from the second terminal device 80 via the network NW. The second operation information is information indicating an operation input by the expert B via the second controller 60 that accepts an operation on the robot 100 that is the remote operation target. The first operation information and the second operation information include, for example, operator arm information.
[0064] The information acquisition unit 351 also acquires first operator sensor value information from the first terminal device 40 via the network NW. The first operator sensor value information is information indicating a detection result of a state change of the novice A, who is the first operator, detected by a sensor, and includes, for example, information indicating a detection result of a sensor mounted on the first HMD 10 or the first controller 20. The information acquisition unit 351 also acquires second operator sensor value information from the second terminal device 80 via the network NW. The second operator sensor value information is information indicating a detection result of a state change of the expert B, who is the second operator, detected by a sensor, and includes, for example, information indicating a detection result of a sensor mounted on the second HMD 50 or the second controller 60. The first operator sensor value information and the second operator sensor value information include, for example, gaze information and head movement information detected by the first HMD 10 or the second HMD 50.
[0065] The information acquisition unit 351 also acquires first environmental sensor value information from the first terminal device 40 via the network NW. The first environmental sensor value information is information indicating a detection result of a sensor detecting an environmental change around where the robot 100 is remotely operated by the first operator, rookie A, and includes, for example, information indicating a detection result of a sensor mounted on the first environmental sensor 30. The information acquisition unit 351 also acquires second environmental sensor value information from the second terminal device 80 via the network NW. The second environmental sensor value information is information indicating a detection result of a sensor detecting an environmental change around where the robot 100 is remotely operated by the second operator, expert B, and includes, for example, information indicating a detection result of a sensor mounted on the second environmental sensor 70. The information acquisition unit 351 also acquires third environmental sensor value information from the third environmental sensor 200 via the network NW. The third environmental sensor value information is information indicating the detection results of environmental changes around the robot 100 remotely operated by a first operator, a second operator, etc., detected by a sensor, and includes, for example, information indicating the detection results of a sensor mounted on the third environmental sensor 200. The first environmental sensor value information, the second environmental sensor value information, and the third environmental sensor value information include, for example, object position information, an image captured by the imaging device 31, and a sensor value detected by the sensor 32.
[0066] The intention estimation unit 352 estimates the operator's intention based on the information acquired by the information acquisition unit 351. For example, the intention estimation unit 352 estimates the operator's intention using at least one of gaze information, operator arm information, and head motion information. Note that the intention estimation unit 352 may also estimate the intention using an environmental sensor value.
[0067] The intention estimation unit 352 may estimate the intention of the action that the operator is about to perform by inputting the operator sensor value and gaze information acquired by the information acquisition unit 351 into a trained model. By generating control information based on such an estimation result and controlling the robot 100 based on the generated control information, the robot 100 can perform accurate alignment even if the operator does not remotely operate the robot 100 for accurate alignment. Therefore, the robot 100 can perform the work with high accuracy. The trained model is data that is trained by inputting the operator sensor value and gaze information in advance and using the work that the operator is about to perform as teacher data, for example.
[0068] The robot operation determination unit 353 determines an operation method for an object to be operated by the robot 100 based on the intention of the operation estimated by the intention estimation unit 352, the detection result detected by the sensor 130 of the robot 100, the result of image processing of the image captured by the image capture device of the third environment sensor 200, etc. The operation method for the object includes various operation methods such as a gripping method for gripping the object, a moving method for moving the object, and a transformation method for transforming the object.
[0069] The control information generating unit 354 refers to the storage unit 340, and generates control information for operating the robot 100, which is the remote-operated target, based on the first operation information or the second operation information acquired by the information acquiring unit 351. For example, the control information generating unit 354 refers to the robot information 341, acquires operation parameters according to the operation by the operator, and generates control information for operating the robot 100 based on the acquired operation parameters. The robot information 341 is, for example, information in which operation parameters are associated with the amount of operation.
[0070] The control information generating unit 354 may generate control information for operating the robot 100 based on the operation method determined by the robot operation determining unit 353. In this way, the intention of the operator can be estimated, and based on the estimation result, the robot 100 can be made to perform an operation that is closer to the operator's intention.
[0071] The control information generator 354 may generate control information for operating the robot 100 based on the first operator sensor information and the first environmental sensor value information in addition to the first operation information. The control information generator 354 may generate control information for operating the robot 100 based on the second operator sensor information and the second environmental sensor value information in addition to the second operation information.
[0072] Furthermore, the control information generating unit 354 may generate control information for operating the robot 100 based on a result of combining the contents of the remote operation by the new operator A, who is the first operator, and the contents of the remote operation by the expert B, who is the second operator. For example, the control information generating unit 354 generates the control information based on the sum of the operation parameters according to the first operation information and the operation parameters according to the second operation information. Without being limited to this, the control information generating unit 354 may generate the control information based on a value obtained by subtracting the operation parameters according to the second operation information from the operation parameters according to the first operation information. Note that the method of combining the contents of the remote operation by the new operator A, who is the first operator, and the contents of the remote operation by the expert B, who is the second operator, is not limited to this, and various existing methods can be adopted. In this way, the robot 100 can be operated according to remote operations by two or more operators. Therefore, the expert B can guide the work of the new operator A in real time.
[0073] Furthermore, when synthesizing the contents of the remote operation by the first operator and the contents of the remote operation by the second operator, the control information generating unit 354 may generate control information in which the operation by the second operator is prioritized over that by the first operator. For example, when the operation by the first operator and the operation by the second operator are simultaneously performed, the control information generating unit 354 performs a first weighting on the operation parameters based on the operation by the first operator, and performs a second weighting larger than the first weighting on the operation parameters based on the operation by the second operator to generate control information. The first weighting may be 0, and the operation by the first operator may not be reflected in the control information. Furthermore, when the operation direction by the first operator and the operation direction by the second operator are different, the control information generating unit 354 may generate control information to operate in the operation direction by the second operator. In this way, the remote operation by the second operator can be reflected in the operation of the robot 100 more than the remote operation by the first operator, and the remote operation by the first operator can be corrected to the remote operation by the second operator.
[0074] Furthermore, when synthesizing the contents of the remote operation by the first operator and the contents of the remote operation by the second operator, the control information generating unit 354 may be configured to execute the remote operation timing by the first operator and the remote operation timing by the second operator consecutively. For example, the control information generating unit 354 may generate control information that executes an operation corresponding to the remote operation by the second operator immediately after executing an operation corresponding to the remote operation by the first operator. In this way, the remote operation by the first operator can be corrected to the remote operation by the second operator immediately after the remote operation by the first operator.
[0075] The feedback information generating unit 355 generates second feedback information for transmitting the operation content of the first operator to the second operator via the second controller 60, based on the movement of the robot 100 based on the operation input by the novice A who is the first operator. Also, the feedback information generating unit 355 generates first feedback information for transmitting the operation content of the second operator to the first operator via the first controller 20, based on the movement of the remotely operated target based on the operation input by the expert B who is the second operator. In this way, the movement of the robot or the like remotely operated by one operator can be transmitted to the other operator.
[0076] For example, the feedback information generating unit 355 refers to the feedback parameter information 342 in the storage unit 340, and generates first feedback information for feeding back the operation details of the robot 100, which is the remote control target, to the first controller 20, based on the control information generated by the control information generating unit 354. Also, the feedback information generating unit 355 refers to the feedback parameter information 342 in the storage unit 340, and generates second feedback information for feeding back the operation details of the robot 100, which is the remote control target, to the second controller 60, based on the control information generated by the control information generating unit 354. The feedback information generating unit 355 refers to the feedback parameter information 342, acquires feedback parameters according to the control information, and generates feedback information based on the acquired feedback parameters. The feedback parameter information 342 is, for example, information in which feedback parameters according to the operation parameters and operation amounts of the robot 100 are associated with each other.
[0077] For example, when the robot 100 is remotely operated by the first operator, the feedback information generating unit 355 generates second feedback information for causing the second controller 60 to reproduce the motion of the remotely operated robot 100 in order to feed back the content of the remote operation by the first operator to the second operator. On the other hand, when the robot 100 is remotely operated by the second operator, the feedback information generating unit 355 generates first feedback information for causing the first controller 20 to reproduce the motion of the remotely operated robot 100 in order to feed back the content of the remote operation by the second operator to the first operator.
[0078] As described above, when the control information generating unit 354 generates control information based on the synthesis result of the content of remote operation by the first operator and the content of remote operation by the second operator, the feedback information generating unit 355 may generate first feedback information for causing the first controller 20 to reproduce the movement of the remotely controlled robot 100 and second feedback information for causing the second controller 60 to reproduce the movement of the remotely controlled robot 100, in order to feed back the movement of the robot 100 based on the synthesis result to the first operator and the second operator.
[0079] Furthermore, as described above, when synthesizing the contents of the remote operation by the first operator and the contents of the remote operation by the second operator, if the control information generating unit 354 generates control information that prioritizes the remote operation by the second operator over the first operator, the feedback information generating unit 355 may generate first feedback information that prioritizes the feedback of the operation of the robot 100 to the first operator over the second operator. For example, the feedback information generating unit 355 generates first feedback information and second feedback information in which the contents of the feedback to the first operator are emphasized compared to the contents of the feedback to the second operator. Examples of emphasizing the feedback contents include applying vibrations, applying stronger air pressure, and restricting hand movements.
[0080] Furthermore, the feedback information generating unit 355 generates third feedback information for transmitting the sensation or feel obtained on the robot 100 side to the first operator via the first controller 20 based on the detection results detected by the sensor 130 and the hand sensor 151 of the robot 100. Furthermore, the feedback information generating unit 355 generates fourth feedback information for transmitting the sensation or feel obtained on the robot 100 side to the second operator via the second controller 60 based on the detection results detected by the sensor 130 and the hand sensor 151 of the robot 100. In this way, it is possible to transmit to the operator the sensation or feel obtained when the operator actually operates a remote-controlled object, rather than the remote control using the robot 100.
[0081] The robot state image generating unit 356 performs image processing (edge detection, binarization, feature extraction, image enhancement, image extraction, clustering processing, etc.) on the image captured by the imaging device 31 of the third environment sensor 200, for example.
[0082] In addition, the robot state image generation unit 356 estimates the position and movement of the robot 100's hands and the movement of the operator's hands based on information indicating the results of the above-mentioned image processing (hereinafter referred to as image processing result information) and control information generated by the control information generation unit 354, and creates a robot state image to be displayed on the first HMD 10 and the second HMD 50 based on the estimated results.
[0083] The robot status image may include system status information indicating the system status, such as information regarding the process that the robot remote control control device 300 is about to perform and error information.
[0084] The comparison image generating unit 357 generates image data of a comparison image showing a comparison result between an operation input by the first operator and an operation input by the second operator. The image data of the comparison image is image data to be displayed on a first display unit for the first operator (e.g., the display unit of the first HMD 10) or a second display unit for the second operator (e.g., the display unit of the second HMD 50). The comparison image includes, for example, an image showing the difference between a remote operation by the first operator and a remote operation by the second operator, and an image emphasizing a remote operation by the second operator more than a remote operation by the first operator. In this way, the first operator can check, from the image, his / her own erroneous operation corrected by the second operator, the difference between his / her operation and the operation by the second operator, and the like.
[0085] The communication control unit 358 controls the communication unit 310 to communicate with the first terminal device 40, the second terminal device 80, the robot 100, and the third environmental sensor 200 via the network NW.
[0086] <Example of comparison image> 5 is a diagram showing an example of a comparison image. The comparison image shown is an example of a comparison image generated by, for example, a novice A and an expert B performing remote operation when transforming an operation object 500. Note that the comparison image shown shows the robot arm 150L in the left hand of the robot 100, and the robot arm in the right hand of the robot 100 and other images are omitted.
[0087] For example, newcomer A remotely controls the robot 100 to make a teacup using a potter's wheel. Expert B remotely controls the robot 100 to support newcomer A in making a teacup. The control object 500 is, for example, clay placed on a potter's wheel. Note that the work performed by newcomer A and expert B is not limited to this, and may include various other works such as lathe processing, painting a plate, and glasswork.
[0088] The robot arm image LA1 is an image showing the state of the robot arm 150L when remotely operated by newcomer A at the first timing. The robot arm image LA2 is an image showing the state of the robot arm 150L when remotely operated by newcomer A at the second timing. The robot arm image LB3 is an image showing the state of the robot arm 150L when remotely operated by expert B at the third timing. The first timing is the earliest timing, the second timing is the timing next to the first timing, and the third timing is the timing next to the second timing.
[0089] From the illustrated comparison image, it can be seen that the operation of novice A at the first and second timings is corrected by expert B to the position at the third timing. By displaying this comparison image on the first HMD 10, novice A can easily recognize the difference between his remote operation and the remote operation by expert B. In addition, by displaying this comparison image on the second HMD 50, expert B can verbally communicate the key points of the work and the corrections to be made while sharing the comparison image with novice A.
[0090] In the illustrated example, two arm images when remotely operated by novice A are displayed in the comparative image, but the present invention is not limited to this and may have only one. Also, although the illustrated example shows one arm image when remotely operated by expert B is displayed in the comparative image, the present invention is not limited to this and may have two or more.
[0091] <Sequence diagram (part 1)> 6 is a sequence diagram showing an example of processing in the robot remote operation control system 1. Here, an example will be described in which a first operator remotely operates the robot 100, and then a second operator remotely operates the robot 100, and the robot 100 is controlled based on operation information.
[0092] First, the first controller 20 generates first operation information based on an operation input by the first operator (step S1), and transmits the first operation information to the robot remote operation control device 300 via the first terminal device 40 (step S2). The robot remote operation control device 300 generates control information based on the received first operation information (step S3), and transmits the control information to the robot 100 (step S4). The robot 100 operates the robot arm 150 and the like based on the received control information (step S5). Meanwhile, the robot remote operation control device 300 generates second feedback information based on the control information generated in step S3 (step S6), and transmits the generated second feedback information to the second controller 60 via the second terminal device 80 (step S7). The second controller 60 reproduces the operation of the robot 100 remotely operated by the first operator in step S1 based on the received feedback information (step S8).
[0093] Next, the second controller 60 generates second operation information based on the operation input by the second operator (step S11), and transmits the second operation information to the robot remote operation control device 300 via the second terminal device 80 (step S12). The robot remote operation control device 300 generates control information based on the received second operation information (step S13), and transmits the control information to the robot 100 (step S14). The robot 100 operates the robot arm 150 and the like based on the received control information (step S15). Meanwhile, the robot remote operation control device 300 generates first feedback information based on the control information generated in step S13 (step S16), and transmits the generated first feedback information to the first controller 20 via the first terminal device 40 (step S17). The first controller 20 reproduces the operation of the robot 100 remotely operated by the second operator in step S11 based on the received feedback information (step S18).
[0094] <Flowchart (part 1)> 7 is a flowchart showing an example of processing by the robot remote operation control device 300. Here, an example will be described in which a first operator remotely operates the robot, and then a second operator remotely operates the robot 100 based on operation information. Also, processing by the intention estimation unit 352 and the robot operation determination unit 353, and processing by the robot state image generation unit 356 are omitted.
[0095] First, the information acquisition unit 351 judges whether or not the first operation information (or the second operation information) has been acquired (step S101). When the first operation information (or the second operation information) has been acquired, the control information generation unit 354 generates control information for operating the robot 100 based on the acquired first operation information (or the second operation information) (step S102), and transmits the generated control information to the robot 100 (step S103).
[0096] Next, the feedback information generator 355 generates second feedback information when the control information is generated based on the first operation information, and generates first feedback information when the control information is generated based on the second operation information (step S104).Then, the communication controller 358 transmits the second feedback information to the second controller 60 and transmits the first feedback information to the first controller 20 (step S105).
[0097] Next, in the case where the control information is generated based on the first operation information and the control information was previously generated based on the second operation information, the comparison image generating unit 357 generates image data of a first comparison image showing a comparison result between the operation input by the first operator and the operation input by the second operator (step S106). On the other hand, in the case where the control information is generated based on the second operation information and the control information was previously generated based on the first operation information, the comparison image generating unit 357 generates image data of a second comparison image showing a comparison result between the operation input by the second operator and the operation input by the first operator. Then, the communication control unit 358 transmits the image data of the first comparison image to the first HMD 10 and transmits the image data of the second comparison image to the second HMD 50 (step S107).
[0098] <Sequence diagram (part 2)> 8 is a sequence diagram showing another example of the processing in the robot remote operation control system 1. Here, an example will be described in which the remote operation by the first operator and the remote operation by the second operator are performed at the same timing, and the robot 100 is controlled based on operation information. The same timing is not limited to simultaneously, and also includes cases where there is a time difference of several seconds.
[0099] First, the first controller 20 generates first operation information based on an operation input by the first operator (step S21), and transmits the first operation information to the robot remote operation control device 300 via the first terminal device 40 (step S22). At the same time, the second controller 60 generates second operation information based on an operation input by the second operator (step S23), and transmits the second operation information to the robot remote operation control device 300 via the second terminal device 80 (step S24).
[0100] Next, the robot remote operation control device 300 generates control information by combining the contents of the remote operation by the first operator and the contents of the remote operation by the second operator based on the received first operation information and second operation information (step S25), and transmits the control information to the robot 100 (step S26). The robot 100 operates the robot arm 150 and the like based on the received control information (step S27).
[0101] Next, the robot remote operation control device 300 generates second feedback information based on the control information generated in step S25 (step S28), and transmits the generated second feedback information to the second controller 60 via the second terminal device 80 (step S29). The second controller 60 reproduces the motion of the robot 100 remotely operated by the first operator in step S21 based on the received feedback information (step S30).
[0102] In addition, the robot remote operation control device 300 generates first feedback information based on the control information generated in step S25 (step S31), and transmits the generated first feedback information to the first controller 20 via the first terminal device 40 (step S32). The first controller 20 reproduces the motion of the robot 100 remotely operated by the second operator in step S23 based on the received feedback information (step S33).
[0103] <Flowchart (part 2)> 9 is a flowchart showing another example of the processing by the robot remote operation control device 300. Here, an example will be described in which remote operations by a first operator and a second operator are performed at the same timing, and the robot 100 is controlled based on operation information. Also, the processing by the intention estimation unit 352 and the robot operation determination unit 353, and the processing by the robot state image generation unit 356 are omitted.
[0104] First, the information acquisition unit 351 judges whether the first operation information and the second operation information are acquired (step S201). When the first operation information and the second operation information are acquired, the control information generation unit 354 generates control information by combining the contents of the remote operation by the first operator and the contents of the remote operation by the second operator based on the acquired first operation information and second operation information (step S202), and transmits the generated control information to the robot 100 (step S203).
[0105] Next, the feedback information generator 355 generates the first feedback information and the second feedback information based on the control information generated in step S202 (step S204). Then, the communication controller 358 transmits the first feedback information to the first controller 20 and transmits the second feedback information to the second controller 60 (step S205).
[0106] Next, the comparison image generating unit 357 generates image data of the comparison image based on the first operation information and the second operation information acquired in step S201 (step S206). Then, the communication control unit 358 transmits the generated image data of the comparison image to the first HMD 10 and the second HMD 50 (step S207).
[0107] According to the above-described embodiment, in a remote operation control system in which an operator remotely operates a remote operation target which is a real object or a virtual object in a virtual world, the system includes an information acquisition unit 351 which acquires first operation information indicating an operation input by the first operator via the first controller 20 which accepts operations on the remote operation target, and second operation information indicating an operation input by the second operator via the second controller 60 which accepts operations on the remote operation target, a control information generation unit 354 which generates control information for operating the remote operation target based on the first operation information and the second operation information acquired by the information acquisition unit 351, and a feedback information generation unit 355 which generates first feedback information for feeding back the operation content of the remote operation target to the first controller 20 based on the control information, and generates second feedback information for feeding back the operation content of the remote operation target to the second controller 60, thereby making it possible to experience an operation by another operator on the same remote operation target.
[0108] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, In a remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world, acquiring first operation information indicating an operation input by a first operator via a first controller 20 that accepts an operation on a remote control target, and second operation information indicating an operation input by a second operator via a second controller 60 that accepts an operation on a remote control target; generating control information for operating a remote-controlled object based on the acquired first operation information and second operation information; generating first feedback information for feeding back the operation details of the remote-controlled object to the first controller 20 based on the control information, and generating second feedback information for feeding back the operation details of the remote-controlled object to the second controller 60; The remote operation control system is configured as follows.
[0109] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.
[0110] For example, a part of the robot remote operation control device 300 may be mounted on the robot 100. For example, the functions of the processing unit 350 and the memory unit 340 may be executed by the robot 100. [Explanation of symbols]
[0111] 1. Remote control system 10…1st HMD 20…1st controller 30…First environment sensor 40...First terminal device 50…2nd HMD 60…Second controller 70…Second environmental sensor 80...Second terminal device 100…Robot 200…Third environmental sensor 300...Robot remote control device 351…Information acquisition department 352…Intention estimation unit 353…Robot operation decision-making unit 354...Control information generation unit 355...Feedback information generation unit 356…Robot state image generation unit 357…Comparison image generation unit 358…Communication control unit< / hmd>
Claims
1. In a remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world, an information acquisition unit that acquires first operation information indicating an operation input by a first operator via a first controller that accepts an operation on the remote operation target, and second operation information indicating an operation input by a second operator via a second controller that accepts an operation on the remote operation target; a control information generating unit that generates control information for operating the remote control target based on the first operation information and the second operation information acquired by the information acquiring unit; a feedback information generation unit that generates first feedback information for feeding back a content of an operation of the remote-controlled object to the first controller based on the control information, and generates second feedback information for feeding back the content of the operation of the remote-controlled object to the second controller; a comparison image generating unit that generates a comparison image showing a comparison result between an operation input by the first operator and an operation input by the second operator, the comparison image being image data to be displayed on a first display unit for the first operator or a second display unit for the second operator; A remote control system comprising:
2. The feedback information generation unit generating first feedback information for transmitting an operation content of the first operator to the second operator via the second controller, based on a motion of the remote-controlled object based on an operation input by the first operator; generating second feedback information for transmitting the operation content by the second operator to the first operator via the first controller, based on a motion of the remote-operated target based on the operation input by the second operator; The remote control system according to claim 1 .
3. The control information generation unit generates the control information based on a result of combining the content of the remote operation by the first operator and the content of the remote operation by the second operator.
3. The remote control system according to claim 1 or 2.
4. A remote operation control system in which an operator remotely operates a remote operation target which is a real object or a virtual object in a virtual world, an information acquisition unit that acquires first operation information indicating an operation input by a first operator via a first controller that accepts an operation on the remote operation target, and second operation information indicating an operation input by a second operator via a second controller that accepts an operation on the remote operation target; a control information generating unit that generates control information for operating the remote control target based on the first operation information and the second operation information acquired by the information acquiring unit; a feedback information generation unit that generates first feedback information for feeding back a content of an operation of the remote-controlled object to the first controller based on the control information, and generates second feedback information for feeding back the content of the operation of the remote-controlled object to the second controller; Equipped with The control information generation unit generates the control information based on a result of combining a content of the remote operation by the first operator and a content of the remote operation by the second operator, the control information generation unit generates the control information in which the operation of the second operator is prioritized over that of the first operator when synthesizing the content of the remote operation by the first operator and the content of the remote operation by the second operator. Remote operation control system.
5. A remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world, an information acquisition unit that acquires first operation information indicating an operation input by a first operator via a first controller that accepts an operation on the remote operation target, and second operation information indicating an operation input by a second operator via a second controller that accepts an operation on the remote operation target; a control information generating unit that generates control information for operating the remote control target based on the first operation information and the second operation information acquired by the information acquiring unit; a feedback information generation unit that generates first feedback information for feeding back a content of an operation of the remote-controlled object to the first controller based on the control information, and generates second feedback information for feeding back the content of the operation of the remote-controlled object to the second controller; Equipped with The feedback information generation unit generating third feedback information for transmitting a sensation or feel obtained at the remote-operated target to the first operator via the first controller based on a detection result by a sensor mounted on the remote-operated target; generating fourth feedback information for transmitting a sensation or feel obtained at the remote-operated target to the second operator via the second controller based on a detection result by a sensor mounted on the remote-operated target; Remote operation control system.
6. The computer In a remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world, acquiring first operation information indicating an operation input by a first operator via a first controller that accepts an operation on the remote control target, and second operation information indicating an operation input by a second operator via a second controller that accepts an operation on the remote control target; generating control information for operating the remote-controlled object based on the acquired first operation information and the acquired second operation information; generating first feedback information for feeding back the operation content of the remote-controlled object to the first controller based on the control information; and generating second feedback information for feeding back the operation content of the remote-controlled object to the second controller, generating a comparison image, which is image data to be displayed on a first display unit for the first operator or a second display unit for the second operator, showing a comparison result between an operation input by the first operator and an operation input by the second operator; Remote operation control method.
7. In a remote control system in which an operator remotely controls a remote control target which is a real object or a virtual object in a virtual world, On the computer, acquiring first operation information indicating an operation input by a first operator via a first controller that accepts an operation on the remote control target, and second operation information indicating an operation input by a second operator via a second controller that accepts an operation on the remote control target; generating control information for operating the remote-operated object based on the acquired first operation information and the acquired second operation information; generating first feedback information for feeding back the operation content of the remote-controlled object to the first controller based on the control information, and generating second feedback information for feeding back the operation content of the remote-controlled object to the second controller; generating a comparison image, which is image data to be displayed on a first display unit for the first operator or a second display unit for the second operator, showing a comparison result between an operation input by the first operator and an operation input by the second operator; program.
Citation Information
Patent Citations
Method and system for assisting the operation of robotic machines in congested environments
JP2013521137A
Remote control type mobile robot and robot control system
JP2019005819A
