Server, remote operation system, remote operation method, and program

Abstract

To make a robot appropriately operate.SOLUTION: A server includes: a communication part communicating with one or more robot devices and one or more operation terminals remote-controlling at least one of the one or more robot devices via network; an acquisition part acquiring operation contents input by the operation terminal; an intention estimation part estimating intention of an operator at the operation terminal on the basis of the operation contents acquired by the acquisition part and also updating intention estimation information; and an operation control part generating operation control information for operating the target robot device on the basis of an intention result estimated by the intention estimation part and the updated intention estimation information, and controlling the operation of the robot device on the basis of the generated operation control information.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a server, a remote control system, a remote control method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for remotely controlling a robot to operate experimental equipment or devices using data transmitted from a monitor terminal or another terminal (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-092749 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, there are limitations to the instructions that can be given from a terminal device, so the robot can only be made to perform specific tasks, and there are cases where the robot cannot be made to operate appropriately.

[0005] The aspects of the present invention have been made in consideration of the above circumstances, and one of the objects is to provide a server, a remote control system, a remote control method, and a program that can operate a robot more appropriately. [Means for solving the problem]

[0006] The server, the remote control system, the remote control method, and the program according to the present invention employ the following configuration. (1): A server according to one aspect of the present invention includes a communication unit that communicates via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices; an acquisition unit that acquires operation content input by the operation terminal; an intention estimation unit that estimates an intention of an operator of the operation terminal based on the operation content acquired by the acquisition unit and updates intention estimation information; and an operation control unit that generates operation control information for operating a target robotic device based on the intention result estimated by the intention estimation unit and the updated intention estimation information, and controls the operation of the robotic device based on the generated operation control information, wherein the intention includes an intention of an action that the operator wants the robotic device to perform, and a prediction of the action, and the intention estimation information is a learning model trained using teacher data for inputting operation content for the robotic device and outputting correct intention information for the operation content, and the intention estimation unit updates the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robotic device.

[0007] (2): In the above aspect (1), the acquisition unit acquires robot control information regarding the operation of the robot device and sensor information detected by a sensor provided in the robot device, and further includes a management unit that sets different access rights to the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed.

[0008] (3): In the above aspect (1) or (2), when the operation control unit controls the operation of one robot device based on operation contents obtained from multiple operation terminals, the operation control information is generated based on the priority of operators who operate the operation terminals.

[0009] (4): In the above aspect (3), the intention estimation unit further includes an output control unit that estimates the intention of each of the operators of a plurality of operation terminals that operate the single robot device, and outputs the intention estimation results of the respective operators estimated by the intention estimation unit to each of the plurality of operation terminals.

[0010] (5): Another aspect of the present invention is a remote control system including a server according to any one of (1) to (4) above, the one or more robot devices, and the one or more operation terminals.

[0011] (6): Another aspect of the present invention is a remote operation method, in which a computer communicates via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices, acquires operation content input by the operation terminal, estimates the intention of an operator of the operation terminal based on the acquired operation content, and updates intention estimation information, generates operation control information for operating a target robotic device based on the estimated intention result and the updated intention estimation information, and controls the operation of the robotic device based on the generated operation control information, the intention including an intention of an action that the operator wants the robotic device to perform and a prediction of the action, the intention estimation information being a learning model trained using teacher data for inputting operation content for the robotic device and outputting correct intention information for the operation content, and updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robotic device.

[0012] (7): Another aspect of the present invention is a program that causes a computer to communicate via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices, to acquire operation content input by the operation terminal, to estimate the intention of an operator of the operation terminal based on the acquired operation content, and to update intention estimation information, to generate operation control information for operating a target robotic device based on the estimated intention result and the updated intention estimation information, and to control the operation of the robotic device based on the generated operation control information, wherein the intention includes an intention of an action that the operator wants the robotic device to perform, and a prediction of the action, and the intention estimation information is a learning model trained using teacher data for inputting operation content for the robotic device and outputting correct intention information for the operation content, and the intention estimation information is updated based on history information indicating the operation content input by the operator and the actual operation content of the robotic device. Effect of the Invention

[0013] According to the above aspects (1) to (7), it is possible to make the robot operate more appropriately.

[0014] According to the above aspect (2), the confidentiality of data is protected. Therefore, for example, it is possible to prevent sensor information from being stolen or tampered with by companies that do not have the access rights, and the security of the entire system can be improved. In addition, by managing the information provided based on the access rights, various companies can use the service with peace of mind.

[0015] According to the above aspect (3), even when the robot is operated by a plurality of operators, the robot can be operated appropriately. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an example of a configuration of a remote control system 1 according to an embodiment. [Diagram 2] 1 is a diagram illustrating an example of a configuration of a management server 100 according to an embodiment. [Diagram 3] FIG. 13 is a diagram for explaining the contents of intention estimation information 172B. [Figure 4] FIG. 13 is a diagram for explaining the contents of company-specific user information 174A. [Diagram 5] FIG. 13 is a diagram for explaining the contents of company-specific sensor information 174B. [Figure 6] FIG. 2 is a diagram illustrating an example of a configuration of a robot device 200 according to an embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a configuration of an operation terminal 300 according to the embodiment. [Figure 8] FIG. 2 is a diagram for explaining remote control of the robot device 200 according to the present embodiment. [Figure 9] 1 is a diagram showing an example of an image IM10 displayed by an operation terminal 300. FIG. [Figure 10] FIG. 13 is a diagram showing an example of an image IM20 displayed on the operation terminal 300 when a task based on a priority is executed. [Figure 11] FIG. 13 is a diagram showing an example of an image IM30 for inquiring of an operator about an operation to be executed by the robot device. [Figure 12] FIG. 2 is a diagram showing an example of an image IM40 displayed on a monitor 214 of the robot device 200. [Figure 13] FIG. 2 is a sequence diagram illustrating a process executed by the remote control system 1 in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of a server, a remote control system, a remote control method, and a program according to the present invention will be described with reference to the drawings.

[0018] [System Configuration] FIG. 1 is a diagram showing an example of the configuration of a remote control system 1 according to an embodiment. The remote control system 1 shown in FIG. 1 includes, for example, a management server 100, a robot device 200, and an operation terminal 300. The management server 100 may also include a robot manufacturer terminal 400 and a telecommunications company terminal 500 in addition to the above configuration. The management server 100 is communicatively connected to the robot device 200, the operation terminal 300, the robot manufacturer terminal 400, and the telecommunications company terminal 500 via a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a WAN (Wide Area Network), a LAN (Local Area Network), a provider device, a wireless base station, and the like. The remote control system 1 may include one or more robot devices 200 and one or more operation terminals 300. In the example of FIG. 1, robot devices 200A, 200B, and 200C are shown, but the number and types of the robot devices are not limited to this. 1 shows operation terminals 300A-1 to 300A-n (n is a natural number of 2 or more) and 300B-1 to 300B-n managed for each of a plurality of companies A and B, but the numbers are not limited to this. In the following description, when the robot device and the operation terminal are not particularly identified, they will be simply referred to as "robot device 200" and "operation terminal 300". The management server 100 is an example of a "server".

[0019] The management server 100 manages one or more operation terminals 300 in association with one or more robotic devices 200 to be operated. In this case, the management server 100 may associate one robotic device 200 with each operation terminal 300, may associate one robotic device 200 with multiple operation terminals 300, or may associate multiple robotic devices 200 with one operation terminal 300. The management server 100 also acquires operation content input from the operation terminal 300 and transmits operation control information corresponding to the operation content to the robotic device 200, thereby causing the robotic device 200 to perform remote operation. The management server 100 also transmits information acquired by the robotic device 200 to the operation terminal 300.

[0020] The robot device 200 executes a predetermined operation based on control information transmitted from the management server 100 via the network NW. The robot device 200 includes a moving mechanism that can move by driving wheels, a dolly, a crawler, a crane, or the like. In the example of FIG. 1, the robot device 200A is a robot with at least both arms that can move by a crane. The robot device 200B is a robot with both arms that can move by wheels provided on the lower part. The robot device 200C is a robot with both arms that can move by bipedal walking. The robot device 200 also includes an arm unit that performs tasks such as grasping, moving, and operating an object. The robot device 200 also includes a plurality of various sensors such as vibration, temperature, pressure, and tactile sensors, and transmits data detected by each sensor to the management server 100 at a predetermined cycle or at the timing when a request is received. The robot device 200 may also include a camera that captures the surroundings, a monitor that displays images, a speaker that outputs sound, a microphone that captures surrounding sounds, and the like.

[0021] For example, one or more operation terminals 300 are provided for each company that uses the robot device 200. The operation terminal 300 inputs the operation contents of the robot device 200 and notifies the operator of information acquired by the robot device 200. The operation terminal 300 can operate one or more robot devices 200. The operation terminal 300 may also operate some parts of the robot device 200 (for example, the right arm or the left arm, etc.). The operation terminal 300 may also be provided with, for example, an HMD (head mounted display), an operation device, and an environmental sensor that detects the movement of the operator. Note that the operation terminal 300 is not limited to being provided in a facility within the company, and may be a terminal provided in the operator's home, or a terminal provided in a telework station installed in a station, department store, public facility, internet cafe, etc.

[0022] The robot manufacturer terminal 400 is, for example, a terminal used by a manufacturer or a management company of the robot device 200. The robot manufacturer terminal 400, for example, updates the version of software (programs) installed in each robot device 200, acquires error information from the robot device 200, and performs remote operation or stop control of the robot device 200 based on the acquired error information.

[0023] The communication company terminal 500 is, for example, a terminal used by a communication carrier or the like that manages communication on the network NW. The communication company terminal 500 manages the amount of data communicated for remote operation of the remote operation system 1, manages communication delays, and performs maintenance of the communication environment within the system. The communication company terminal 500 may also handle cases where a failure occurs on the network NW.

[0024] Next, the functions of the management server 100, the robot device 200, and the operation terminal 300 will be specifically described. [Management Server] FIG. 2 is a diagram showing an example of the configuration of the management server 100 according to the embodiment. The management server 100 includes, for example, a communication unit 110, a communication control unit 120, an acquisition unit 130, a management unit 140, a control unit 150, and a storage unit 170. The communication control unit 120, the acquisition unit 130, the management unit 140, and the control unit 150 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device equipped with a non-transient storage medium) such as a hard disk drive (HDD) or flash memory equipped in the management server 100, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and the storage medium may be installed in the HDD or flash memory equipped in the management server 100 by mounting the storage medium in a drive device equipped in the management server 100. The management server 100 may be realized in a server device or a storage device incorporated in a cloud computing system. In this case, the functions of the management server 100 may be realized by a plurality of server devices and storage devices in the cloud computing system.

[0025] The communication unit 110 communicates with one or more robot devices 200, an operation terminal 300 that remotely operates one or more robot devices 200, and other external devices via a network NW under the control of the communication control unit 120. The communication control unit 120 controls the communication in the communication unit 110.

[0026] The communication control unit 120 controls the communication in the communication unit 110. For example, the communication control unit 120 receives the operation content of the robot device 200 input from the operation terminal 300, and transmits operation control information corresponding to the operation content to the robot device 200. The communication control unit 120 also receives information such as the detection results of sensors and camera images transmitted from the robot device 200, and transmits to the operation terminal information to be provided based on the received information.

[0027] The acquisition unit 130 acquires various types of information from the robot device 200 or the operation terminal 300. For example, the acquisition unit 130 acquires operation information for operating the robot device 200 from the operation terminal 300. The acquisition unit 130 also acquires information from a sensor or a camera provided in the robot device 200. The acquisition unit 130 may also acquire information from an external device connected to the network NW.

[0028] The management unit 140 includes, for example, an authentication management unit 142 and an information management unit 144. The authentication management unit 142 manages the authentication and use authority of the operator (system user) who operates the robot device 200. Furthermore, the authentication management unit 142 manages the robot device 200 operated by the operation terminal 300 in association with the operator (system user) based on the authentication result.

[0029] The information management unit 144 manages information available for each user. For example, the information management unit 144 stores robot control information for controlling the robot and information (sensor information) acquired from a sensor provided in the robot device 200 in the storage unit 170, and also sets different access rights for each, and transmits information to the operation terminal 300 that has the corresponding authority to provide information when the set access rights are confirmed. In addition, the information management unit 144 may, for example, set the strength of the access rights for the sensor information to be stronger than that for the robot control information. Increasing the strength of the access rights means, for example, storing the information in a storage area that cannot be accessed unless a specific authentication is permitted, encrypting and managing the data itself, increasing the number and types of authentication conditions such as passwords for acquiring data, and the like. This protects the confidentiality of the data. Therefore, for example, it is possible to prevent the sensor information from being stolen or tampered with by a company that does not have the access rights, and the security of the entire system can be improved. In addition, by managing the information provided by the access rights, various companies and the like can use the remote operation system 1 with peace of mind.

[0030] The control unit 150 includes, for example, a robot information control unit 151, a customized data control unit, an intention estimation unit 153, an operation control unit 154, an operation cancellation unit 155, and an output control unit 156. The robot information control unit 151 manages, for example, what operation the robot device 200 actually performed in response to the operation control information for the robot device 200 generated by the operation control unit 154. The robot information control unit 151 also causes the robot device 200 to perform a basic operation based on the robot control information 172A stored in the storage unit 170. The basic operation is, for example, an operation of moving the robot device 200, changing its posture, and holding or placing an object.

[0031] The customized data control unit 152 generates customized data for causing the robot device 200 to execute specific operations (e.g., assembling an object, soldering, screwing, various cooking processes, etc.) for each company.

[0032] The intention estimation unit 153 estimates the intention of the operator from at least a part of the information of the operation contents inputted by the operation terminal 300. The intention of the operator includes, for example, not only the intention of the action (motion) that the operator wants the robot device 200 to perform, but also prediction of the action and assisting the driving of the robot device 200. For example, in remote operation, a discrepancy occurs between the feeling that the operator is instructing through communication and the like and the motion of the robot device 200. Therefore, the intention estimation unit 153 does not make the robot device 200 directly execute the operation contents from the operator, but estimates the intention of the operator based on the operation contents and generates motion control information for the robot device 200 based on the estimation result, thereby more appropriately assisting the movement of the robot device 200 and realizing remote operation with less discomfort. For example, the intention estimation unit 153 refers to a learning model (intention estimation information 172B) that is learned using teacher data, and estimates the intention corresponding to the operation contents. Note that, during learning, for example, the operation contents are inputted, and learning is performed using teacher data for outputting correct intention information set by the operator or the like for the operation contents.

[0033] Furthermore, the intention estimation unit 153 may update the intention estimation information 172B based on history information indicating the operation content input by the operator and the actual operation content of the robot device 200. This makes it possible to realize more appropriate intention estimation.

[0034] The intention estimation unit 153 may estimate the intention of the operator, for example, by using the Grasp Taxonomy method (for example, see Reference 1). In the embodiment, the intention of the operator is estimated by classifying the operator state by classifying the posture, i.e., the gripping posture, of the operator or the robotic device 200, for example, by using the Grasp Taxonomy method. Reference 1; Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77.

[0035] The operation control unit 154 generates operation control information for operating the target robot device 200 based on the intention of the operator estimated by the intention estimation unit 153, transmits the generated operation control information to the target robot device 200, and controls the operation of the robot device 200. The operation control unit 154 may acquire information on the operation of the robot device 200 obtained by the robot information control unit 151, the customized data control unit 152, etc., in addition to (or instead of) the intention of the operator, and generate the operation control information based on the acquired information. For example, when multiple operators operate two arms simultaneously, the operation control unit 154 may perform synthesis processing such as addition or subtraction of forces based on sensor information such as tactile sense, and perform operation control for working in cooperation. This makes it possible to realize well-balanced operations even when one robot device 200 is operated by multiple operators.

[0036] The action cancellation unit 155 selects an action different from the operator's intention among the actions to be executed by the robotic device 200 by the action control unit 154, and cancels the execution of the selected action. In this case, the action cancellation unit 155 inquires of the operator whether or not the operation content input by the operator is the action to be executed by the robotic device 200, and based on the inquiry result, causes the action control unit 154 to generate action control information and update the intention estimation information 172B.

[0037] The output control unit 156 generates information to be provided to the operator who operates the operation terminal 300, based on the voice and image acquired from the robot device 200. The information to be provided includes images and voice. In addition, the output control unit 156 outputs sensor information and the like, based on the result of the authentication process by the authentication management unit 142.

[0038] The storage unit 170 may be realized by the above-mentioned various storage devices, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM). The storage unit 170 includes, for example, a first storage unit 172 and a second storage unit 174. The first storage unit 172 and the second storage unit 174 are storage areas with different strengths of access rights. The first storage unit 172 stores, for example, information that can be used by all users of the remote control system 1. The second storage unit 174 stores, for example, information that can be acquired only by companies or users whose access rights have been confirmed in advance. The second storage unit 174 may be an area in which data encrypted for each company is stored and cannot be viewed unless decrypted using a key or the like assigned to each company. The setting of the access rights by the first storage unit 172 and the second storage unit 174 is managed by the authentication management unit 142. In addition, the first storage unit 172 and the second storage unit 174 may be different storage units, and one or both may be managed by an external device such as a data server.

[0039] For example, the first storage unit 172 stores, for example, robot control information 172A and intention estimation information 172B. The robot control information 172A is basic control information for causing the robot device 200 to execute a predetermined motion. The predetermined motion is, for example, motion such as moving, changing direction, moving an arm, holding (grasping) an object, and placing an object. These may be set for each type of the robot device 200, for example.

[0040] The intention estimation information 172B is, for example, information on an intention estimation result estimated by the intention estimation unit 153. Fig. 3 is a diagram for explaining the contents of the intention estimation information 172B. In the intention estimation information 172B, for example, information on an estimated intention is associated with the operation content acquired from the operation terminal 300. These pieces of information may be managed for each operator, or the intention estimation information 172B of a plurality of operators may be integrated to store only information on frequently occurring intention estimation, or may be updated based on history information or the like.

[0041] Information accessible to a user whose access right has been confirmed is stored in the second storage unit 174. For example, company-specific user information 174A and company-specific sensor information 174B are stored in the second storage unit 174.

[0042] FIG. 4 is a diagram for explaining the contents of the company-specific user information 174A. In the example of FIG. 4, user information for each company is stored, and each piece of information is managed so that it cannot be accessed unless the access right of each company is confirmed. The company-specific user information 174A is, for example, authentication information for authenticating a user when using a service by the remote operation system 1, and information such as a name, authority, priority, and image information are associated with the authentication information. The authentication information includes, for example, a user ID and a password that are identification information for identifying a user. The authentication information may also include biometric authentication information such as fingerprint information and iris information. The authority includes, for example, authority information such as an access right assigned to a user. Based on this authority information, sensor information acquired by the robot device 200 can be acquired. The information management unit 144 can manage who can access which data, which data cannot be accessed, and so on, by referring to the information stored in the authority. The priority information stores, for example, priorities in the case where a plurality of operators remotely operate the robot device 200 and the operation contents of each operator are the same or similar. The image information is an image for recognizing a user, and includes, for example, a facial image of the user, an avatar image created to resemble the user, an illustrated image, and the like.

[0043] FIG. 5 is a diagram for explaining the contents of the company-specific sensor information 174B. In the example of FIG. 5, sensor information for each company is stored, and each piece of information is managed so that it cannot be accessed unless the access right of each company is confirmed. In the company-specific sensor information 174B, a robot ID, which is identification information for identifying the robot device 200, is stored with a date and time, an operator ID, operation parts, sensor information, etc. The operator ID stores the user ID of the operator who performed the operation. The operation parts store the parts of the robot ID operated by the operator. This makes it possible to manage the operation history of each operator even when one robot device 200 is operated by multiple operators.

[0044] [Robot device] 6 is a diagram showing an example of the configuration of a robot device 200 according to an embodiment. The robot device 200 includes, for example, a communication unit 202, a sensor 204, a camera (an example of an imaging unit) 206, a microphone 208, a driving unit 210, a monitor 214, a speaker 216, a control device 240, and a storage unit 260. The driving unit 210 includes, for example, an arm unit 210A and a moving driving unit 210B. The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are each realized by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device having a non-transient storage medium) such as an HDD or flash memory provided in the robot device 200, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed in the HDD or flash memory provided in the robot device 200 by mounting the storage medium in a drive device provided in the robot device 200.

[0045] The communication unit 202 will be described later. Under the control of the communication control unit 242, the communication unit 202 communicates with the management server 100 and other external devices via the network NW. The communication unit 202 may also communicate with other robot devices 200, or may communicate with the operation terminal 300.

[0046] The sensor 204 includes a position sensor that detects the position of the robot device 200, a speed sensor that detects the speed, a temperature sensor that detects the temperature at a specific position such as the periphery of the robot device 200 or the tip of the arm unit 210A, and the like. The position sensor receives information from, for example, a GPS (Global Positioning System) receiving device, and obtains position information (longitude and latitude information) based on the received information. The sensor 204 may also include a humidity sensor that detects the surrounding humidity, and a vibration sensor that detects vibrations of an object operated by the robot device 200. The sensor 204 may also include an object detection sensor that detects surrounding objects. The surrounding objects are, for example, other robot devices 200, people, obstacles, and the like.

[0047] The camera 206 is, for example, a digital camera using a solid-state imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 206 captures an image of the periphery of the robotic device 200 at a predetermined timing, for example. The number of cameras 206 is not limited to one, and a plurality of cameras may be provided in the robotic device 200. For example, the camera 206 is provided in the head of the robotic device 200. The camera 206 may also be provided near the tip of the arm unit 210A. This allows an image of the object to be worked on to be captured at a close distance, making it easier to perform more detailed work.

[0048] The microphone 208 receives sounds from around the robot device 200. The microphone 208 outputs information based on the input sounds to the control device 240.

[0049] The arm unit 210A grasps a target object and performs a predetermined task on the target object. The arm unit 210A is, for example, a multi-joint robot arm, and includes, for example, an actuator, a gear, an artificial muscle, and the like. For example, the arm unit 210A includes a first arm unit in which one end of the multi-joint robot arm is connected near the right side of the robot device main body, and a second arm unit in which one end of the other multi-joint robot arm is connected near the left side of the robot device main body. The arm unit 210A may also include other arm units in addition to the first arm unit and the second arm unit. Hereinafter, when there is no need to distinguish between the respective arms, they will simply be referred to as the "arm unit 210A". The other end of the arm unit is configured with a gripping unit capable of gripping a predetermined object. The arm unit 210A is driven based on the control of the control device 220. The arm unit is capable of movement equivalent to that of a human arm.

[0050] The moving drive unit 210B is a drive unit for moving the robot device 200 on the floor or the ground. The moving drive unit 210B may be, for example, two legs, or may be provided with a moving mechanism such as wheels, a dolly, or a crawler. For example, when the moving drive unit 210B is a leg, the leg operates to make the robot device 200 walk based on the control of the control device 220. The moving drive unit 210B may also be structured to be movable along a ceiling or a rail by driving a crane or the like. With these configurations, the robot device 200 can move in a desired direction. Note that the drive unit 210 may be provided with a mechanism for driving other joints such as a waist or a head in addition to the arm unit 210A or the moving drive unit 210B. The drive unit 210 is driven based on the control of the drive control unit 246.

[0051] The monitor 214 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The monitor 214 displays the information output by the output control unit 248 as an image. A plurality of monitors 214 may be provided in the robot device 200. The monitor 214 may be provided, for example, on the head, abdomen, or back. The speaker 216 outputs the information output by the output control unit 248 as sound.

[0052] The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuit units) such as an LSI, ASIC, FPGA, or GPU, or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and may be installed by mounting the storage medium in a drive device.

[0053] The communication control unit 242 communicates with the management server 100, for example, by wireless communication via the communication unit 202, to transmit and receive information. The communication control unit 242 may also communicate with other robotic devices 200 via the communication unit 202.

[0054] The acquisition unit 244 acquires the detection result (sensor information) of the sensor 204 provided in the robot device 200. The acquisition unit 244 also acquires an image captured by the camera 206. The acquisition unit 244 also acquires information based on a sound input from the microphone 208.

[0055] The drive control unit 246 operates the arm unit 210A and the moving drive unit 210B based on the motion control information acquired from the management server 100. The drive control unit 246 also drives other parts of the robot device 200 (e.g., the head, torso, waist, etc.) based on the motion control information to change the posture, etc. The drive control unit 246 may also drive the robot device 200 based on the basic motion information 262 and customized motion information 264 stored in the storage unit 260. The basic motion information 262 is drive control information for executing a basic motion according to the type, etc. of the robot device 200. The customized motion information 264 is drive control information for causing the robot device 200 to execute a specific motion, for example, registered in advance for each operator.

[0056] The output control unit 248 causes the monitor 214 to output an image based on the information acquired from the management server 100. The output control unit 248 also causes the speaker 216 to output a voice, a warning sound, or the like based on the information acquired from the management server 100.

[0057] The storage unit 260 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM, etc. The storage unit 260 stores, for example, basic operation information 262, customized operation information 264, programs, and other information.

[0058] [Operation terminal] 7 is a diagram showing an example of a configuration of an operation terminal 300 according to an embodiment. The operation terminal 300 includes, for example, a communication unit 310, an HMD 320, an operation device 330, an environmental sensor 340, a control device 360, and a storage unit 380. The HMD 320, the operation device 330, and the environmental sensor 340 are examples of an "HMI (Human Machine Interface)."

[0059] The communication unit 310 communicates with the management server 100 via the network NW under the control of the control device 360. The communication unit 310 may also communicate with other operation terminals 300 and the robot device 200.

[0060] The HMD 320 includes, for example, an image display unit 322, a gaze detection unit 324, a sensor 326, a speaker 328, and a control unit 329. The HMD 320 outputs and displays the state image and sound of the robot device 200 that the control device 360 ​​receives from the management server 100, and detects the movement of the operator's gaze, etc.

[0061] The image display unit 322 is, for example, an LCD, an organic EL display, etc. Under the control of the control unit 329, the image display unit displays an image output by an output control unit 368, which will be described later.

[0062] The gaze detection unit 324 detects the gaze of the operator wearing the HMD 320 under the control of the control unit 329, and outputs the detected gaze information (operator sensor value) to the control device 360. The gaze information is information including, for example, a gaze vector.

[0063] The sensor 326 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 326 detects the tilt and rotation of the head of the operator wearing the HMD 320, and outputs the detected head operation information (operator sensor value) to the control device 360.

[0064] The speaker 328 outputs voice, warning sounds, and the like output by the output control unit 368 .

[0065] The control unit 329 controls the execution of gaze detection by the gaze detection unit 324 based on control information from the control device 360, controls detection by the sensor 326, controls the image display on the image display unit 322, and accepts information input from the operator via the HMD 320.

[0066] The operation device 330 includes, for example, a sensor (operator sensor) 332, a control unit 334, a feedback unit 336, and an input unit 338. The operation device 330 includes, for example, a tactile data glove that is worn on the hand of the operator.

[0067] The sensor 332 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 332 includes a plurality of sensors. The sensor 332 tracks the movement of each finger using, for example, two sensors. The sensor 332 detects operator arm information (operator sensor value) which is information on the orientation and movement of each finger of the operator, the posture and position of the operator's arm such as the movement of the hand, etc., under the control of the control unit 334, for example. The operator arm information (operator sensor value) includes information covering the entire human arm, such as hand tip position and posture information, angle information of each finger, elbow position and posture information, and information tracking the movement of each part.

[0068] The control unit 334 outputs the operator's arm information detected by the sensor 332 to the control device 360. In addition, the control unit 334 controls the feedback unit 336 based on feedback information acquired from the control device 360.

[0069] The feedback unit 336 feeds back feedback information to the operator in response to the control of the control unit 334. The feedback unit 336 feeds back sensations to the operator in response to the feedback information, for example, by means of vibration applying means (not shown), air pressure applying means (not shown), hand movement restricting means (not shown), temperature sensing means (not shown), hardness or softness sensing means (not shown), vibration sensing means (not shown), or the like, which are attached to the arm unit 210A of the robot device 200.

[0070] The input unit 338 is, for example, an input device other than the tactile data glove, such as a keyboard, a mouse, a lever, a touch panel, a microphone, etc. The input unit 338 accepts input of operation details for the robot device 200 from each input device.

[0071] The environment sensor 340 detects, for example, the operation of the operator. The environment sensor 340 includes, for example, a camera (an example of an imaging unit) 342, a sensor 344, and an object position detection unit 346. The camera 342 captures an image including the operator. The camera 342 is, for example, an RGB camera. The camera 342 outputs the captured image to the object position detection unit 346. Note that in the environment sensor 340, the positional relationship between the camera 342 and the sensor 344 is known.

[0072] The sensor 344 is, for example, a depth sensor. The sensor 344 outputs a detection result to the object position detection unit 346. Note that the camera 342 and the sensor 344 may be distance sensors.

[0073] The object position detection unit 346 detects the three-dimensional position, size, shape, and the like of the target object in the captured image by using a known method based on the image captured by the camera 342 and the detection result detected by the sensor 344. The object position detection unit 346 performs image processing (edge ​​detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, and the like) on the image captured by the camera 342 with reference to a pattern matching model and the like stored in the object position detection unit 346 to estimate the position of the object. When multiple objects are detected from the captured image, the object position detection unit 346 detects the position of each object. The object position detection unit 346 transmits the detected object position information to the control device 360. The data transmitted by the environment sensor 340 may be, for example, a point cloud having position information.

[0074] The control device 360 ​​includes, for example, a communication control unit 362, an acquisition unit 364, an operation content generation unit 366, and an output control unit 368. The communication control unit 362, the acquisition unit 364, the operation content generation unit 366, and the output control unit 368 are realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuit units) such as an LSI, ASIC, FPGA, or GPU, or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and may be installed by mounting the storage medium in a drive device.

[0075] The communication control unit 362 communicates with the management server 100 by wireless communication via the communication unit 310, for example, to transmit and receive information. The communication control unit 362 may also communicate with another operation terminal 300 via the communication unit 310. The communication control unit 362 also communicates with the HMD 320, the operation device 330, and the environmental sensor 340.

[0076] The acquisition unit 364 acquires information obtained from the HMD 320, the operation device 330, and the environment sensor 340. The operation content generation unit 366 generates operation content for the robotic device 200 based on the information acquired by the acquisition unit 364. For example, the operation content generation unit 366 generates operation content related to gaze information and head direction based on information acquired from the HMD 320. The operation content generation unit 366 also generates operation content related to the movement of the arm unit 210A of the robotic device 200 based on information acquired from the operation device 330. The operation content generation unit 366 also generates operation content related to the posture, movement direction, and movement amount of the robotic device 200 based on information acquired from the environment sensor 340. The information generated by the operation content generation unit 366 is transmitted to the management server 100 via the communication unit 310.

[0077] The output control unit 368 outputs the provided information acquired from the management server 100 to the HMD 320 and the operation device 330. For example, the output control unit 368 causes an image acquired from the management server 100 to be output to the image display unit 322 of the HMD 320. The output control unit 368 also causes a speaker 328 to output a voice, a warning sound, or the like based on the information acquired from the management server 100. The output control unit 368 also causes the operation device 330 to output feedback information for conveying tactile information acquired from the management server 100 to the operator.

[0078] The storage unit 380 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM, etc. The storage unit 380 stores, for example, programs and other information.

[0079] [Regarding remote control of the robot device according to the embodiment] Next, the remote operation of the robot device 200 by the remote operation system 1 according to the embodiment will be specifically described. FIG. 8 is a diagram for explaining the remote operation of the robot device 200 according to the present embodiment. FIG. 8 shows an example in which one robot device 200 is operated based on the operation contents of two operators U1 and U2. As shown in FIG. 8, the operators U1 and U2 each wear an HMD 320 and an operation device 330. The operation terminals 300A-1 and 300A-2 are terminals of the same company (for example, company A), but may be installed in different places.

[0080] In addition, the operators U1 and U2 are provided with an environmental sensor 340 for measuring the operator's movements and the surrounding environment. Note that a configuration similar to the environmental sensor 340 may be attached to, for example, the robot device 200. In addition, the operation device 330 includes, for example, an operation device 330a worn on the operator's left hand and an operation device 330b worn on the right hand. The operation contents input from each of the operation devices 330a and 330b are used to control the movements of the left hand LH and right hand RH of the robot device 200. The position of the operator's arm, the direction of the face, and the like are detected by the environmental sensor 340. The detected data is transmitted to the management server 100 via a network.

[0081] The management server 100 estimates the intention of each user based on the operation contents obtained from the operation terminals 300A-1 and 300A-2. Here, an example of information estimated by the intention estimation unit 153 will be specifically described. The intention estimation unit 153 estimates the operator's intention of operation based on the acquired operation contents from the operator. The intention estimation unit 153 classifies the posture of the arm including the grip of the robot device by classifying the posture of the arm of the operator based on the operator sensor value of the operation terminal 300. In addition, the intention estimation unit 153 estimates the intention of operation that the operator wants the robot device 200 to perform based on the classification result. The intention estimation unit 153 estimates, for example, the movement of hands and fingers at each time, the purpose of the work that the robot device 200 wants to perform, the work contents, the movement of hands and fingers at each time, and the like as the operator's intention of operation. The purpose of the work is, for example, grasping an object, moving an object, and the like. The work contents are, for example, grasping and lifting an object, grasping and moving an object, and the like.

[0082] Furthermore, the intention estimation unit 153 may estimate the operator's intention to operate by, for example, inputting the operation content into a learned model or intention estimation information 172B stored in the storage unit 170. In the embodiment, the intention estimation is performed by classifying the grip posture, so that the operator's intention to operate can be estimated with high accuracy. Note that other methods may be used to classify the grip posture.

[0083] The intention estimation unit 153 may also perform an integrated estimation using the gaze and the movement of the arm. In this case, the intention estimation unit 153 may input gaze information, hand movement information, and position information of the object OB on the table TB to a trained model to estimate the operator's intention.

[0084] In the example of FIG. 8, the intention estimation unit 153 estimates the object to be held based on, for example, line of sight information. Next, the intention estimation unit 153 estimates the posture of the operator's hand based on the estimated object to be held. Also, the intention estimation unit 153 may first estimate the posture of the operator's hand based on the operation content, and then estimate the object to be held from the estimated posture of the operator's hand. Note that, when the same or similar operation content is input from the operators U1 and U2, the intention estimation is performed based on the operation content from the operator with the higher priority. Also, when the operations of different parts are assigned to each of the operators U1 and U2, the intention estimation unit 153 estimates the intention based on the operation content of each. In the following, it is assumed that the operator U1 operates the arm unit 210A with his right hand, and the operator U2 operates the arm unit 210A with his left hand.

[0085] For example, when an object OB is placed on a table TB shown in Fig. 8, the intention estimation unit 153 estimates that the object OB will be grasped based on the posture of the hand. The intention estimation unit 153 may also estimate in advance the future trajectory of the hand intended by the operator based on the operation content and state information of the robot device 200. The intention estimation unit 153 may also estimate the object to be grasped and the position of the object by using the detection result detected by the sensor, the result of image processing of the image captured by the environment sensor 340, and the like.

[0086] In addition, since the coordinate system is different between the environment in which the operator operates and the robot operating environment, for example, calibration between the operator's operating environment and the robot operating environment may be performed when starting the robot device 200. In addition, when grasping, the management server 100 may determine the grasping position based on the grasping force of the robot device and the frictional force between the object and the grasping part, etc., taking into consideration the error in the grasping position at the time of grasping.

[0087] When the robot device 200 is not remotely operated, its operation is controlled according to the control of the control device 240, and when it is remotely operated, its operation is controlled according to operation control information generated by the management server 100 based on the estimated intention, etc.

[0088] The control device 240 of the robot device 200 controls the driving unit 210 based on the operation control information from the management server 100. The robot device 200 outputs information such as sensor information detected by the sensor 204, images captured by the camera 206, and audio output by the microphone 208 to the management server 100. The management server 100 generates information to be provided to the operation terminals 300A-1 and 300A-2 based on the acquired sensor information, images, audio, and the like. In this case, the management server 100 may generate an image in which at least a part of the sensor information is superimposed on the image. The management server 100 generates information to be provided including information to be transmitted by the sense of touch. Since the robot device 200 can be operated by a plurality of operators, an image in which information for identifying the operator of each part of the robot device 200 is superimposed on the camera image may be generated. The operation terminal 300 displays the image output from the management server 100.

[0089] FIG. 9 is a diagram showing an example of an image IM10 displayed by the operation terminal 300. The display mode of the image IM10, such as the layout and display contents, is not limited to the example of FIG. 9. The same applies to the display modes of other images described below. In the image IM10 shown in FIG. 9, a sensor information display area A11, an operator information display area A12, and a work status display area A13 are superimposed on the image IM10 including the arm part (left hand LH, right hand RH) of the robot device 200 captured by the camera 206. In the example of FIG. 9, an object OB1 is placed on a table TB in front of the robot device 200. The left hand LH of the robot device 200 is holding and lifting an object OB2, and the right hand RH of the robot device 200 is holding and lifting an object OB3. The sensor information display area A11, the operator information display area A12, and the work status display area A13 may be displayed with a predetermined transmittance so that objects in the real space are not blocked.

[0090] In the example of FIG. 9, the sensor information display area A11L displays sensor information related to the left hand of the robot device 200. Moreover, the sensor information display area A11R displays sensor information (e.g., position, pressure, temperature, etc.) related to the right hand of the robot device 200. Moreover, in the example of FIG. 9, the operator information display area A12L displays information related to the operator of the left hand of the robot device 200. The operator information display area A12R displays information related to the operator of the left hand of the robot device 200. This allows, for example, when the robot device 200 is operated by multiple people, the other operators to accurately know who is operating each part of the robot device 200. Moreover, since it is possible to know not only the sensor information of the part being operated by oneself but also information related to the operation of other operators, more accurate processing can be executed.

[0091] The work status display area A13 displays the results of each intention estimation. In the example of FIG. 9, information on the intentions estimated from the operation contents of each of the operators U1 and U2 is displayed. Furthermore, when the management server 100 estimates through intention estimation that each operator will perform the same or similar processing, it executes one intention with priority based on priority information. In this case, the work status display area A13 may be displayed for the other operator, and a warning or the like may be presented by changing the sound or display mode. In the example of FIG. 9, since both operators U1 and U2 are trying to place the object they are holding on the object OB1, warning information is presented. The warning mode may be made different depending on the degree of danger.

[0092] In this case, the operation control unit 154, for example, refers to the priority of each operator and causes the robot device 200 to execute an operation based on the intention of the operator with the highest priority. Furthermore, when a process is executed based on the priority, the management server 100 causes the display unit of the operation terminal 300 to display an image including the process content.

[0093] FIG. 10 is a diagram showing an example of an image IM20 displayed on the operation terminal 300 when a task based on a priority is executed. In the example of FIG. 10, the content displayed in the work status display area A13 is different from that of the image IM10 shown in FIG. 9. Therefore, hereinafter, the work status display area A13 will be mainly described. In the example of FIG. 10, the work status display area A13 displays information indicating that the instruction of the operator U2 has been preferentially executed based on the priority. This allows the operators U1 and U2 to more accurately grasp the intention estimation results for their respective operation instructions and the actual work status.

[0094] Furthermore, the intention estimation unit 153 may store the estimation result in the storage unit and use it for the next or subsequent intention estimation. Furthermore, when detecting an unknown action, the intention estimation unit 153 may inquire of the operator whether to cause the robot device 200 to execute the action.

[0095] FIG. 11 is a diagram showing an example of an image IM30 for inquiring of an operator about an operation to be performed by the robot apparatus. The image IM30 includes, for example, an operation inquiry information display area A31 and a switch display area A32. The operation inquiry information display area A31 displays information for inquiring whether or not the robot apparatus 200 needs to execute an operation content input by the operation device 330. For example, when an operation of touching the head is detected but the intention of this operation cannot be estimated, the intention estimation unit 153 displays an image showing text information such as "An operation of touching the head has been detected. Do you want the robot to perform this operation?" on the operation terminal 300, as shown in the operation inquiry information display area A31 of FIG. 11.

[0096] In the switch display area A32, for example, icons IC31 and IC32 are displayed. The icons IC31 and IC32 are, for example, GUI (Graphical User Interface) switches. When the icon IC31 is selected, the management server 100 generates operation control information for making the robot device 200 execute the instructed operation content. When the icon IC12 is selected, the HMI control unit 180 does not make the robot device 200 execute the instructed operation content. Furthermore, the management server 100 may execute a process to prevent the robot device from inputting the same operation even if the same operation is input in the future. This makes it possible to suppress the robot from executing an operation that is unconsciously executed due to the operator's habit, etc., and to prevent the robot from performing unnecessary operations.

[0097] Furthermore, operators around the robotic device 200 may feel uneasy because they do not know who is operating the robotic device 200. Therefore, the management server 100 may display information about the operator of the robotic device 200 on the monitor of the robotic device 200. In this case, when the robotic device 200 is operated by multiple operators, the operator of each part may be presented.

[0098] FIG. 12 is a diagram showing an example of an image IM40 displayed on the monitor 214 of the robot device 200. The image IM40 includes, for example, one or both of the operator information display areas A41 and A42. The operator information display area A41 displays character information related to the operator who operates the robot device 200. The operator information display area A41 may also include information related to the parts of the robot device 200 operated by the operator. In the example of FIG. 12, the operator information display area A41 displays character information such as "The right hand of the operator U2 and the left hand of the operator U1 are operating this robot." The operator information display area A42 displays an avatar image registered in the user DB. This allows surrounding operators to more accurately recognize the person remotely operating the robot and the parts they are in charge of.

[0099] [Processing sequence] Next, the process executed by the remote control system in the embodiment will be described with reference to a sequence diagram. Fig. 13 is a sequence diagram for explaining the process executed by the remote control system 1 in the embodiment. In the example of Fig. 13, the management server 100, the robot device 200, and the operation terminals 300-1 and 300-2 are used for explanation. Also, in Fig. 13, an example in which the robot device 200 is operated by the two operation terminals 300-1 and 300-2 is explained.

[0100] The operation terminal 300-1 and the operation terminal 300-2 each access the management server 100 to remotely operate the robot device 200 (steps S100, S102). The management server 100 performs an authentication process to determine whether or not the remote operation system 1 is available based on authentication information and the like input from the operation terminals 300-1 and 300-2 (step S104). The following describes a case where use is permitted by authentication. If use is permitted, the management unit of the management server 100 assigns the robot device 200 to be operated by the operation terminals 300-1 and 300-2 (step S106) and requests the assigned robot device for startup control (step S108).

[0101] The robot device 200 starts up the device based on a start-up request from the management server 100 (step S110), and outputs the detection results of the sensor of the robot device 200, images captured by the camera, and the like to the management server 100 (step S112).

[0102] The management server 100 acquires the information transmitted from the robot device 200, and generates information (provided information) to be provided to the operation terminals 300-1 and 300-2 based on the acquired information (step S114). Then, the management server 100 transmits the generated provided information to the operation terminals 300-1 and 300-2, respectively (steps S116 and S118).

[0103] Each of the operation terminals 300-1 and 300-2 receives the provided information transmitted from the management server 100, and provides the information by outputting the received information from the HMD, the operation device, etc. (steps S120 and S122).

[0104] Next, the operation terminals 300-1 and 300-2 transmit operation details related to the robot device 200 to the management server 100 (steps S124 and S126). The management server 100 performs intention estimation based on the operation details obtained from each of the operation terminals 300-1 and 300-2 (step S128), generates operation control information based on each intention estimation result (step S130), and transmits the generated operation control information to the robot device 200 (step S132).

[0105] The robot device 200 performs an operation based on the operation control information (step S134), and transmits information obtained from sensors or the like (sensor information), camera images, and the like during or as a result of the operation to the management server 100 (step S136).

[0106] The management server 100 stores the received information in the storage unit 170 (step S138), generates information to be provided for transmitting the acquired information to the operation terminals 300-1, 300-2 (step S140), and transmits the generated information to be provided to the operation terminals 300-1, 300-2 (steps S142, S144). Each of the operation terminals 300-1, 300-2 outputs the acquired information to provide the information to the operator (steps S146, S148). After that, the processes of steps S124 to S148 are continuously performed until the operation of the robot device 200 is completed.

[0107] <Modification> A part or all of the processes of the management server 100, the robot device 200, and the operation terminal 300 in the embodiment may be realized by AI (Artificial Intelligence) technology. In addition, a part or all of the functional units or information stored in the functional units or storage units included in the management server 100, the robot device 200, and the operation terminal 300 may be included in other devices.

[0108] In the above embodiment, the case where a plurality of operators operate one robot device 200 has been described, but a single operator may operate a plurality of robot devices 200. In this case, sensor information detected by each of the plurality of robot devices 200 is displayed on the image display unit 322 of the HMD 320. The management server 100 may transmit operation history information, intention estimation information, and the like to the robot manufacturer terminal 400. This can be used for developing the robot manufacturer terminal 400 and products, updating programs, and the like. In this case, a discount or reduction service for system usage may be provided to a company that performs eco-driving of the robot device 200 (for example, low-power driving, low-load driving, low-communication-amount driving), or a company that provides a robot control method suitable for an operator or telework characteristics. This can promote information provision from users, which can be useful for improving services.

[0109] In the above embodiment, when a single robot device 200 is operated by multiple operators, the operation intentions and work statuses of each operator are displayed. However, for example, the above-mentioned intention estimation result, work status, and information about the operator may be provided between multiple robot devices 200 that exist within a predetermined distance or between robot devices performing a joint task. By using the intention estimation function between robots, it becomes possible to share consciousness more effectively than between humans. In addition, since the statuses of surrounding robot devices 200 can be acquired, group work, etc. can be performed smoothly.

[0110] The embodiment described above can operate the robot more appropriately by, for example, providing in a server a communication unit that communicates via a network NW with one or more robotic devices 200 and one or more operation terminals 300 that remotely operate at least one of the one or more robotic devices 200, an acquisition unit that acquires operation content input by the operation terminal 300, an intention estimation unit that estimates the intention of the operator of the operation terminal 300 based on the operation content acquired by the acquisition unit, and an operation control unit that controls the operation of the robotic device 200 based on the intention result estimated by the intention estimation unit.

[0111] According to the above-described embodiment, when performing teleworking by remotely controlling a robot, various devices can be used more generally and can be operated by a team. Furthermore, while in the past, only limited tasks could be performed using a limited number of dedicated robots even when remotely controlled, according to the present embodiment, anyone can easily perform teleworking using a highly versatile robot.

[0112] According to the embodiment, the robot information and the sensor information (information that may include corporate secrets) can be managed separately, and a server capable of handling confidential information can be provided. Furthermore, remote operators can access the same server and can share the control rights and sensor information of the same robot or different robots. For example, when operating the same robot device, multiple operators can operate different arms, so that a senior and a junior can simultaneously operate the same arm of the same robot, and can add or subtract force while sharing sensor information such as touch. Furthermore, when operating different robots, telepathy is possible by sharing the intention estimation function and disclosing the intention of others. Furthermore, by using the intention estimation function, a warning can be issued to prevent the robots from moving in the same direction or colliding when trying to pick up the same thing. Furthermore, according to the embodiment, one operator can operate two robots. In this case, for example, if two robots are to carry a large desk, one robot is brought to the end of the desk in advance and synchronized with the other robot. Furthermore, when the other robot is brought to the other end and an action to lift it is attempted, the robots are lifted together by synchronization processing. Furthermore, in the embodiment, more accurate work can be achieved than when humans call out to each other to perform the work. Also, in the embodiment, when one operator makes many robots perform the same work, the supported teleoperation function can absorb the individual differences between the robots and the individual differences between the objects to be worked on.

[0113] The above-described embodiment can be expressed as follows. A server computer that communicates with a mobile object and a mobile terminal of a user of the mobile object comprises: a storage device that stores a program; a hardware processor; The hardware processor executes the program stored in the storage device, communicating with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network; Acquire operation contents inputted by the operation terminal; Estimating an intention of an operator of the operation terminal based on the acquired operation content, and updating intention estimation information; generating motion control information for operating the target robot device based on the estimated intention result and the updated intention estimation information, and controlling the motion of the robot device based on the generated motion control information; the intention includes an intention of a behavior that the operator wants the robot device to perform and a prediction of the behavior; the intention estimation information is a learning model trained using training data for inputting an operation content for the robot device and outputting correct intention information for the operation content, updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot apparatus; The server is configured as follows:

[0114] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0115] 1... remote operation system, 100... management server, 110, 202, 310... communication unit, 120... communication control unit, 130... acquisition unit, 140... management unit, 150... control unit, 170... memory unit, 200... robot device, 204... sensor, 206... camera, 208... microphone, 210... drive unit, 214... monitor, 216... speaker, 240... control device, 260... memory unit, 300... operation terminal, 320... HMD, 330... operation device, 340... environment sensor, 360... control device, 380... memory unit

Claims

1. a communication unit that communicates with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network; an acquisition unit that acquires operation content inputted by the operation terminal; an intention estimation unit that estimates an intention of an operator of the operation terminal based on the operation content acquired by the acquisition unit, and updates intention estimation information; a motion control unit that generates motion control information for operating the target robot device based on the intention result estimated by the intention estimation unit and the updated intention estimation information, and controls the motion of the robot device based on the generated motion control information, the intention includes an intention of a behavior that the operator wants the robot device to perform and a prediction of the behavior; the intention estimation information is a learning model trained using training data for inputting an operation content for the robot device and outputting correct intention information for the operation content, the intention estimation unit updates the intention estimation information based on history information indicating an operation content input by the operator and an actual operation content of the robot device. server.

2. the acquiring unit acquires robot control information relating to an operation of the robot device and sensor information detected by a sensor provided in the robot device; a management unit that sets different access rights to the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed, The server of claim 1 .

3. the operation control unit generates the operation control information based on priorities of operators who operate the operation terminals when controlling an operation of one robot device based on operation contents acquired from a plurality of operation terminals. The server according to claim 1 or 2.

4. the intention estimation unit estimates an intention of each of operators of a plurality of operation terminals that operate the one robot device, an output control unit that outputs an intention estimation result of each of the operators estimated by the intention estimation unit to each of the multiple operation terminals; The server according to claim 3.

5. A server according to any one of claims 1 to 4; the one or more robotic devices; The one or more operation terminals; A remote control system comprising:

6. The computer communicating with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network; Acquire operation contents inputted by the operation terminal; Estimating an intention of an operator of the operation terminal based on the acquired operation content, and updating intention estimation information; generating motion control information for operating the target robot device based on the estimated intention result and the updated intention estimation information, and controlling the motion of the robot device based on the generated motion control information; the intention includes an intention of a behavior that the operator wants the robot device to perform and a prediction of the behavior; the intention estimation information is a learning model trained using training data for inputting an operation content for the robot device and outputting correct intention information for the operation content, updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot apparatus; Remote operation method.

7. On the computer, Communicating with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network; Acquiring operation details inputted by the operation terminal; An intention of an operator of the operating terminal is estimated based on the acquired operation content, and intention estimation information is updated; generating motion control information for operating the target robot device based on the estimated intention result and the updated intention estimation information, and controlling the motion of the robot device based on the generated motion control information; the intention includes an intention of a behavior that the operator wants the robot device to perform and a prediction of the behavior; the intention estimation information is a learning model trained using training data for inputting an operation content for the robot device and outputting correct intention information for the operation content, updating the intention estimation information based on history information indicating the operation content input by the operator and the actual operation content of the robot device; program.

Images